When you work with the public cloud, Virtual Machines and containers, you encounter a lot of different Linux distributions. It’s hard to keep up with all the changes, but even harder when you think you know them, and turn out to be wrong.

I almost got burned last week, because I assumed that all Linux distros in 2018 were all systemd-based. It prompted me to make this Google Presentation for our weekly tech meetings.

What follows below is a blog version of that presentation.

Overview of init systems

Several different packages providing /sbin/init binaries exist.

  • System V Init - great-granddaddy of them all, serially starts services
  • Upstart - started by Ubuntu, achieved boot-time improvements through parallelism
  • Systemd - much newer, comprehensive service management infrastructure

There are, of course, many, many others: BSD init, runit, Apple launchd, minit, jinit, cinit, monit, depinit even tini (init spelled backwards), a mini-init for use inside Docker containers

Fun Facts about /sbin/init

  • Most distributions went through their own separately-timed civil wars over adopting the systemd package, replacing the sysvinit packages. So the adoption date of systemd depends on which distribution you’re using.
  • Different commands are required to operate each init system.

  • Compatibility shims can be included with newer systems, but not always.

  • You can change which program the kernel boots from within GRUB.
    • Example: init=/bin/bash boots a shell instead of init, bypassing root password.
    • This is handy for resetting a forgotten root password.

Init’s role in the Linux boot process

Boot Sequence

  • BIOS / POST sequence chooses boot device
  • GRUB - allows you to select a Kernel and Linux boot parameters
  • Linux kernel boots, mounts / partition
  • Linux starts /sbin/init as PID 1
  • /sbin/init starts all other processes
  • /sbin/init monitors some (but not all) processes, restarts when necessary
  • Linux is ready to use at some point.

SystemV Init system

  • Very old, very stable init system from the original AT&T / Bell Labs greybeards. Well known across the *BSD and Unix worlds, in addition to Linux. (Think AIX, Solaris, SCO, Sequent, HP-UX, etc)
  • Processes to be automatically respawned go in /etc/inittab
  • Other processes started by executing per-service scripts in /etc/init.d.
  • Configuration controlled by symlinks in /etc/rcX.d that point to scripts in /etc/init.d
  • Uses the concept of ‘runlevels’ to layer functionality.
  • Default ‘runlevel’ defined in /etc/inittab.
    • 0 - Shutdown with Power Off
    • 1 - (alternative S, or s) - Single User mode.
    • 6 - Shutdown with Immediate Reboot
    • runlevels 2, 3, 4 & 5 vary depending on Distribution and Version.
    • Level 5 often used for application and desktop services.

Upstart

  • Started by Ubuntu as “event-based replacement for init”
  • Actively developed between 2009 and 2014.
  • Written by Scott James Remnant, formerly of Canonical, Ltd.
  • Benefits over SystemV Init:
  • Developed in an asynchronous, event-oriented manner, capable of managing start/stop operations in parallel. Some of the events it responds to:
    • Userland agents to start, stop, restart, reload services
    • Insertion or removal of hot-pluggable devices, including USB, SATA and Network devices.
    • Death of managed services.
  • It can run the old-style init scripts.
  • Configuration held in /etc/init (notice there is no ‘.d’ suffix on this directory).
  • /etc/init is populated by specifications on how to manage each process.
  • It was adopted by Ubuntu from 9.10 to 14.10 - Ubuntu 15.04+ is systemd-based now.
  • It was adopted by Fedora from 9 to 14 - Fedora went with systemd starting in release 15.
  • It was adopted by Redhat in RHEL6, and by extension CentOS, Oracle Linux and Amazon Linux. RHEL and its derivatives also went went systemd starting in RHEL 7.x.

Systemd Init system

  • Newest, most widely deployed modern init system as of 2018.
  • Started in 2009, by Lennart Poettering, of RedHat.
  • Achieves most of the benefits that upstart has over SystemV init, but includes more sweeping changes, such as robust dependency management, hotplugging of devices, cron-like and timer-based job scheduling, handling of automounts, tmpfiles, logins,
  • Replaces the concept of ‘runlevels’ with ‘unit files`.
  • Configuration is held in /etc/systemd and also /lib/systemd
  • Vastly larger footprint and solution domain than systemv or upstart.
  • With 69 included binaries, it was criticized for creeping featurism, trying to solve too many problem and abandoning 40 years of Unix wisdom & tradition. It was even compared to svchost.exe, which is responsible for dozens of tasks within Windows.
  • Systemd eventually won over most current Linux distributions, with improved stability, and a scope that has mostly stopped growing.
  • It will probably dominate the Linux world for years to come.

Which one do I have?

Only your package manager knows for sure…

Determining your init system

The one thing all init systems have in common is the /sbin/init binary. Use your package manager to find which package your /sbin/init belongs to.

All I thought I need to know, but was wrong…

SystemV Init Systemd
service httpd start systemctl start httpd
service httpd stop systemctl stop httpd
service httpd status systemctl status httpd
service httpd restart systemctl restart httpd
service httpd reload systemctl reload httpd

I happened to remember that systemd distros provide a /usr/sbin/service compatibility shim. I made the mistake trying to use it in my EC2 boot script.

Screenshot from an Ubuntu VM.

I was working on setting up Apache on a custom-baked Amazon Linux AMI.

This customized AMI had a highly customized boot script.

It periodically polls AWS Parameters Store for list of customers, and generates 4 Apache configurations for each, reloading Apache when necessary. All was fine, until under load testing, apache would receive a SIGTERM.
(i.e. not a crash).

What I didn’t know…

  • My Amazon Linux AMI, despite having a release date of 2017.09, was actually based on CentOS 6, and therefore had upstart. (not SystemV Init or Systemd)
  • Upstart didn’t provide a compatibility shim for the service command. The service command was actually for SystemV init.

  • One of the features of Upstart is to steer the kernel’s OOM killer away from Upstart-managed jobs, by giving it a different way to compute a custom “oom score”. http://upstart.ubuntu.com/cookbook/#oom-score
  • I was starting httpd via the old systemv “service” binary, which provided httpd no protection against OOM purges. This also meant that upstart wasn’t going to restart httpd for any reason. But upstart gladly restarted several things that are worthless in an EC2 environment like ‘serial’, ‘tty’, and ‘splash-manager’, because there were files called /etc/init/serial, /etc/init/tty, and /etc/init/splash-manager.

  • Upstart didn’t provide a compatibility shim for /sbin/service.
  • Therefore I was running httpd outside of upstart.

So, when OOM struck, no more Apache!

Everything I needed to know…

SystemV Init Upstart Systemd
service httpd start start httpd systemctl start httpd
service httpd stop stop httpd systemctl stop httpd
service httpd status status httpd systemctl status httpd
service httpd restart restart httpd systemctl restart httpd
service httpd reload reload httpd systemctl reload httpd

Plus, I needed to tell upstart about httpd.

And one commit later,

upstart is now working in my favor!

References