From power button to shell

Why this matters

A few seconds pass between pressing the power button and the login prompt, and in that time control passes through four owners in turn. As long as everything works, you will never need to know this chain.

The need comes when something breaks. The system won’t boot after a kernel update. You get emergency mode instead of a desktop. A service starts fine by hand but not at boot. Each of these maps to a specific link in the chain, so half of the diagnosis is figuring out which link the system reached at all.

Prerequisites. Execution modes (module 2), PID 1 and the process tree (module 6).

The chain

Each link knows exactly enough to find and start the next one. A problem at any step stops everything after it, so diagnosis starts with the question of which step the system reached.

It all starts with the firmware. The CPU fetches its first instruction from a fixed address, and that is where the firmware sits. It checks and initializes memory, buses and controllers, then looks for what to boot next.

Modern firmware is UEFI. It understands the FAT32 file system and reads the bootloader as an ordinary file from a separate partition called the ESP (EFI System Partition). The list of boot entries lives in the motherboard’s non-volatile memory, and you can edit it from the running system with efibootmgr.

The old BIOS + MBR scheme works completely differently: the BIOS reads the first 512 bytes of the disk and executes them. Those 512 bytes also hold the partition table, which leaves so little room for code that the bootloader had to be split into stages. The limits of MBR come from the same place: four primary partitions and 2 TiB, both of which GPT removes.

The bootloader, GRUB or systemd-boot, shows a menu, finds the kernel image and the initramfs, and passes the kernel its command-line parameters. That command line decides what the kernel does next: where to look for the root, whether to turn on debug output, which drivers to disable.

The kernel decompresses itself, initializes its subsystems, discovers the CPUs and memory, starts drivers and mounts the initramfs.

Why initramfs exists

The kernel has to mount the root file system. But getting to it may need a controller driver, an LVM module, RAID assembly or partition decryption, and all of that lives on the very root that isn’t there yet.

The initramfs breaks the circle: a small file system in memory with a minimal set of drivers and utilities. It mounts the real root, switches to it and disappears.

That is why updating the kernel without rebuilding the initramfs is a classic way to get an unbootable system. The new kernel is in place, but the modules needed to reach the disk are not in the image.

The last thing the kernel does is start the first user-mode process, PID 1. It is special in several ways at once: an ordinary signal can’t kill it, it adopts every orphan process (module 6), and if it exits, the kernel halts the system.

systemd

The classic init ran scripts one after another by number. Simple, but slow: every service waited for the previous one, even when the two had nothing to do with each other.

systemd instead describes the system as a dependency graph and starts in parallel everything that can run in parallel.

The thing it manages is called a unit. There are several types: .service for a process, .socket for a socket that starts a service when someone connects to it, .timer in place of cron, .mount for a mount point and .target for a group of units.

Think of a target not as a runlevel but as a point in the graph. multi-user.target means the system is ready to work at the console, and graphical.target adds a graphical environment on top.

Socket activation deserves its own mention. systemd opens the socket right away and starts the service only on the first connection. The client sees no difference, because the connection is accepted and waits, but boot is no longer held up by services nobody uses.

Environment variables at boot

If a service starts by hand but fails under systemd, the cause is almost always the environment. The unit doesn’t have your PATH, HOME or anything from .bashrc, its working directory is different, and stdout goes to journald instead of a terminal. Check this first.

How it actually works in Linux

[ -d /sys/firmware/efi ] && echo 'UEFI' || echo 'BIOS (legacy)'

Whether the system booted through UEFI. The directory exists only in UEFI mode.

efibootmgr -v 2>/dev/null | head; lsblk -o NAME,PARTTYPENAME,MOUNTPOINT | head

The boot entries in the motherboard’s memory and the ESP partition, usually mounted at /boot/efi.

cat /proc/cmdline

The parameters the running kernel was booted with, that is, what the bootloader passed. This shows where the root came from and which options apply.

lsinitramfs /boot/initrd.img 2>/dev/null | head -20 || lsinitrd 2>/dev/null | head -20

The contents of the initramfs: the set of modules needed to reach the root.

This deliberately uses /boot/initrd.img, a symbolic link to the image of the default kernel. If you use /boot/initrd.img-$(uname -r), the command fails exactly when things get interesting: after a kernel update, when the new image already exists but the system is still running the old kernel, which may have no image. It’s worth comparing both.

systemd-analyze; systemd-analyze blame | head -10

How long boot took, broken down into firmware, bootloader, kernel and user space, and which units took the most time.

systemd-analyze critical-chain

The critical chain: the sequence that actually determined boot time. Only what is in it is worth optimizing.

systemctl list-units --type=target --state=active
systemctl get-default

Active targets and the default target.

journalctl -b -p err --no-pager | tail -10

Errors from the current boot. -b -1 shows the previous one, which is invaluable when the system crashed and managed to reboot.

Common misconceptions

“BIOS and UEFI are the same thing, just a new name.” They are different schemes. The BIOS executes the first 512 bytes of the disk, while UEFI reads an .efi file from a FAT32 partition and keeps its list of entries in the motherboard’s memory.

“initramfs is there for speed.” It is there to break a circle: mounting the root needs drivers that live on the root itself.

“systemd is just init.” It is a service manager with a dependency graph, socket activation, timers, mount points and a journal. The sheer amount it took on is exactly why it is controversial.

“A target is a runlevel.” Targets aren’t ordered by number and don’t exclude each other. They are just synchronization points in the graph.

“The service works by hand, so the unit is correct.” The unit has a different environment: none of your variables, a different working directory, stdout going to journald. The mismatch is usually there.

“The system won’t boot, I’ll reinstall.” First look at journalctl -b -1 -p err and the parameters in /proc/cmdline. Two things are to blame most often: an initramfs not rebuilt after a kernel update and a wrong root UUID in /etc/fstab.

Check yourself

1. Why is initramfs needed?

To speed up bootTo break a circle: the drivers needed to reach the root live on the root itselfTo store bootloader settingsTo start the graphical environment

2. What is the fundamental difference between UEFI and BIOS?

UEFI has a graphical interfaceUEFI reads the bootloader as a file from a FAT32 partition (the ESP), while BIOS executes the first 512 bytes of the diskUEFI is fasterUEFI does not need a bootloader

3. What happens if the process with PID 1 exits?

The kernel restarts itThe kernel halts the system with a panicThe system is put into emergency modeNothing, the kernel adopts it

4. A service starts by hand but fails when started by systemd. Where do you start?

Reinstall systemdCompare the environments: the unit has no PATH or HOME of yours, a different working directory, and stdout goes to journaldChange the default targetRebuild the initramfs

5. What does socket activation give you in systemd?

Encrypted connectionsThe socket is opened immediately and the service starts on the first connection, so boot does not wait for unused servicesLoad balancing between servicesAutomatic restart after a crash

6. Where can you see the parameters the running kernel was booted with?

/etc/fstab/proc/cmdline/boot/grub/grub.cfgdmesg | head

Lab

B1: installing the system. GPT partitioning, an ESP partition, booting through UEFI and your own .service unit with dependencies. The mandatory part is to break the boot on purpose and fix it from emergency mode.

Sources

• Nemeth et al., UNIX and Linux System Administration Handbook, chapter 2
• man 7 boot, man 1 systemd, man 5 systemd.unit, man 5 systemd.service
• systemd for Administrators: a series by the author
• UEFI Specification: if you need the format details