systemd random jots

This post was written by eli on December 1, 2017
Posted Under: Linux,systemd

As systemd seems to be here to stay (or at least I hope so), this is a post of random notes to self that I jot down as I explore it. It will probably grow with time, and become a mixture of basic issues and rather advanced stuff.

Useful references

  • man systemd.service and man systemd.unit as well as others. Really. These are the best sources, it turns out.
  • The excellent Systemd for Admins series (with several relevant and specific topics).
  • The primer for systemd: Basic concepts explained.
  • Red Hat’s guide to creating custom targets (and daemons)
  • The FAQ (with actually useful info!)
  • On the Network Target (and how to run a target only when the network is up)
  • man systemd.special for a list of built-in targets, their meaning and recommended use

systemctl is the name of the game

Forget “service”, “telinit” and “initctl”. “systemctl” is the new swiss knife for starting, stopping, enabling and disabling services, as well as obtaining information on how services are doing. And it’s really useful.

To get an idea on what runs on the system and what unit triggered it off, go

# systemctl status

Note that “systemd status” lists, among others, all processes initiated by each login session for each user. Which is an extremely useful variant of “ps”.

And just a list of all services

# systemctl

Ask about a specific service:

# systemctl status ssh
 ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2017-12-01 10:37:21 IST; 1h 17min ago
 Main PID: 1018 (sshd)
   CGroup: /system.slice/ssh.service
           └─1018 /usr/sbin/sshd -D

Dec 01 12:26:26 machine sshd[2841]: Accepted publickey for eli from 192.168.1.12 port 45220 ssh2: RSA SHA256:xxx
Dec 01 12:26:26 machine sshd[2841]: pam_unix(sshd:session): session opened for user eli by (uid=0)

Really, isn’t it sweet?

Turning off a service: Find it with the “systemctl status” command above (or just “systemctl”), and then (this is an example of a service not found in the status, because it’s an LSB service);

# systemctl disable tvheadend
tvheadend.service is not a native service, redirecting to systemd-sysv-install
Executing /lib/systemd/systemd-sysv-install disable tvheadend
insserv: warning: current start runlevel(s) (empty) of script `tvheadend' overrides LSB defaults (2 3 4 5).
insserv: warning: current stop runlevel(s) (0 1 2 3 4 5 6) of script `tvheadend' overrides LSB defaults (0 1 6).

And “enable” for enabling a service.

Needless to say, services can be started, stopped and restarted with “systemctl X service” where X is either start, stop or restart.

For a list of all services (including disactivated):

$ systemctl --all

No more fishing in /var/log/syslog

/var/log/syslog is still there, but forget about it: journalctl is the way read logs. And it doesn’t require root privileges, which is reason enough.

To get the log message since the current boot:

$ journalctl -b

(that alone justifies using the utility).

There’s also the -u flag to see the logs from a specific (systemd) unit (systemctl status gives that as well), -g for grep, and the -f (follow) flag as in tail -f.

What makes a systemd service run (on boot)

  • It’s enabled, which means that there’s a symbolic link from some /etc/systemd/*/*.wants directory to the unit file. In the example below, it’s a .path file, so it activates a watch on the path specified, but if it’s a service, it’s kicked off at boot
    # systemctl enable foo.path
        Created symlink from /etc/systemd/system/paths.target.wants/foo.path to /etc/systemd/system/foo.path
  • In the unit file symlinked to, there’s an [Install] section, which says when it should be kicked off with the WantedBy directive. More precisely, which target or service should be active. Once again, for a plain .service unit file, this kicks off the service, and for an e.g. .path file, this starts the monitoring. man systemd.special for a list of built in targets, and targets can be generated, of course. The most common target for “run me at boot” is multi-user.target. Dependency on services is expressed by using the service name with a .service suffix instead.
  • By default, unit files such as .path files kick off the a .service file with the same non-suffix part (this can be changed with a directive in the file. But why?)

See /etc/systemd/system/multi-user.target.wants for a list of services that are activated on boot. In particular note that not all are symlinks to .service unit files.

General memo jots

  • Always run the systemctl daemon-reload command after creating new unit files or modifying existing unit files. Otherwise, the systemctl start or systemctl enable commands could fail due to a mismatch between states of systemd and actual service unit files on disk.
  • Services are best run in the foreground. Unlike classic UNIX services, there’s no point in daemonizing. All processes belonging to the relevant service are enclosed in a Cgroup anyhow, and systemd handles the daemonizing for Type=simple services. In a clean and uniform manner.
  • Unit files’ suffix indicate their type. When the non-suffix part of two files is the same, they indicate a functional relationship. For example, systemd-tmpfiles-clean.timer says when to launch systemd-tmpfiles-clean.service. Or that systemd-ask-password-console.path gives the path to be watched, and systemd-ask-password-console.service is the service to fire off.
  • After= doesn’t imply a dependency, and Requires= doesn’t imply the order of starting services. Both are needed if one service depends on the other running when it starts.
  • The Type= directive’s main influence is determining when the service is active, i.e. when other services that depend on it can be launched.
  • There’s also “loginctl” which lists the current users and their sessions

Where to find unit files

The configuration files are considered in the following order (later overrules earlier):

  • /lib/systemd/system — where installation scripts write to
  • /run/systemd/system — runtime files
  • /etc/systemd/system — per-system user preferences

Per-user files can be found in ~/.config/systemd/user and possibly also in ~/.local/share/systemd/user.

Enabling console autologin on tty1 and ttyPS0

Following this page and as explained on this page, add /etc/systemd/system/getty@tty1.service.d/autologin.conf (after creating the getty@tty1.service.d directory) as follows:

[Service]
ExecStart=
ExecStart=-/sbin/agetty -a root --noclear %I $TERM

Note that the filename autologin.conf has no significance. It’s suffix and the directory it resides in that matter.

The idea is to override the ExecStart parameter given in /lib/systemd/system/getty@.service template unit, which reads

ExecStart=-/sbin/agetty --noclear %I $TERM

but otherwise have it running the same. The reason for two ExecStart lines is that the empty assignment clears the existing assignment (or otherwise it would have been added on top of it), and the second sets the command.

Note the %I substitute parameter, which stands for the instance of the current tty.

The leading dash means that the exit value of the command is ignored, and may be nonzero without the unit considered as failed (see man systemd.service).

This can’t be repeated with ttyPS0, because systemd goes another way for setting up the serial console: At an early stage in the boot, systemd-getty-generator automatically sets a target, serial-getty@ttyPS0.service, which is implemented by the /lib/systemd/system/serial-getty@.service template unit.

# systemctl status

[ ... ]

         │ ├─system-serial\x2dgetty.slice
           │ │ └─serial-getty@ttyPS0.service
           │ │   └─2337 /sbin/agetty --keep-baud 115200 38400 9600 ttyPS0 vt220

So the solution is adding /etc/systemd/system/serial-getty@ttyPS0.service.d/autologin.conf saying

[Service]
ExecStart=
ExecStart=-/sbin/agetty -a root --keep-baud 115200,38400,9600 %I $TERM

Disable renaming of Ethernet interfaces

Ditch those tedious Ethernet interface names (e.g. enp3s0, enp0s31f6, wlp2s0, really, come on), and bring back the good old eth0, eth1 etc. Note that the kernel still assigns the good old interface names. It’s systemd that renames them. So the trick is simple: Mask the default naming policy, which causes this to happen:

# ln -s /dev/null /etc/systemd/network/99-default.link

(try man systemd.link for more info, and there’s a lot — including how to change MAC address)

And update the initramfs:

# update-initramfs -u

It won’t work without updating the initramfs, because the network interface names are set way before the root filesystem is mounted. The files are copied into the initramfs’ /lib/systemd/network/ directory (note that it’s /lib, even though they’re saved in /etc on the root filesystem. Which makes sense, because on the initramfs there’s no point with the /lib vs. /etc distinction).

Also, renaming can’t be used to achieve persistent allocation of ethN names (so says the manpage of systemd.link) because of a race with the kernel’s name assignments. As of kernel v4.15, that is.

Resetting failed services

To get the system out of the “degraded” state due to an irrelevant failed service, for example:

● lvm2-pvscan@253:9.service  loaded failed     failed    LVM2 PV scan on device 253:9

Just go (as root)

# systemctl reset-failed

and the failed service disappears, returning the overall status to “running” again (assuming there’s no real problem).

Ditching NetworkManager

I can’t say I was very fond of it ever (not only because of the capital letters in its name), and systemd can now do its job. Odds are that NetworkManager will become history in a matter of a few years, so better give it the boot now:

# systemctl disable NetworkManager
# systemctl disable NetworkManager-wait-online

A word of warning: The GUI for handling Ethernet connection requires NetworkManager, so for better or for worse, it won’t work anymore.

I also ditched ModemManager, as I have a solution for my ADSL modem. But I’m not sure about others.

And enable systemd’s cutie instead

# systemctl enable systemd-networkd

You might want to get rid of the service that waits for networking to be “online”:

# systemctl mask systemd-networkd-wait-online.service

This service runs /lib/systemd/systemd-networkd-wait-online, which is supposed to wait for at least one Ethernet card being configured. In practice, it waited until its 120 seconds timeout, and then said it failed. As a result, some services that depend on the network.online were kicked off only after these two minutes, and the overall system’s status was marked as “degraded”.

What does “online” mean, and why is it important? Good question, discussed here. My conclusion: As any contemporary Linux system should be able to tolerate hotplugging of its NICs, there’s no need to wait. Handle them as they appear.

A simple definition file for a NIC can be, for example, /etc/systemd/network/20-eth0.network

[Match]
MACAddress=1c:1b:0d:45:0f:eb

[Network]
DHCP=yes

Note that I detected the card by its MAC address. It’s also possible to select it by its ifconfig name, and other methods. But this is safest.

DHCP is “no” by default. Replace it with a line saying e.g.

Address=10.10.10.10/16

for a static IP address. Go “man systemd.network” for the whole set of options. They cover basically everything needed.

After making changes to such file (or adding one), go

# systemctl restart systemd-networkd

to make them effective. There’s no need to update the initramfs. In fact, .network files aren’t copied into it (unlike .link files, as said above).

As for responding to hotplugging events of network devices, there’s this post.

Add a Comment

required, use real name
required, will not be published
optional, your blog address