systemd dependencies and processing /etc/fstab
The problem I wanted to solve
On an embedded arm system running LUbuntu 16.04 LTS (systemd 229), I wanted to add a mount point to the /etc/fstab file, which originally was like this:
# UNCONFIGURED FSTAB FOR BASE SYSTEM
Empty, except for this message. Is that a bad omen or what? This is the time to mention that the system was set up with debootstrap.
So I added this row to /etc/fstab:
#/dev/mmcblk0p1 /mnt/tfcard vfat defaults 0 0
The result: Boot failed, and systemd asked me for the root password for the sake of getting a rescue shell (the overall systemd status was “maintenance”, cutely enough).
Journalctl was kind enough to offer some not-so-helpful hints:
systemd[1]: dev-mmcblk0p1.device: Job dev-mmcblk0p1.device/start timed out. systemd[1]: Timed out waiting for device dev-mmcblk0p1.device. systemd[1]: Dependency failed for /mnt/tfcard. systemd[1]: Dependency failed for Local File Systems. systemd[1]: local-fs.target: Job local-fs.target/start failed with result 'dependency'. systemd[1]: local-fs.target: Triggering OnFailure= dependencies. systemd[1]: mnt-tfcard.mount: Job mnt-tfcard.mount/start failed with result 'dependency'. systemd[1]: dev-mmcblk0p1.device: Job dev-mmcblk0p1.device/start failed with result 'timeout'.
Changing the “defaults” option to “nofail” got the system up and running, but without the required mount. That said, running “mount -a” did successfully mount the requested partition.
Spoiler: I didn’t really find a way to solve this, but I offer a fairly digestible workaround at the end of this post.
This is nevertheless an opportunity to discuss systemd dependencies and how systemd relates with /etc/fstab.
The main takeaway: Use “nofail”
This is only slightly related to the main topic, but nevertheless the most important conclusion: Open your /etc/fstab, and add “nofail” as an option to all mount points, except those that really are crucial for booting the system. That includes “/home” if it’s on a separate file system, as well as those huge file systems for storing a lot of large files. The rationale is simple: If any of these file systems are corrupt during boot, you really don’t want to be stuck on that basic textual screen (possibly with very small fonts), not having an idea what happened, and fsck asking you if you want to fix this and that. It’s much easier to figure out the problem when the failing mount points simply don’t mount, and take it from there. Not to mention that it’s much less stressful when the system is up and running. Even if not fully so.
Citing “man systemd.mount”: “With nofail, this mount will be only wanted, not required, by local-fs.target or remote-fs.target. This means that the boot will continue even if this mount point is not mounted successfully”.
To do this properly, make a tarball of /run/systemd/generator. Keep an eye on what’s in local-fs.target.requires/ vs. local-fs.target.wants/ in this directory. Only the absolutely necessary mounts should be in local-fs.target.requires/. In my system there’s only one symlink for “-.mount”, which is the root filesystem. That’s it.
Regardless, it’s a good idea that the root filesystem has the errors=remount-ro option, so if it’s mountable albeit with errors, the system still goes up.
This brief discussion will be clearer after reading through this post. Also see “nofail and dependencies” below for an elaboration on how systemd treats the “nofail” option.
/etc/fstab and systemd
At an early stage of the boot process, systemd-fstab-generator reads through /etc/fstab, and generates *.mount systemd units in /run/systemd/generator/. Inside this directory, it also generates and populates a local-fs.target.requires/ directory as well local-fs.target.wants/ in order to reflect the necessity of these units for the boot process (more about dependencies below).
So for example, in response to this row in /etc/fstab,
/dev/mmcblk0p1 /mnt/tfcard vfat nofail 0 0
the automatically generated file can be found as /run/systemd/generator/mnt-tfcard.mount as follows:
# Automatically generated by systemd-fstab-generator [Unit] SourcePath=/etc/fstab Documentation=man:fstab(5) man:systemd-fstab-generator(8) [Mount] What=/dev/mmcblk0p1 Where=/mnt/tfcard Type=vfat Options=nofail
Note that the file name of a mount unit must match the mount point (as it does in this case). This is only relevant when writing mount units manually, of course.
Which brings me to my first attempt to solve the problem: Namely, to copy the automatically generated file into /etc/systemd/system/, and enable it as if I wrote it myself. And then remove the row in /etc/fstab.
More precisely, I created this file as /etc/systemd/system/mnt-tfcard.mount:
[Unit] Description=Mount /dev/mmcblk0p1 as /mnt/tfcard [Mount] What=/dev/mmcblk0p1 Where=/mnt/tfcard Type=vfat Options=nofail [Install] WantedBy=multi-user.target
Note that the WantedBy is necessary for enabling the unit. multi-user.target is a bit inaccurate for this purpose, but it failed for a different reason anyhow, so who cares. It should have been “WantedBy=local-fs.target”, I believe.
Anyhow, I then went:
# systemctl daemon-reload # systemctl enable mnt-tfcard.mount Created symlink from /etc/systemd/system/multi-user.target.wants/mnt-tfcard.mount to /etc/systemd/system/mnt-tfcard.mount.
And then ran
# systemctl start mnt-tfcard.mount
but that was just stuck for a few seconds, and then an error message. There was no problem booting (because of “nofail”), but the mount wasn’t performed.
To investigate further, I attached strace to the systemd process (PID 1) while running the “systemctl start” command, and there was no fork. In other words, I had already then a good reason to suspect that the problem wasn’t a failed mount attempt, but that systemd didn’t go as far as trying. Which isn’t so surprising, given that the original complaint was a dependency problem.
Also, the command
# systemctl start dev-mmcblk0p1.device
didn’t finish. Checking with “systemctl”, it just said:
UNIT LOAD ACTIVE SUB JOB DESCRIPTION
dev-mmcblk0p1.device loaded inactive dead start dev-mmcblk0p1.device
[ ... ]
So have we just learned? That it’s possible to create .mount units instead of populating /etc/fstab. And that the result is exactly the same. Why that would be useful, I don’t know.
See “man systemd-fstab-generator” about how the content of fstab is translated automatically to systemd units. Also see “man systemd-fsck@.service” regarding the service that runs fsck, and refer to “man systemd.mount” regarding mount units.
A quick recap on systemd dependencies
It’s important to make a distinction between dependencies that express requirements and dependencies that express the order of launching units (“temporal dependencies”).
I’ll start with the first sort: Those that express that if one unit is activated, other units need to be activated too (“pulled in” as it’s often referred to). Note that the requirement kind of dependencies don’t say anything about one unit waiting for another to complete its activation or anything of that sort. The whole point is to start a lot of units in parallel.
So first, we have Requires=. When it appears in a unit file, it means that the listed unit must be started if the current unit is started, and that if that listed unit fails, the current unit will fail as well.
In the opposite direction, there’s RequiredBy=. It’s exactly like putting a Required= directive in the unit that is listed.
Then we have another famous couple, Wants= and its opposite WantedBy=. These are the merciful counterparts of Requires= and RequriedBy=. The difference: If the listed unit fails, the other unit may activate successfully nevertheless. The failure is marked by the fact that “systemctl status” will declare the system as “degraded”.
So the fact that basically every service unit ends with “WantedBy=multi-user.target” means that the unit file says “if multi-user.target is a requested target (i.e. practically always), you need to activate me”, but at the same time it says “if I fail, don’t fail the entire boot process”. In other words, had RequriedBy= been used all over the place instead, it would have worked just the same, as long as all units started successfully. But if one of them didn’t, one would get a textual rescue shell instead of an almost fully functional (“degraded”) system.
There are a whole lot of other directives for defining dependencies. In particular, there are also Requisite=, ConsistsOf=, BindsTo=, PartOf=, Conflicts=, plus tons of possible directives for fine-tuning the behavior of the unit, including conditions for starting and whatnot. See “man systemd.unit”. There’s also a nice summary table on that man page.
It’s important to note that even if unit X requires unit Y by virtue of the directives mentioned above, systemd may (and often will) activate them at the same time. That’s true for both “require” and “want” kind of directives. In order to control the order of activation, we have After= and Before=. After= and Before= also ensure that the stopping of units occur in the reverse order.
After= and Before= only define when the units are started, and are often used in conjunction with the directives that define requirements. But by themselves, After= and Before= don’t define a relation of requirement between units.
It’s therefore common to use Required= and After= on the same listed unit, meaning that the listed unit must complete successfully before activating the unit for which these directives are given. Same with RequiredBy= and Before=, meaning that this unit must complete before the listed unit can start.
Once again, there are more commands, and more to say on those I just mentioned. See “man systemd.unit” for detailed info on this. Really, do. It’s a good man page. There’s a lot of “but if” to be aware of.
“nofail” and dependencies
One somewhat scary side-effect of adding “nofail” is that the line saying “Before=local-fs.target” doesn’t appear in the *.mount unit files that are automatically generated. Does it mean that the services might be started before these mounts have taken place?
I can’t say I’m 100% sure about this, but it would be really poor design if it was that way, as it would have have added a very unexpected side-effect to “nofail”. Looking at the system log of a boot with almost all mounts with “nofail”, it’s evident that they were mounted after “Reached target Local File Systems” is announced, but before “Reached target System Initialization”. Looking at a system log before adding these “nofail” options, all mounts were made before “Reached target Local File Systems”.
So what happens here? It’s worth mentioning that in local-fs.target, it only says After=local-fs-pre.target in relation to temporal dependencies. So that can explain why “Reached target Local File Systems” is announced before all mounts have been completed.
But then, local-fs.target is listed in sysinit.target’s After= as well as Wants= directives. I haven’t found any clear-cut definition to whether “After=” waits until all “wanted” units have been fully started (or officially failed). The term that is used in “man systemd.unit” in relation to waiting is “finished started up”, but what does that mean exactly? This manpage also says “Most importantly, for service units start-up is considered completed for the purpose of Before=/After= when all its configured start-up commands have been invoked and they either failed or reported start-up success”. But what about “want” relations?
So I guess systemd interprets the “After” directive on local-fs.target in sysinit.target as “wait for local-fs.target’s temporal dependencies as well, even if local-fs.target doesn’t wait for them”.
Querying dependencies
So how can we know which unit depends on which? In order to get the system’s tree of dependencies, go
$ systemctl list-dependencies
This recursively shows the dependency tree that is created by Requires, RequiredBy, Wants, WantedBy and similar relations. Usually, only target units are followed recursively. In order to display the entire tree, add the –all flag.
The units that are listed are those that are required (or “wanted”) in order to reach the target that is being queried (“default.target” if no target specified).
As can be seen from the output, the filesystem mounts are made in order to reach the local-fs.target target. To get the tree of only this target:
$ systemctl list-dependencies local-fs.target
The output of this command are the units that are activated for the sake of reaching local-fs.target. “-.mount” is the root mount, by the way.
It’s also possible to obtain the reverse dependencies with the –reverse flag. In other words, in order to see which targets rely on a local-fs.target, go
$ systemctl list-dependencies --reverse local-fs.target
list-dependencies doesn’t make a distinction between “required” or “wanted”, and neither does it show the nuances of the various other possibilities for creating dependencies. For this, use “systemctl show”, e.g.
$ systemctl show local-fs.target
This lists all directives, explicit and implicit, that have been made on this unit. A lot of information to go through, but that’s the full picture.
If the –after flag is added, the tree of targets with an After= directive (and other implicit temporal dependencies, e.g. mirrored Before= directives) is printed out. Recall that this only tells us something about the order of execution, and nothing about which unit requires which. For example:
$ systemctl list-dependencies --after local-fs.target
Likewise, there’s –before, which does the opposite.
Note that the names of the flags are confusing: –after tells us about the targets that are started before local-fs.target, and –before tells us about the targets started after local-fs.target. The rationale: The spirit of list-dependencies (without –reverse) is that it tells us what is needed to reach the target. Hence following the After= directives tells us what had to be before. And vice versa.
Red herring #1
After this long general discussion about dependencies, let’s go back to the original problem: The mnt-tfcard.mount unit failed with status “dependency” and dev-mmcblk0p1.device had the status inactive / dead.
Could this be a dependency thing?
# systemctl list-dependencies dev-mmcblk0p1.device dev-mmcblk0p2.device ● └─mnt-tfcard.mount # systemctl list-dependencies mnt-tfcard.mount mnt-tfcard.mount ● ├─dev-mmcblk0p1.device ● └─system.slice
Say what? dev-mmcblk0p2.device depends on mnt-tfcard.mount? And even more intriguing, mnt-tfcard.mount depends on dev-mmcblk0p2.device, so it’s a circular dependency! This must be the problem! (not)
This was the result regardless of whether I used /etc/fstab or my own mount unit file instead. I also tried adding DefaultDependencies=no on mnt-tfcard.mount’s unit file, but that made no difference.
Neither did this dependency go away when the last parameter in /dev/fstab was changed from 0 to 2, indicating that fsck should be run on the block device prior to mount. The dependency changed, though. What did change was mnt-tfcard.mount’s dependencies, but that’s not the problem.
# systemctl list-dependencies mnt-tfcard.mount mnt-tfcard.mount ● ├─dev-mmcblk0p1.device ● ├─system.slice ● └─systemd-fsck@dev-mmcblk0p1.service # systemctl list-dependencies systemd-fsck@dev-mmcblk0p1.service systemd-fsck@dev-mmcblk0p1.service ● ├─dev-mmcblk0p1.device ● ├─system-systemd\x2dfsck.slice ● └─systemd-fsckd.socket
But the thing is that the cyclic dependency isn’t an error. This is the same queries on the /boot mount (/dev/sda1, ext4) of another computer:
$ systemctl list-dependencies boot.mount boot.mount ● ├─-.mount ● ├─dev-disk-by\x2duuid-063f1689\x2d3729\x2d425e\x2d9319\x2dc815ccd8ecaf.device ● ├─system.slice ● └─systemd-fsck@dev-disk-by\x2duuid-063f1689\x2d3729\x2d425e\x2d9319\x2dc815ccd8ecaf.service
And the back dependency:
$ systemctl list-dependencies 'dev-disk-by\x2duuid-063f1689\x2d3729\x2d425e\x2d9319\x2dc815ccd8ecaf.device'
dev-disk-by\x2duuid-063f1689\x2d3729\x2d425e\x2d9319\x2dc815ccd8ecaf.device
● └─boot.mount
The mount unit depends on the device unit and vice versa. Same thing, but this time it works.
Why does the device unit depend on the mount unit, one may ask. Not clear.
Red herring #2
At some point, I thought that the reason for the problem was that the filesystem’s “dirty bit” was set:
# fsck /dev/mmcblk0p1 fsck from util-linux 2.27.1 fsck.fat 3.0.28 (2015-05-16) 0x25: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt. 1) Remove dirty bit 2) No action
By the way, using “fsck.vfat” instead of just “fsck” did exactly the same. The former is what the systemd-fsck service uses, according to the man page.
But this wasn’t the problem. Even after removing the dirty bit, and with fsck reporting success, dev-mmcblk0p1.device would not start.
The really stinking fish
What really is fishy on the system is this output of “systemctl –all”:
dev-mmcblk0p2.device loaded activating tentative /dev/mmcblk0p2
/dev/mmcblk0p2 is the partition that is mounted as root! It should be “loaded active plugged”! All devices have that status, except for this one.
Lennart Poettering (i.e. the horse’s mouth) mentions that “If the device stays around in “tentative” state, then this indicates that a device appears in /proc/self/mountinfo with some name, and systemd can’t find a matching device in /sys for it, probably because for some reason it has a different name”.
And indeed, this is the relevant row in /proc/self/mountinfo:
16 0 179:2 / / rw,relatime shared:1 - ext4 /dev/root rw,data=ordered
So the root partition appears as /dev/root. This device file doesn’t even exist. The real one is /dev/mmcblk0p2, and it’s mentioned in the kernel’s command line. On this platform, Linux boots without any initrd image. The kernel mounts root by itself, and takes it from there.
As Lennart points out in a different place, /dev/root is a special creature that is made up in relation to mounting a root filesystem by the kernel.
At this point, I realized that I’m up against a quirk that has probably been solved silently during the six years since the relevant distribution was released. In other words, no point wasting time looking for the root cause. So this calls for…
The workaround
With systemd around, writing a service is a piece of cake. So I wrote a trivial service which runs mount when it’s started and umount when it’s stopped. Not a masterpiece in terms of software engineering, but it gets the job done without polluting too much.
The service file, /etc/systemd/system/mount-card.service is as follows:
[Unit] Description=TF card automatic mount [Service] ExecStart=/bin/mount /dev/mmcblk0p1 /mnt/tfcard/ ExecStop=/bin/umount /mnt/tfcard/ Type=simple RemainAfterExit=yes [Install] WantedBy=local-fs.target
Activating the service:
# systemctl daemon-reload # systemctl enable mount-card Created symlink from /etc/systemd/system/local-fs.target.wants/mount-card.service to /etc/systemd/system/mount-card.service.
And reboot.
The purpose of “RemainAfterExit=yes” is to make systemd consider the service active even after the command exits. Without this row, the command for ExecStop runs immediately after ExecStart, so the device is unmounted immediately after it has been mounted. Setting Type to oneshot doesn’t solve this issue, by the way. The only difference between “oneshot” and “simple” is that oneshot delays the execution of other units until it has exited.
There is no need to add an explicit dependency on the existence of the root filesystem, because all services have an implicit Required= and After= dependency on sysinit.target, which in turn depends on local-fs.target. This also ensures that the service is stopped before an attempt to remount root as read-only during shutdown.
As for choosing local-fs.target for the WantedBy, it’s somewhat questionable, because no service can run until sysinit.target has been reached, and the latter depends on local-fs.target, as just mentioned. More precisely (citing “man systemd.service”), “service units will have dependencies of type Requires= and After= on sysinit.target, a dependency of type After= on basic.target”. So this ensures that the service is started after the Basic Target has been reached and shuts down before this target is shut down.
That said, setting WantedBy this way is a more accurate expression of the purpose of this service. Either way, the result is that the unit is marked for activation on every boot.
I don’t know if this is the place to write “all well that ends well”. I usually try to avoid workarounds. But in this case it was really not worth the effort to insist on a clean solution. Not sure if it’s actually possible.