overlayroot (and permission issues) on Linux Mint 18.1
Introduction
Wanting my media center computer to maintain consistent behavior and be tolerant to sudden power outages, I looked for a way to have the root filesystem read-only. This can cause problems with programs that expect to write to all kind of files allover, and media related software has this tendency to break when things divert from exactly as expected. So overlayroot is an elegant solution, as the system behaves completely as usual, only the changes are stored in RAM, and vanish on the next boot.
Only risk: Working hard on some changes, just to discover that the system was in overlay mode, and all went poof.
My system is a Linux Mint 18.1 (”Serena”) which is a derivative of Ubuntu 16.04 (”Xenial”). It runs a 4.4.0-53-generic kernel.
Starting off
Following this excellent post:
$ sudo apt-get install overlayroot
Note that the installation generates a new initramfs in /boot with the update-initramfs command-line utility, but comparing with the old one I couldn’t find anything related to overlayroot.
Overlayroot isn’t active by default. It’s possible to edit /etc/overlayroot.conf, but since I’m planning to switch back and forth with GRUB, I might as well enable it with the Linux kernel command rather than disabling it with the same.
In order to create a GRUB2 menu entry, copy /etc/grub.d/10_linux into /etc/grub.d/15_linux_overlay, and change:
os="$1"
into
os="RAM OVERLAY $1"
but even more important, add the following line somewhere at the beginning of the file:
GRUB_CMDLINE_LINUX_DEFAULT="overlayroot=\"tmpfs:swap=0,recurse=0\" $GRUB_CMDLINE_LINUX_DEFAULT"
and run update-grub. Let’s break it down:
- overlayroot=tmpfs: obviously means use tmpfs for overlaying the root
- swap=0: Disable swap. This is the default, but why rely on that
- recurse=0: Don’t overlay any of root’s subdirectories, if mounted separately. /boot/, in my case. No reason to overlay it, as I’ve set it to mount read-only anyhow
By the way, there are empty “custom” entries in /etc/grub.d/, but since I want them exactly like the original, only with a couple of changes, copying those long scripts is more accurate.
To make this the default boot, change the GRUB_DEFAULT variable in /etc/default/grub to point to the desired index, or move 15_linux_overlay to say, 09_linux_overlay. And run update-grub, or course.
Sometimes it’s required to have SHIFT pressed when the computer boots to get the GRUB menu up, and alternative kernels appear under “Advanced options” (it wasn’t this case, but just to have this written down). And if I’m at it, on an older system, update-grub doesn’t make any difference, but
# grub-mkconfig > /boot/grub/grub.cfg
does.
But then it didn’t work
When booting Linux with overlayroot enabled, the Avahi daemon failed to start. And indeed, trying to kick it off manually (as root):
# avahi-daemon -s avahi-daemon: error while loading shared libraries: libavahi-common.so.3: cannot stat shared object: Permission denied
Sounds weird. Let’s look at the strace of the same command:
execve("/usr/sbin/avahi-daemon", ["avahi-daemon", "-s"], [/* 30 vars */]) = 0
brk(NULL) = 0x20a8000
[ ... yada yada ... ]
open("/usr/lib/x86_64-linux-gnu/libavahi-common.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@/\0\0\0\0\0\0"..., 832) = 832
fstat(3, 0x7fff3f2caaa0) = -1 EACCES (Permission denied)
close(3) = 0
writev(2, [{"avahi-daemon", 12}, {": ", 2}, {"error while loading shared libra"..., 36}, {": ", 2}, {"libavahi-common.so.3", 20}, {": ", 2}, {"cannot stat shared object", 25}, {": ", 2}, {"Permission denied", 17}, {"\n", 1}], 10) = 119
exit_group(127) = ?
+++ exited with 127 +++
WHATWHATWHAT? It’s fine to read from the file, but fstat() is denied?
And there was a similar problem with ping:
$ ping 1.1.1.1 ping: error while loading shared libraries: libcap.so.2: cannot stat shared object: Permission denied
Hint: If I moved /bin/ping to /bin/ping2, and ran ping2 instead, it ran normally.
Let’s talk about permissions
I know four types of permissions systems:
- Good old POSIX permissions
- Access Control List (ACL, Extended Attributes, xattrs)
- SELinux
- Apparmor (spoiler: This was the problem)
Since I issued the command as root, POSIX permissions won’t stop me. SELinux isn’t installed at all on Linux Mint 18.1, so maybe ACL?
$ cd /usr/lib/x86_64-linux-gnu $ ls -l libavahi-common.so.* lrwxrwxrwx 1 root root 24 Feb 3 13:00 libavahi-common.so.3 -> libavahi-common.so.3.5.3 -rw-r--r-- 1 root root 47952 Nov 24 2015 libavahi-common.so.3.5.3 $ getfacl libavahi-common.so.3 # file: libavahi-common.so.3 # owner: root # group: root user::rw- group::r-- other::r-- $ getfacl libavahi-common.so.3.5.3 # file: libavahi-common.so.3.5.3 # owner: root # group: root user::rw- group::r-- other::r--
Nope, nothing fishy about these files.
And this brings me back to the major hint from above: Changing the name of the executable made a difference. What functionality is sensitive to the executable’s path? Apparmor.
This gave me a major déjà vu from SELinux: That thing that’s supposed to make your system secure, but also causes mysterious failures until you’ve had enough of it, and go
$ sudo update-rc.d apparmor remove
Reboot, and then all was fine again. I have a feeling that many others will use this same command.
What it looks like
With overlayroot up and running, I get
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 0 3.9G 0% /dev
tmpfs 784M 9.5M 774M 2% /run
/dev/mapper/vg_ssd2-lv_root 118G 6.5G 106G 6% /media/root-ro
tmpfs-root 3.9G 3.0M 3.9G 1% /media/root-rw
overlayroot 3.9G 3.0M 3.9G 1% /
tmpfs 3.9G 772K 3.9G 1% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sda1 239M 99M 123M 45% /boot
cgmfs 100K 0 100K 0% /run/cgmanager/fs
tmpfs 784M 16K 784M 1% /run/user/1000
and
$ mount sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) udev on /dev type devtmpfs (rw,nosuid,relatime,size=3987932k,nr_inodes=996983,mode=755) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=801940k,mode=755) /dev/mapper/vg_ssd2-lv_root on /media/root-ro type ext4 (ro,relatime,data=ordered) tmpfs-root on /media/root-rw type tmpfs (rw,relatime) overlayroot on / type overlay (rw,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) [...etc...]
So clearly, my SSD was mounted read-only, and used as “lowerdir”. The overlay directory is at /media/root-rw/overlay/, and the working directory (where files are prepared before going to the overlay, in order to ensure atomicity) is /media/root-rw/overlay-workdir/.
Preventing confusion
Trying to avoid confusion on whether the system is volatile or not, I added this script to the “Startup Applications” on my Cinnamon desktop (Menu > Preferences > Startup Applications):
#!/bin/bash
# Set a dedicated background for volatile (overlay) sessions
if grep -q overlayroot /proc/cmdline ; then
/usr/bin/gsettings set org.cinnamon.desktop.background picture-uri \
'file:///usr/local/share/wallpaper.jpg'
I put the script in /usr/local/bin/overlay-wallpaper, and the wallpaper as /usr/local/share/wallpaper.jpg. Since this script runs only when the system is volatile, this setting is gone when the system reboots. More about setting up Gnome attributes in this post.
FWIW, adding this script as a startup application resulted in a file named “.config/autostart/Special wallpaper for overlay session.desktop” which reads:
[Desktop Entry] Type=Application Exec=/usr/local/bin/overlay-wallpaper X-GNOME-Autostart-enabled=true NoDisplay=false Hidden=false Name[en_US]=Overlay wallpaper Comment[en_US]=Special wallpaper for overlay session X-GNOME-Autostart-Delay=0
Notes
- overlayroot is a great way to tell what files are changed while the system is running. Just
$ ls -RC on /media/root-rw/overlay/
tells the whole story.
- The size of tmpfs (and hence the overlay) is half the physical RAM by default. On my machine, that’s almost 4 GB, which is probably more than ever necessary. But if tmpfs becomes full, the computer will deadlock, as stated in the kernel’s documentation. Meaning: A simple program that decides to write a huge file can freeze the system completely. If that’s a problem, use some disk partition as the lowerdir, and wipe it clean (make2fs) on each boot before mounting.
- If you want to save something permanently, it’s possible to remount the underlying root directory in read-write mode,
$ sudo mount -o remount,rw /media/root-ro
Note that this doesn’t disable the overlay — changes are not saved. To have something written permanently, change or create files under /media/root-ro/.
This may or may not have an immediate effect on the currently seen root directory, depending on whether it has an entry in the overlay directory. So it’s useful mostly for minor fixes or to save something specific. - Every change in the file causes a copy into the overlay directory, including changes in permissions etc. If the file is deleted, a “whiteout” file is created in the respective position in the overlay, marking that it’s absent. Something like this (after deleting eli.tar.gz from my Desktop):
$ cd /media/root-rw/overlay/home/eli/Desktop/ $ ls -l total 0 c--------- 1 root root 0, 0 Feb 21 13:48 eli.tar.gz
- Judging by the vast amount of patches under fs/overlayfs/ in the kernel tree since v4.4 (to say, v4.10), the whole thing seems a bit shaky. On the other hand, who cares, if it messes up just pull the plug and reboot. And frankly, it works great.