overlayroot (and permission issues) on Linux Mint 18.1

This post was written by eli on February 21, 2017
Posted Under: Linux,Software


Wanting my media center computer to maintain consistent behavior and be tolerant to sudden power outages, I looked for a way to have the root filesystem read-only. This can cause problems with programs that expect to write to all kind of files allover, and media related software has this tendency to break when things divert from exactly as expected. So overlayroot is an elegant solution, as the system behaves completely as usual, only the changes are stored in RAM, and vanish on the next boot.

Only risk: Working hard on some changes, just to discover that the system was in overlay mode, and all went poof.

My system is a Linux Mint 18.1 (”Serena”) which is a derivative of Ubuntu 16.04 (”Xenial”). It runs a 4.4.0-53-generic kernel.

Starting off

Following this excellent post:

$ sudo apt-get install overlayroot

Note that the installation generates a new initramfs in /boot with the update-initramfs command-line utility, but comparing with the old one I couldn’t find anything related to overlayroot.

Overlayroot isn’t active by default. It’s possible to edit /etc/overlayroot.conf, but since I’m planning to switch back and forth with GRUB, I might as well enable it with the Linux kernel command rather than disabling it with the same.

In order to create a GRUB2 menu entry, copy /etc/grub.d/10_linux into /etc/grub.d/15_linux_overlay, and change:




but even more important, add the following line somewhere at the beginning of the file:

GRUB_CMDLINE_LINUX_DEFAULT="overlayroot=\"tmpfs:swap=0,recurse=0\" $GRUB_CMDLINE_LINUX_DEFAULT"

and run update-grub. Let’s break it down:

  • overlayroot=tmpfs:  obviously means use tmpfs for overlaying the root
  • swap=0: Disable swap. This is the default, but why rely on that
  • recurse=0: Don’t overlay any of root’s subdirectories, if mounted separately. /boot/, in my case. No reason to overlay it, as I’ve set it to mount read-only anyhow

By the way, there are empty “custom” entries in /etc/grub.d/, but since I want them exactly like the original, only with a couple of changes, copying those long scripts is more accurate.

To make this the default boot, change the GRUB_DEFAULT variable in /etc/default/grub to point to the desired index, or move 15_linux_overlay to say, 09_linux_overlay. And run update-grub, or course.

But then it didn’t work

When booting Linux with overlayroot enabled, the Avahi daemon failed to start. And indeed, trying to kick it off manually (as root):

# avahi-daemon -s
avahi-daemon: error while loading shared libraries: libavahi-common.so.3: cannot stat shared object: Permission denied

Sounds weird. Let’s look at the strace of the same command:

execve("/usr/sbin/avahi-daemon", ["avahi-daemon", "-s"], [/* 30 vars */]) = 0
brk(NULL)                               = 0x20a8000

[ ... yada yada ... ]

open("/usr/lib/x86_64-linux-gnu/libavahi-common.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@/\0\0\0\0\0\0"..., 832) = 832
fstat(3, 0x7fff3f2caaa0)                = -1 EACCES (Permission denied)
close(3)                                = 0
writev(2, [{"avahi-daemon", 12}, {": ", 2}, {"error while loading shared libra"..., 36}, {": ", 2}, {"libavahi-common.so.3", 20}, {": ", 2}, {"cannot stat shared object", 25}, {": ", 2}, {"Permission denied", 17}, {"\n", 1}], 10) = 119
exit_group(127)                         = ?
+++ exited with 127 +++

WHATWHATWHAT? It’s fine to read from the file, but fstat() is denied?

And there was a similar problem with ping:

$ ping
ping: error while loading shared libraries: libcap.so.2: cannot stat shared object: Permission denied

Hint: If I moved /bin/ping to /bin/ping2, and ran ping2 instead, it ran normally.

Let’s talk about permissions

I know four types of permissions systems:

Since I issued the command as root, POSIX permissions won’t stop me. SELinux isn’t installed at all on Linux Mint 18.1, so maybe ACL?

$ cd /usr/lib/x86_64-linux-gnu
$ ls -l libavahi-common.so.*
lrwxrwxrwx 1 root root    24 Feb  3 13:00 libavahi-common.so.3 -> libavahi-common.so.3.5.3
-rw-r--r-- 1 root root 47952 Nov 24  2015 libavahi-common.so.3.5.3

$ getfacl libavahi-common.so.3
# file: libavahi-common.so.3
# owner: root
# group: root

$ getfacl libavahi-common.so.3.5.3
# file: libavahi-common.so.3.5.3
# owner: root
# group: root

Nope, nothing fishy about these files.

And this brings me back to the major hint from above: Changing the name of the executable made a difference. What functionality is sensitive to the executable’s path? Apparmor.

This gave me a major déjà vu from SELinux: That thing that’s supposed to make your system secure, but also causes mysterious failures until you’ve had enough of it, and go

$ sudo update-rc.d apparmor remove

Reboot, and then all was fine again. I have a feeling that many others will use this same command.

What it looks like

With overlayroot up and running, I get

$ df -h
Filesystem                   Size  Used Avail Use% Mounted on
udev                         3.9G     0  3.9G   0% /dev
tmpfs                        784M  9.5M  774M   2% /run
/dev/mapper/vg_ssd2-lv_root  118G  6.5G  106G   6% /media/root-ro
tmpfs-root                   3.9G  3.0M  3.9G   1% /media/root-rw
overlayroot                  3.9G  3.0M  3.9G   1% /
tmpfs                        3.9G  772K  3.9G   1% /dev/shm
tmpfs                        5.0M  4.0K  5.0M   1% /run/lock
tmpfs                        3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1                    239M   99M  123M  45% /boot
cgmfs                        100K     0  100K   0% /run/cgmanager/fs
tmpfs                        784M   16K  784M   1% /run/user/1000


$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=3987932k,nr_inodes=996983,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=801940k,mode=755)
/dev/mapper/vg_ssd2-lv_root on /media/root-ro type ext4 (ro,relatime,data=ordered)
tmpfs-root on /media/root-rw type tmpfs (rw,relatime)
overlayroot on / type overlay (rw,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)

So clearly, my SSD was mounted read-only, and used as “lowerdir”. The overlay directory is at /media/root-rw/overlay/, and the working directory (where files are prepared before going to the overlay, in order to ensure atomicity) is /media/root-rw/overlay-workdir/.

Preventing confusion

Trying to avoid confusion on whether the system is volatile or not, I added this script to the “Startup Applications” on my Cinnamon desktop (Menu > Preferences > Startup Applications):


# Set a dedicated background for volatile (overlay) sessions

if grep -q overlayroot /proc/cmdline ; then
  /usr/bin/gsettings set org.cinnamon.desktop.background picture-uri \

I put the script in /usr/local/bin/overlay-wallpaper, and the wallpaper as /usr/local/share/wallpaper.jpg. Since this script runs only when the system is volatile, this setting is gone when the system reboots. More about setting up Gnome attributes in this post.

FWIW, adding this script as a startup application resulted in a file named “.config/autostart/Special wallpaper for overlay session.desktop” which reads:

[Desktop Entry]
Name[en_US]=Overlay wallpaper
Comment[en_US]=Special wallpaper for overlay session


  • overlayroot is a great way to tell what files are changed while the system is running. Just
    $ ls -RC on /media/root-rw/overlay/

    tells the whole story.

  • The size of tmpfs (and hence the overlay) is half the physical RAM by default. On my machine, that’s almost 4 GB, which is probably more than ever necessary. But if tmpfs becomes full, the computer will deadlock, as stated in the kernel’s documentation. Meaning: A simple program that decides to write a huge file can freeze the system completely. If that’s a problem, use some disk partition as the lowerdir, and wipe it clean (make2fs) on each boot before mounting.
  • If you want to save something permanently, it’s possible to remount the underlying root directory in read-write mode,
    $ sudo mount -o remount,rw /media/root-ro

    Note that this doesn’t disable the overlay — changes are not saved. To have something written permanently, change or create files under /media/root-ro/.
    This may or may not have an immediate effect on the currently seen root directory, depending on whether it has an entry in the overlay directory. So it’s useful mostly for minor fixes or to save something specific.

  • Every change in the file causes a copy into the overlay directory, including changes in permissions etc. If the file is deleted, a “whiteout” file is created in the respective position in the overlay, marking that it’s absent. Something like this (after deleting eli.tar.gz from my Desktop):
    $ cd /media/root-rw/overlay/home/eli/Desktop/
    $ ls -l
    total 0
    c--------- 1 root root 0, 0 Feb 21 13:48 eli.tar.gz
  • Judging by the vast amount of patches under fs/overlayfs/ in the kernel tree since v4.4 (to say, v4.10), the whole thing seems a bit shaky. On the other hand, who cares, if it messes up just pull the plug and reboot. And frankly, it works great.

Add a Comment

required, use real name
required, will not be published
optional, your blog address