Upgrading to Linux Mint 19, running the old system in a chroot
Background
Archaeological findings have revealed that prehistoric humans buried their forefathers under the floor of their huts. Fast forward to 2018, yours truly decided to continue running the (ancient) Fedora 12 as a chroot when migrating to Linux Mint 19. That’s an eight years difference.
While a lot of Linux users are happy to just install the new system and migrate everything “automatically”, this isn’t a good idea if you’re into more than plain tasks. Upgrading is supposed to be smooth, but small changes in the default behavior, API or whatever always make things that worked before fail, and sometimes with significant damage. Of the sort of not receiving emails, backup jobs not really working as before etc. Or just a new bug.
I’ve talked with quite a few sysadmins who were responsible for computers that actually needed to work continuously and reliably, and it’s not long before the apology for their ancient Linux distribution arrived. There’s no need to apologize: Upgrading is not good for keeping the system running smoothly. If it ain’t broke, don’t fix it.
But after some time, the hardware gets old and it becomes difficult to install new software. So I had this idea to keep running the old computer, with all of its properly running services and cronjobs, as a virtual machine. And then I thought, maybe go VPS-style. And then I realized I don’t need the VPS isolation at all. So the idea is to keep the old system as a chroot inside the new one.
Some services (httpd, mail handling, dhcpd) will keep running in the chroot, and others (the desktop in particular, with new shiny GUI programs) running natively. Old and new on the same machine.
The trick is making sure one doesn’t stamp on the feet of the other. These are my insights as I managed to get this up and running.
The basics
The idea is to place the old root filesystem (only) into somewhere in the new system, and chroot into it for the sake of running services and oldschool programs:
- The old root is placed as e.g. /oldy-root/ in the new filesystem (note that oldy is a legit alternative spelling for oldie…).
- bind-mounts are used for a unified view of home directories and those containing data.
- Some services are executed from within the chroot environment. How to run them from Mint 19 (hence using systemd) is described below.
- Running old programs is also possible by chrooting from shell. This is also discussed below.
Don’t put the old root on a filesystem that contains useful data, because odds are that such file system will be bind-mounted into the chrooted filesystem, which will cause a directory tree loop. Then try to calculate disk space or backup with tar. So pick a separate filesystem (i.e. a separate partition or LVM volume), or possibly as a subdirectory of the same filesystem as the “real” root.
Bind mounting
This is where the tricky choices are made. The point is to make the old and new systems see more or less the same application data, and also allow software to communicate over /tmp. So this is the relevant part in my /etc/fstab:
# Bind mounts for oldy root: system essentials /dev /oldy-root/dev none bind 0 2 /dev/pts /oldy-root/dev/pts none bind 0 2 /dev/shm /oldy-root/dev/shm none bind 0 2 /sys /oldy-root/sys none bind 0 2 /proc /oldy-root/proc none bind 0 2 # Bind mounts for oldy root: Storage /home /oldy-root/home none bind 0 2 /storage /oldy-root/storage none bind 0 2 /tmp /oldy-root/tmp none bind 0 2 /mnt /oldy-root/mnt none bind 0 2 /media /oldy-root/media none bind 0 2
Most notable are /mnt and /media. Bind-mounting these allows temporary mounts to be visible at both sides. /tmp is required for the UNIX domain socket used for playing sound from the old system. And other sockets, I suppose.
Note that /run isn’t bind-mounted. The reason is that the tree structure has changed, so it’s quite pointless (the mounting point used to be /var/run, and the place of the runtime files tend to change with time). The motivation for bind mounting would have been to let software from the old and new software interact, and indeed, there are a few UNIX sockets there, most notably the DBus domain UNIX socket.
But DBus is a good example of how hopeless it is to bind-mount /run: Old software attempting to talk with the Console Kit on the new DBus server fails completely at the protocol level (or namespace? I didn’t really dig into that).
So just copy the old /var/run into the root filesystem and that’s it. CUPS ran smoothly, GUI programs run fairly OK, and sound is done through a UNIX domain socket as suggested in the comments of this post.
I opted out on bind mounting /lib/modules and /usr/src. This makes manipulations of kernel modules (as needed by VMware, for example) impossible from the old system. But gcc is outdated for compiling under the new Linux kernel build system, so there was little point.
/root isn’t bind-mounted either. I wasn’t so sure about that, but in the end, it’s not a very useful directory. Keeping them separate makes the shell history for the root user distinct, and that’s actually a good thing.
Make /dev/log for real
Almost all service programs (and others) send messages to the system log by writing to the UNIX domain socket /dev/log. It’s actually a misnomer, because /dev/log is not a device file. But you don’t break tradition.
WARNING: If the logging server doesn’t work properly, Linux will fail to boot, dropping you into a tiny busybox rescue shell. So before playing with this, reboot to verify all is fine, and then make the changes. Be sure to prepare yourself for reverting your changes with plain command-line utilities (cp, mv, cat) and reboot to make sure all is fine.
In Mint 19 (and forever on), logging is handled by systemd-journald, which is a godsend. However for some reason (does anyone know why? Kindly comment below), the UNIX domain socket it creates is placed at /run/systemd/journal/dev-log, and /dev/log is a symlink to it. There are a few bug reports out there on software refusing to log into a symlink.
But that’s small potatoes: Since I decided not to bind-mount /run, there’s no access to this socket from the old system.
The solution is to swap the two: Make /dev/log the UNIX socket (as it was before), and /run/systemd/journal/dev-log the symlink (I wonder if the latter is necessary). To achieve this, copy /lib/systemd/system/systemd-journald-dev-log.socket into /etc/systemd/system/systemd-journald-dev-log.socket. This will make the latter override the former (keep the file name accurate), and make the change survive possible upgrades — the file in /lib can be overwritten by apt, the one in /etc won’t be by convention.
Edit the file in /etc, in the part saying:
[Socket]
Service=systemd-journald.service
ListenDatagram=/run/systemd/journal/dev-log
Symlinks=/dev/log
SocketMode=0666
PassCredentials=yes
PassSecurity=yes
and swap the files, making it
ListenDatagram=/dev/log Symlinks=/run/systemd/journal/dev-log
instead.
All in all this works perfectly. Old programs work well (try “logger” command line utility on both sides). This can cause problems if the program expects “the real thing” on /run/systemd/journal/dev-log. Quite unlikely.
As a side note, I had this idea to make journald listen to two UNIX domain sockets: Dropping the Symlinks assignment in the original .socket file, and copying it into a new .socket file, setting ListenDatagram to /dev/log. Two .socket files, two UNIX sockets. Sounded like a good idea, only it failed with an error message saying “Too many /dev/log sockets passed”.
Running old services
systemd’s take on sysV-style services (i.e. those init.d, rcN.d scripts) is that when systemctl is called with reference to a service, it first tries with its native services, and if none is found, it looks for a service of that name in /etc/init.d.
In order to run old services, I wrote a catch-all init.d script, /etc/init.d/oldy-chrooter. It’s intended to be symlinked to, so it tells which service it should run from the command used to call it, then chroots, and executes the script inside the old system. And guess what, systemd plays along with this.
The script follows. Note that it’s written in Perl, but it has the standard info header, which is required on init scripts. String manipulations are easier this way.
#!/usr/bin/perl
### BEGIN INIT INFO
# Required-Start: $local_fs $remote_fs $syslog
# Required-Stop: $local_fs $remote_fs $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# X-Interactive: false
# Short-Description: Oldy root wrapper service
# Description: Start a service within the oldy root
### END INIT INFO
use warnings;
use strict;
my $targetroot = '/oldy-root';
my ($realcmd) = ($0 =~ /\/oldy-([^\/]+)$/);
die("oldy chroot delegation script called with non-oldy command \"$0\"\n")
unless (defined $realcmd);
chroot $targetroot or die("Failed to chroot to $targetroot\n");
exec("/etc/init.d/$realcmd", @ARGV) or
die("Failed to execute \"/etc/init.d/$realcmd\" in oldy chroot\n");
To expose the chroot’s httpd service, make a symlink in init.d:
# cd /etc/init.d/ # ln -s oldy-chrooter oldy-httpd
And then enable with
# systemctl enable oldy-httpd oldy-httpd.service is not a native service, redirecting to systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable oldy-httpd
which indeed runs /lib/systemd/systemd-sysv-install, a shell script, which in turn runs /usr/sbin/update-rc.d with the same arguments. The latter is a Perl script, which analyzes the init.d file, and, among others, parses the INFO header.
The result is the SysV-style generation of S01/K01 symbolic links into /etc/rcN.d. Consequently, it’s possible to start and stop the service as usual. If the service isn’t enabled (or disabled) with systemctl first, attempting to start and stop the service will result in an error message saying the service isn’t found.
It’s a good idea to install the same services on the “main” system and disable them afterwards. There’s no risk for overwriting the old root’s installation, and this allows installation and execution of programs that depend on these services (or they would complain based upon the software package database).
Running programs
Running stuff inside the chroot should be quick and easy. For this reason, I wrote a small C program, which opens a shell within the chroot when called without argument. With one argument, it executes it within the chroot. It can be called by a non-root user, and the same user is applied in the chroot.
This is compiled with
$ gcc oldy.c -o oldy -Wall -O3
and placed /usr/local/bin with setuid root:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>
#include <sys/types.h>
#include <pwd.h>
int main(int argc, char *argv[]) {
const char jail[] = "/oldy-root/";
const char newhome[] = "/oldy-root/home/eli/";
struct passwd *pwd;
if ((argc!=2) && (argc!=1)){
printf("Usage: %s [ command ]\n", argv[0]);
exit(1);
}
pwd = getpwuid(getuid());
if (!pwd) {
perror("Failed to obtain user name for current user(?!)");
exit(1);
}
// It's necessary to set the ID to 0, or su asks for password despite the
// root setuid flag of the executable
if (setuid(0)) {
perror("Failed to change user");
exit(1);
}
if (chdir(newhome)) {
perror("Failed to change directory");
exit(1);
}
if (chroot(jail)) {
perror("Failed to chroot");
exit(1);
}
// oldycmd and oldyshell won't appear, as they're overridden by su
if (argc == 1)
execl("/bin/su", "oldyshell", "-", pwd->pw_name, (char *) NULL);
else
execl("/bin/su", "oldycmd", "-", pwd->pw_name, "-c", argv[1], (char *) NULL);
perror("Execution failed");
exit(1);
}
Notes:
- Using setuid root is a number one for security holes. I’m not sure I would have this thing on a computer used by strangers.
- getpwuid() gets the real user ID (not the effective one, as set by setuid), so the call to “su” is made with the original user (even if it’s root, of course). It will fail if that user doesn’t exist.
- … but note that the user in the chroot system is then one having the same user name as in the original one, not uid. There should be no difference, but watch it if there is (security holes…?)
- I used “su -” and not just executing bash for the sake of su’s “-” flag, which sets up the environment. Otherwise, it’s a mess.
It’s perfectly OK to run GUI programs with this trick. However it becomes extremely confusing with command line. Is this shell prompt on the old or new system? To fix this, edit /etc/bashrc in the chroot system only to change the prompt. I went for changing the line saying
[ "$PS1" = "\\s-\\v\\\$ " ] && PS1="[\u@\h \W]\\$ "
to
[ "$PS1" = "\\s-\\v\\\$ " ] && PS1="\[\e[44m\][\u@chroot \W]\[\e[m\]\\$ "
so the “\h” part, which turns into the host’s name now appears as “chroot”. But more importantly, the text background of the shell prompt is changed to blue (as opposed to nothing), so it’s easy to tell where I am.
If you’re into playing with the colors, I warmly recommend looking at this.
Lifting the user processes limit
At some point (it took a few months), I started to have failures of this sort:
$ oldy oldyshell: /bin/bash: Resource temporarily unavailable
and even worse, some of the chroot-based utilities also failed sporadically.
Checking with ulimit -a, it turned out that the limit for the number of processes owned by my “regular” user was limited to 1024. Checking with ps, I had only about 510 processes belonging to that UID. So it’s not clear why I hit the limit. In the non-chroot environment, the limit is significantly higher.
So edit /etc/security/limits.d/90-nproc.conf (the one inside the jail), changing the line saying
-* soft nproc 1024
to
* soft nproc 65536
There’s no need for any reboot or anything of that sort, but the already running processes remain within the limit.
Desktop icons and wallpaper messup
This is a seemingly small, but annoying thing: When Nautilus is launched from within the old system, it restores the old wallpaper and sets all icons on the desktop. There are suggestions on how to fix it, but they rely on gsettings, which came after Fedora 12. Haven’t tested this, but is the common suggestion is:
$ gsettings set org.gnome.desktop.background show-desktop-icons falseSo for old systems as mine, first, check the current value:
$ gconftool-2 --get /apps/nautilus/preferences/show_desktop
and if it’s “true”, fix it:
$ gconftool-2 --type bool --set /apps/nautilus/preferences/show_desktop false
The settings are stored in ~/.gconf/apps/nautilus/preferences/%gconf.xml.
Setting title in gnome-terminal
So someone thought that the possibility to set the title in the Terminal window, directly from the GUI, is unnecessary. That happens to be one of the most useful features, if you ask me. I’d really like to know why they dropped that. Or maybe not.
After some wandering around, and reading suggestions on how to do it in various other ways, I went for the old-new solution: Run the old executable in the new system. Namely:
# cd /usr/bin # mv gnome-terminal new-gnome-terminal # ln -s /oldy-root/usr/bin/gnome-terminal
It was also necessary to install some library stuff:
# apt install libvte9
But then it complained that it can’t find some terminal.xml file. So
# cd /usr/share/ # ln -s /oldy-root/usr/share/gnome-terminal
And then I needed to set up the keystroke shortcuts (Copy, Paste, New Tab etc.) but that’s really no bother.
Other things to keep in mind
- Some users and groups must be migrated from the old system to the new manually. I do this always when installing a new computer to make NFS work properly etc, but in this case, some service-related users and groups need to be in sync.
- Not directly related, but if the IP address of the host changes (which it usually does), set the updated IP address in /etc/sendmail.mc, and recompile. Or get an error saying “opendaemonsocket: daemon MTA: cannot bind: Cannot assign requested address”.