Occasionally, I download / upload huge files, and it kills my internet connection for plain browsing. I don’t want to halt the download or suspend it, but merely calm it down a bit, temporarily, for doing other stuff. And then let it hog as much as it want again.

There are many ways to do this, and I went for firejail. I suggest reading this post of mine as well on this tool.

Firejail gives you a shell prompt, which runs inside a mini-container, like those cheap virtual hosting services. Then run wget or youtube-dl as you wish from that shell.

It has practically access to everything on the computer, but the network interface is controlled. Since firejail is based on cgroups, all processes and subprocesses are collectively subject to the network bandwidth limit.

Using firejail requires setting up a bridge network interface. This is a bit of container hocus-pocus, and is necessary to get control over the network data flow. But it’s simple, and it can be done once (until the next reboot, unless the bridge is configured permanently, something I don’t bother).

Setting up a bridge interface

Remember: Do this once, and just don’t remove the interface when done with it.

You might need to

# apt install bridge-utils

So first, set up a new bridge device (as root):

# brctl addbr hog0

and give it an IP address that doesn’t collide with anything else on the system. Otherwise, it really doesn’t matter which:

# ifconfig hog0 10.22.1.1/24

What’s going to happen is that there will be a network interface named eth0 inside the container, which will behave as if it was connected to a real Ethernet card named hog0 on the computer. Hence the container has access to everything that is covered by the routing table (by means of IP forwarding), and is also subject to the firewall rules. With my specific firewall setting, it prevents some access, but ppp0 isn’t blocked, so who cares.

To remove the bridge (no real reason to do it):

# brctl delbr hog0

Running the container

Launch a shell with firejail (I called it “nethog” in this example):

$ firejail --net=hog0 --noprofile --name=nethog

This starts a new shell, for which the bandwidth limit is applied. Run wget or whatever from here.

Note that despite the –noprofile flag, there are still some directories that are read-only and some are temporary as well. It’s done in a sensible way, though so odds are that it won’t cause any issues. Running “df” inside the container gives an idea on what is mounted how, and it’s scarier than the actual situation.

But be sure to check that the files that are downloaded are visible outside the container.

From another shell prompt, outside the container go something like (doesn’t require root):

$ firejail --bandwidth=nethog set hog0 800 75
Removing bandwith limit
Configuring interface eth0
Download speed  6400kbps
Upload speed  600kbps
cleaning limits
configuring tc ingress
configuring tc egress

To drop the bandwidth limit:

$ firejail --bandwidth=nethog clear hog0

And get the status (saying, among others, how many packets have been dropped):

$ firejail --bandwidth=nethog status

Notes:

• The “eth0″ mentioned in firejail’s output blob relates to the interface name inside the container. So the “real” eth0 remains untouched.
• Actual download speed is slightly slower.
• The existing group can be joined by new processes with firejail –join, as well as from firetools.
• Several containers may use the same bridge (hog0 in the example above), in which case each has its own independent bandwidth setting. Note that the commands configuring the bandwidth limits mention both the container’s name and the bridge.

Working with browsers

When starting a browser from within a container, pay attention to whether it really started a new process. Using firetools can help.

If Google Chrome says “Created new window in existing browser session”, it didn’t start a new process inside the container, in which case the window isn’t subject to bandwidth limitation.

So close all windows of Chrome before kicking off a new one. Alternatively, this can we worked around by starting the container with.

$ firejail --net=hog0 --noprofile --private --name=nethog

The –private flags creates, among others, a new volatile home directory, so Chrome doesn’t detect that it’s already running. Because I use some other disk mounts for the large partitions on my computer, it’s still possible to download stuff to them from within the container.

But extra care is required with this, and regardless, the new browser doesn’t remember passwords and such from the private container.

Firejail is a lightweight security utility which ties the hands of running processes, somewhat like Apparmor and SELinux. However it takes the mission towards Linux kernel’s cgroups and namespaces. It’s in fact a bit of a container-style virtualization utility, which creates sandboxes for running specific programs: Instead of a container for an entire operating system, it makes one for each application (i.e. the main process and its children). Rather than disallowing access from files and directories by virtue of permissions, simply make sure they aren’t visible to the processes. Same goes for networking.

By virtue of Cgroups, several security restrictions are also put in place regardless if so desired. Certain syscalls can be prevented etc. But in the end of the day, think container virtualization. A sandbox is created, and everything happens inside it. It’s also easy to add processes to an existing sandbox (in particular, start a new shell). Not to mention the joy of shutting down a sandbox, that is, killing all processes inside it.

While the main use of Firejail to protect the file system from access and tampering by malicious or infected software, it also allows more or less everything that a container-style virtual machine does: Control of network traffic (volume, dedicated firewall, which physical interfaces are exposed) as well as activity (how many subprocesses, CPU and memory utilization etc.). And like a virtual machine, it also allows statistics on resource usage.

Plus spoofing the host name, restricting access to sound devices, X11 capabilities and a whole range of stuff.

And here’s the nice thing: It doesn’t require root privileges to run. Sort of. The firejail executable is run with setuid.

It’s however important to note that firejail doesn’t create a stand-alone container. Rather, it mixes and matches files from the real file system and overrides selected parts of the directory tree with temporary mounts. Or overlays. Or whiteouts.

In fact, compared with the accurate rules of a firewall, its behavior is quite loose and inaccurate. For a newbie, it’s a bit difficult to predict exactly what kind of sandbox it will set up given this or other setting. It throws in all kind of files of its own into the temporary directories it creates, which is very helpful to get things up and running quickly, but that doesn’t give a feeling of control.

Generally speaking, everything that isn’t explicitly handled by blacklisting or whitelisting (see below) is accessible in the sandbox just like outside it. In particular, it’s the user’s responsibility to hide away all those system-specific mounted filesystems (do you call them /mnt/storage?). If desired, of course.

Major disclaimer: This post is not authoritative in any way, and contains my jots as I get to know the beast. In particular, I may mislead you to think something is protected even though it’s not. You’re responsible to your own decisions.

The examples below are with firejail version 0.9.52 on a Linux Mint 19.

Install

# apt install firejail
# apt install firetools

By all means, go

$ man firejail

after installation. It’s also worth to look at /etc/firejail/ to get an idea on what protection measures are typically used.

Key commands

Launch FireTools, a GUI front end:

$ firetools &

And the “Tools” part has a nice listing of running sandboxes (right-click the ugly thing that comes up).

Now some command line examples. I name the sandboxes in these examples, but I’m not sure it’s worth bothering.

List existing sandboxes (or use FireTools, right-click the panel and choose Tools):

$ firejail --list

Assign a name to a sandbox when creating it

$ firejail --name=mysandbox firefox

Shut down a sandbox (kill all its processes, and clean up):

$ firejail --shutdown=mysandbox

If a name wasn’t assigned, the PID given in the list can be used instead.

Disallow the root user in the sandbox

$ firejail --noroot

Create overlay filesystem (mounts read/write, but changes are kept elsewhere)

$ firejail --overlay firefox

There’s also –overlay-tmpfs for overlaying tmpfs only, as well as –overlay-clean to clean the overlays, which are stored in $HOME/.firejail.

To create a completely new home directory (and /root) as temporary filesystems (private browsing style), so they are volatile:

$ firejail --private firefox

Better still,

$ firejail --private=/path/to/extra-homedir firefox

This uses the directory in the given path as a persistent home directory (some basic files are added automatically). This path can be anywhere in the filesystem, even in parts that are otherwise hidden (i.e. blacklisted) to the sandbox. So this is probably the most appealing choice in most scenarios.

Don’t get too excited, though: Other mounted filesystems remain unprotected (at different levels). This just protects the home directory.

By default, a whole bunch of security rules are loaded when firejail is invoked. To start the container without this:

$ firejail --noprofile

A profile can be selected with the –profile=filename flag.

Writing a profile

If you really want to have a sandbox that protects your computer with relation to a specific piece of software, you’ll probably have to write your own profile. It’s no big deal, except that it’s a bit of trial and error.

First read the manpage:

$ man firejail-profile

It’s easiest to start from a template: Launch FireTools from a shell, right-click the ugly thing that comes up, and pick “Configuration Wizard”, and create a custom security profile for one of the listed application — the one that resembles most the one for which the profile is set up.

Then launch the application from FireTools. The takeaway is that it writes out the configuration file to the console. Start with that.

Whilelisting and blacklisting

First and foremost: Always run a

$ df -h

inside the sandbox to get an idea of what is mounted. Blacklist anything that isn’t necessary. Doing so to entire mounts removes the related mount from the df -h list, which makes it easier to spot things that shouldn’t be there.

It’s also a good idea to start a sample bash session with the sandbox, and get into the File Manager in the Firetool’s “Tools” section for each sandbox.

But then, what is whitelisting and blacklisting, exactly? These two terms are used all over the docs, somehow assuming we know what they mean. So I’ll try to nail it down.

Whitelisting isn’t anywhere near what one would think it is: By whitelisting certain files and/or directories, the original files/directories appear in the sandbox but all other files in their vicinity are invisible. Also, changes in the same vicinity are temporary to the sandbox session. The idea seems to be that if files and/or directories are whitelisted, everything else close to it should be out of sight.

Or as put in the man page:

A temporary file system is mounted on the top directory, and the whitelisted files are mount-binded inside. Modifications to whitelisted files are persistent, everything else is discarded when the sandbox is closed. The top directory could be user home, /dev, /media, /mnt, /opt, /srv, /var, and /tmp.

So for example, if any file or directory in the home directory is whitelisted, the entire home directory becomes overridden by an almost empty home directory plus the specifically whitelisted items. For example, from my own home directory (which is populated with a lot of files):

$ firejail --noprofile --whitelist=/home/eli/this-directory
Parent pid 31560, child pid 31561
Child process initialized in 37.31 ms

$ find .
.
./.config
./.config/pulse
./.config/pulse/client.conf
./this-directory
./this-directory/this-file.txt
./.Xauthority
./.bashrc

So there’s just a few temporary files that firejail was kind enough to add for convenience. Changes made in this-directory/ are persistent since it’s bind-mounted into the temporary directory, but everything else is temporary.

Quite unfortunately, it’s not possible to whitelist a directory outside the specific list of hierarchies (unless bind mounting is used, but that requires root). So if the important stuff is one some /hugedisk, only a bind mount will help (or is this the punishment for not putting it has /mnt/hugedisk?).

But note that the –private= flag allows setting the home directory to anywhere on the filesystem (even inside a blacklisted region). This ad-hoc home directory is persistent, so it’s not like whitelisting, but even better is some scenarios.

Alternatively, it’s possible to blacklist everything but a certain part of a mount. That’s a bit tricky, because if a new directory appears after the rules are set, it remains unprotected. I’ll explain why below.

Or if that makes sense, make the entire directory tree read-only, with only a selected part read-write. That’s fine if there’s no issue with data leaking, just the possibility of malware sabotage.

So now to blacklisting: Firejail implements blacklisting by mounting an empty, read-only-by-root file or directory on top of the original file. And indeed,

$ firejail --blacklist=delme.txt
Reading profile /etc/firejail/default.profile
Reading profile /etc/firejail/disable-common.inc
Reading profile /etc/firejail/disable-passwdmgr.inc
Reading profile /etc/firejail/disable-programs.inc

** Note: you can use --noprofile to disable default.profile **

Parent pid 30288, child pid 30289
Child process initialized in 57.75 ms
$ ls -l
[ ... ]
-r--------  1 nobody nogroup     0 Jun  9 22:12 delme.txt
[ ... ]
$ less delme.txt
delme.txt: Permission denied

There are –noblacklist and –nowhitelist flags as well. However these merely cancel future or automatic black- or whitelistings. In particular, one can’t blacklist a directory and whitelist a subdirectory. It would have been very convenient, but since the parent directory is overridden with a whiteout directory, there is no access to the subdirectory. So each and every subdirectory must be blacklisted separately with a script or something, and even then if a new subdirectory pops up, it’s not protected at all.

There’s also a –read-only flag allows setting certain paths and files as read-only. There’s –read-write too, of course. When a directory or file is whitelisted, it must be flagged read-only separately if so desired (see man firejail).

Mini-strace

Trace all processes in the sandbox (in particular accesses to files and network). Much easier than using strace, when all we want is “which files are accessed?”

$ firejail --trace

And then just run any program to see what files and network sockets it accesses. And things of that sort.

This post is a bit of a coredump of myself attempting to resolve a sudden web server failure. And even more important, understand why it happened (check on that) and try avoiding it from happening in the future (not as lucky there).

I’ve noticed that there are many threads in the Internet on why mysqld died suddenly, so to make a long story short: mysqld has the exact profile that the OOM killer is looking for: Lots of resident RAM, and it’s not a system process. Apache gets killed every now and then for the same reason.

This post relates to a  VPS hosted Debian 8, kernel 3.10.0, x86_64. The MySQL server is a 5.5.62-0+deb8u1 (Debian).

As always, it started with a mail notification from some cronjob complaining about something. Soon enough it was evident that the MySQL server was down. And as usual, the deeper I investigated this issue, the more I realized that this was just the tip of the iceberg (the kind that doesn’t melt due to global warming).

The crash

So first, it was clear that the MySQL had restarted itself a couple of days before disaster:

191007  9:25:17 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
191007  9:25:17 [Note] Plugin 'FEDERATED' is disabled.
191007  9:25:17 InnoDB: The InnoDB memory heap is disabled
191007  9:25:17 InnoDB: Mutexes and rw_locks use GCC atomic builtins
191007  9:25:17 InnoDB: Compressed tables use zlib 1.2.8
191007  9:25:17 InnoDB: Using Linux native AIO
191007  9:25:17 InnoDB: Initializing buffer pool, size = 128.0M
191007  9:25:17 InnoDB: Completed initialization of buffer pool
191007  9:25:17 InnoDB: highest supported file format is Barracuda.
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
191007  9:25:17  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
191007  9:25:19  InnoDB: Waiting for the background threads to start
191007  9:25:20 InnoDB: 5.5.62 started; log sequence number 1427184442
191007  9:25:20 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
191007  9:25:20 [Note]   - '127.0.0.1' resolves to '127.0.0.1';
191007  9:25:20 [Note] Server socket created on IP: '127.0.0.1'.
191007  9:25:21 [Note] Event Scheduler: Loaded 0 events
191007  9:25:21 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.62-0+deb8u1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Debian)
191007  9:25:28 [ERROR] /usr/sbin/mysqld: Table './mydb/wp_options' is marked as crashed and should be repaired
191007  9:25:28 [Warning] Checking table:   './mydb/wp_options'
191007  9:25:28 [ERROR] /usr/sbin/mysqld: Table './mydb/wp_posts' is marked as crashed and should be repaired
191007  9:25:28 [Warning] Checking table:   './mydb/wp_posts'
191007  9:25:28 [ERROR] /usr/sbin/mysqld: Table './mydb/wp_term_taxonomy' is marked as crashed and should be repaired
191007  9:25:28 [Warning] Checking table:   './mydb/wp_term_taxonomy'
191007  9:25:28 [ERROR] /usr/sbin/mysqld: Table './mydb/wp_term_relationships' is marked as crashed and should be repaired
191007  9:25:28 [Warning] Checking table:   './mydb/wp_term_relationships'

And then, two days layer, it crashed for real. Or actually, got killed. From the syslog:

Oct 09 05:30:16 kernel: OOM killed process 22763 (mysqld) total-vm:2192796kB, anon-rss:128664kB, file-rss:0kB

and

191009  5:30:17 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
191009  5:30:17 [Note] Plugin 'FEDERATED' is disabled.
191009  5:30:17 InnoDB: The InnoDB memory heap is disabled
191009  5:30:17 InnoDB: Mutexes and rw_locks use GCC atomic builtins
191009  5:30:17 InnoDB: Compressed tables use zlib 1.2.8
191009  5:30:17 InnoDB: Using Linux native AIO
191009  5:30:17 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
191009  5:30:17 InnoDB: Completed initialization of buffer pool
191009  5:30:17 InnoDB: Fatal error: cannot allocate memory for the buffer pool
191009  5:30:17 [ERROR] Plugin 'InnoDB' init function returned error.
191009  5:30:17 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
191009  5:30:17 [ERROR] Unknown/unsupported storage engine: InnoDB
191009  5:30:17 [ERROR] Aborting

191009  5:30:17 [Note] /usr/sbin/mysqld: Shutdown complete

The mmap() is most likely anonymous (i.e. not related to a file), as I couldn’t find any memory mapped file that is related to the mysql processes (except for the obvious mappings of shared libraries).

The smoking gun

But here comes the good part: It turns out that the OOM killer had been active several times before. It just so happen that the processes are being newborn every time this happens. It was the relaunch that failed this time — otherwise I wouldn’t have noticed this was going on.

This is the output of plain “dmesg”. All OOM entries but the last one were not available with journalctl, as old entries had been deleted.

[3634197.152028] OOM killed process 776 (mysqld) total-vm:2332652kB, anon-rss:153508kB, file-rss:0kB
[3634197.273914] OOM killed process 71 (systemd-journal) total-vm:99756kB, anon-rss:68592kB, file-rss:4kB
[4487991.904510] OOM killed process 3817 (mysqld) total-vm:2324456kB, anon-rss:135752kB, file-rss:0kB
[4835006.413510] OOM killed process 23267 (mysqld) total-vm:2653112kB, anon-rss:131272kB, file-rss:4404kB
[4835006.767112] OOM killed process 32758 (apache2) total-vm:282528kB, anon-rss:11732kB, file-rss:52kB
[4884915.371805] OOM killed process 825 (mysqld) total-vm:2850312kB, anon-rss:121164kB, file-rss:5028kB
[4884915.509686] OOM killed process 17611 (apache2) total-vm:282668kB, anon-rss:11736kB, file-rss:444kB
[5096265.088151] OOM killed process 23782 (mysqld) total-vm:4822232kB, anon-rss:105972kB, file-rss:3784kB
[5845437.591031] OOM killed process 24642 (mysqld) total-vm:2455744kB, anon-rss:137784kB, file-rss:0kB
[5845437.608682] OOM killed process 3802 (systemd-journal) total-vm:82548kB, anon-rss:51412kB, file-rss:28kB
[6896254.741732] OOM killed process 11551 (mysqld) total-vm:2718652kB, anon-rss:144116kB, file-rss:220kB
[7054957.856153] OOM killed process 22763 (mysqld) total-vm:2192796kB, anon-rss:128664kB, file-rss:0kB

Or, after calculating the time stamps (using the last OOM message as a reference):

Fri Aug 30 15:17:36 2019 OOM killed process 776 (mysqld) total-vm:2332652kB, anon-rss:153508kB, file-rss:0kB
Fri Aug 30 15:17:36 2019 OOM killed process 71 (systemd-journal) total-vm:99756kB, anon-rss:68592kB, file-rss:4kB
Mon Sep  9 12:27:30 2019 OOM killed process 3817 (mysqld) total-vm:2324456kB, anon-rss:135752kB, file-rss:0kB
Fri Sep 13 12:51:05 2019 OOM killed process 23267 (mysqld) total-vm:2653112kB, anon-rss:131272kB, file-rss:4404kB
Fri Sep 13 12:51:05 2019 OOM killed process 32758 (apache2) total-vm:282528kB, anon-rss:11732kB, file-rss:52kB
Sat Sep 14 02:42:54 2019 OOM killed process 825 (mysqld) total-vm:2850312kB, anon-rss:121164kB, file-rss:5028kB
Sat Sep 14 02:42:54 2019 OOM killed process 17611 (apache2) total-vm:282668kB, anon-rss:11736kB, file-rss:444kB
Mon Sep 16 13:25:24 2019 OOM killed process 23782 (mysqld) total-vm:4822232kB, anon-rss:105972kB, file-rss:3784kB
Wed Sep 25 05:31:36 2019 OOM killed process 24642 (mysqld) total-vm:2455744kB, anon-rss:137784kB, file-rss:0kB
Wed Sep 25 05:31:36 2019 OOM killed process 3802 (systemd-journal) total-vm:82548kB, anon-rss:51412kB, file-rss:28kB
Mon Oct  7 09:25:13 2019 OOM killed process 11551 (mysqld) total-vm:2718652kB, anon-rss:144116kB, file-rss:220kB
Wed Oct  9 05:30:16 2019 OOM killed process 22763 (mysqld) total-vm:2192796kB, anon-rss:128664kB, file-rss:0kB

anon-rss is the resident RAM consumed by the process itself (anonymous = not memory mapped to a file or something like that).

total-vm is the total size of the Virtual Memory in use. This isn’t very relevant (I think), as it involves shared libraries, memory mapped files and other segments that don’t consume any actual RAM or other valuable resources.

So now it’s clear what happened. Next, to some finer resolution.

The MySQL keepaliver

The MySQL daemon is executed by virtue of an SysV init script, which launches /usr/bin/mysqld_safe, a patch-on-patch script to keep the daemon alive, no matter what. It restarts the mysqld daemon if it dies for any or no reason, and should also produce log messages. On my system, it’s executed as

/usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306

The script issues log messages when something unexpected happens, but they don’t appear in /var/log/mysql/error.log or anywhere else, even though the file exists, is owned by the mysql user, and has quite a few messages from the mysql daemon itself.

Changing

/usr/bin/mysqld_safe > /dev/null 2>&1 &

to

/usr/bin/mysqld_safe --syslog > /dev/null 2>&1 &

Frankly speaking, I don’t think this made any difference. I’ve seen nothing new in the logs.

It would have been nicer having the messages in mysql/error.log, but at least they are visible with journalctl this way.

Shrinking the InnoDB buffer pool

As the actual failure was on attempting to map memory for the buffer pool, maybe make it smaller…?

Launch MySQL as the root user:

$ mysql -u root --password

and check the InnoDB status, as suggested on this page:

mysql> SHOW ENGINE INNODB STATUS;

[ ... ]

----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 137363456; in additional pool allocated 0
Dictionary memory allocated 1100748
Buffer pool size   8192
Free buffers       6263
Database pages     1912
Old database pages 725
Modified db pages  0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 1894, created 18, written 1013
0.00 reads/s, 0.00 creates/s, 0.26 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 1912, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]

I’m really not an expert, but if “Free buffers” is 75% of the total allocated space, I’ve probably allocated too much. So I reduced it to 32 MB — it’s not like I’m running a high-end server. I added /etc/mysql/conf.d/innodb_pool_size.cnf (owned by root, 0644) reading:

# Reduce InnoDB buffer size from default 128 MB to 32 MB
[mysqld]
innodb_buffer_pool_size=32M

Restarting the daemon, it says:

----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 34340864; in additional pool allocated 0
Dictionary memory allocated 1100748
Buffer pool size   2047
Free buffers       856
Database pages     1189
Old database pages 458

And finally, repair the tables

Remember those warnings that the tables were marked as crashed? That’s the easy part:

$ mysqlcheck -A --auto-repair

That went smoothly, with no complaints. After all, it wasn’t really a crash.

Some general words on OOM

This whole idea that the kernel should do Roman Empire style decimation of processes is widely criticized by many, but it’s probably not such a bad idea. The root cause lies in the fact that the kernel agrees to allocate more RAM than it actually has. This is even possible because the kernel doesn’t really allocate RAM when a process asks for memory with a brk() call, but it only allocates the memory space segment. The actual RAM is allocated only when the process attempts to access a page that hasn’t been RAM allocated yet. The access attempt causes a page fault, the kernel quickly fixes some RAM and returns from the page fault interrupt as if nothing happened.

So when the kernel responds with an -ENOMEM, it’s not because it doesn’t have any RAM, but because it doesn’t want to.

More precisely, the kernel keeps account on how much memory it has given away (system-wise and/or cgroup-wise) and make a decision. The common policy is to overcommit to some extent — that is, to allow the total allocated RAM allocated to exceed the total physical RAM. Even, and in particular, if there’s no swap.

The common figure is to overcommit by 50%: For a 64 GiB RAM computer, there might be 96 GiB or promised RAM. This may seem awfully stupid thing to do, but hey, it works. If that concept worries you, modern banking (with real money, that is) might worry you even more.

The problem rises when the processes run to the bank. That is, when the processes access the RAM they’ve been promised, and at some point the kernel has nowhere to take memory from. Let’s assume there’s no swap, all disk buffers have been flushed, all rabbits have been pulled. There’s a process waiting for memory, and it can’t go back running until the problem has been resolved.

Linux’ solution to this situation is to select a process with a lot of RAM and little importance. How the kernel does that judgement is documented everywhere. The important point is that it’s not necessarily the process that triggered the event, and that it will usually be the same victim over and over again. In my case, mysqld is the favorite. Big, fat, and not a system process.

Thinking about it, the OOM is a good solution to get out of a tricky situation. The alternative would have been to deny memory to processes just launched, including the administrator’s attempt to rescue the system. Or an attempt to shut it down with some dignity. So sacrificing a large and hopefully not-so-important process isn’t such a bad idea.

Why did the OOM kick in?

This all took place on a VPS virtual machine with 1 GB leased RAM. With the stuff running on that machine, there’s no reason in the world that the total actual RAM consumption would reach that limit. This is a system that typically has 70% of its memory marked as “cached” (i.e. used by disk cache). This should be taken with a grain of salt, as “top” displays data from some bogus /proc/vmstat, and still.

As can be seen in the dmesg logs above, the amount of resident RAM of the killed mysqld process was 120-150 MB or so. Together with the other memory hog, apache2, they reach 300 MB. That’s it. No reason for anything drastic.

Having said that, it’s remarkable that the total-vm stood at 2.4-4.3 GB when killed. This is much higher than the typical 900 MB visible usually. So maybe there’s some kind of memory leak, even if it’s harmless? Looking at mysql over time, its virtual memory allocation tends to grow.

VPS machines do have a physical memory limit imposed, by virtue of the relevant cgroup’s memory.high and memory.max limits. In particular the latter — if the cgroup’s total consumption exceeds memory.max, OOM kicks in. This is how the illusion of an independent RAM segment is made on a VPS machine. Plus faking some /proc files.

But there’s another explanation: Say that a VPS service provider takes a computer with 16 GB RAM, and places 16 VPS machines with 1 GB leased RAM each. What will the overall actual RAM consumption be? I would expect it to be much lower than 16 GB. So why not add a few more VPS machines, and make some good use of the hardware? It’s where the profit comes from.

Most of the time, there will be no problem. But occasionally, this will cause RAM shortages, in which case the kernel’s global OOM looks for a victim. I suppose there’s no significance to cgroups in this matter. In other words, the kernel sees all processes in the system the same, regardless of which cgroup (and hence VPS machine) they belong to. Which means that the process killed doesn’t necessarily belong to the VPS that triggered the problem. The processes of one VPS may suddenly demand their memory, but some other VPS will have its processes killed.

Conclusion

• Shrinking the buffer pool of mysqld was probably a good idea, in particular if a computer-wide OOM killed the process — odds are that it will kill some other mysqld instead this way.
• Possibly restart mysql with a cronjob every day to keep its memory consumption in control. But this might create problems of its own.
• It’s high time to replace the VPS guest with KVM or similar.

—————————————————————-

Rambling epilogue: Some thoughts about overcomitting

The details for how overcomitting is accounted for is given on the kernel tree’s Documentation/vm/overcommit-accounting. But to make a long story short, it’s done in a sensible way. In particular, if a piece of memory is shared by threads and processes, it’s only accounted for once.

Relevant files: /proc/meminfo and /proc/vmstat

It seems like CommitLimit and Committed_AS are not available on a VPS guest system. But the OOM killer probably knows these values (or was it because /proc/sys/vm/overcommit_memory was set to 1 on my system, meaning “Always overcommit”?).

To get a list of the current memory hogs, run “top” and press shift-M as it’s running.

To get an idea on how a process behaves, use pmap -x. For example, looking at a mysqld process (run as root, or no memory map will be shown):

# pmap -x 14817
14817:   /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
Address           Kbytes     RSS   Dirty Mode  Mapping
000055c5617ac000   10476    6204       0 r-x-- mysqld
000055c5623e6000     452     452     452 r---- mysqld
000055c562457000     668     412     284 rw--- mysqld
000055c5624fe000     172     172     172 rw---   [ anon ]
000055c563e9b000    6592    6448    6448 rw---   [ anon ]
00007f819c000000    2296     320     320 rw---   [ anon ]
00007f819c23e000   63240       0       0 -----   [ anon ]
00007f81a0000000    3160     608     608 rw---   [ anon ]
00007f81a0316000   62376       0       0 -----   [ anon ]
00007f81a4000000    9688    7220    7220 rw---   [ anon ]
00007f81a4976000   55848       0       0 -----   [ anon ]
00007f81a8000000     132       8       8 rw---   [ anon ]
00007f81a8021000   65404       0       0 -----   [ anon ]
00007f81ac000000     132       4       4 rw---   [ anon ]
00007f81ac021000   65404       0       0 -----   [ anon ]
00007f81b1220000       4       0       0 -----   [ anon ]
00007f81b1221000    8192       8       8 rw---   [ anon ]
00007f81b1a21000       4       0       0 -----   [ anon ]
00007f81b1a22000    8192       8       8 rw---   [ anon ]
00007f81b2222000       4       0       0 -----   [ anon ]
00007f81b2223000    8192       8       8 rw---   [ anon ]
00007f81b2a23000       4       0       0 -----   [ anon ]
00007f81b2a24000    8192      20      20 rw---   [ anon ]
00007f81b3224000       4       0       0 -----   [ anon ]
00007f81b3225000    8192       8       8 rw---   [ anon ]
00007f81b3a25000       4       0       0 -----   [ anon ]
00007f81b3a26000    8192       8       8 rw---   [ anon ]
00007f81b4226000       4       0       0 -----   [ anon ]
00007f81b4227000    8192       8       8 rw---   [ anon ]
00007f81b4a27000       4       0       0 -----   [ anon ]
00007f81b4a28000    8192       8       8 rw---   [ anon ]
00007f81b5228000       4       0       0 -----   [ anon ]
00007f81b5229000    8192       8       8 rw---   [ anon ]
00007f81b5a29000       4       0       0 -----   [ anon ]
00007f81b5a2a000    8192       8       8 rw---   [ anon ]
00007f81b622a000       4       0       0 -----   [ anon ]
00007f81b622b000    8192      12      12 rw---   [ anon ]
00007f81b6a2b000       4       0       0 -----   [ anon ]
00007f81b6a2c000    8192       8       8 rw---   [ anon ]
00007f81b722c000       4       0       0 -----   [ anon ]
00007f81b722d000   79692   57740   57740 rw---   [ anon ]
00007f81bc000000     132      76      76 rw---   [ anon ]
00007f81bc021000   65404       0       0 -----   [ anon ]
00007f81c002f000    2068    2052    2052 rw---   [ anon ]
00007f81c03f9000       4       0       0 -----   [ anon ]
00007f81c03fa000     192      52      52 rw---   [ anon ]
00007f81c042a000       4       0       0 -----   [ anon ]
00007f81c042b000     192      52      52 rw---   [ anon ]
00007f81c045b000       4       0       0 -----   [ anon ]
00007f81c045c000     192      64      64 rw---   [ anon ]
00007f81c048c000       4       0       0 -----   [ anon ]
00007f81c048d000     736     552     552 rw---   [ anon ]
00007f81c0545000      20       4       0 rw-s- [aio] (deleted)
00007f81c054a000      20       4       0 rw-s- [aio] (deleted)
00007f81c054f000    3364    3364    3364 rw---   [ anon ]
00007f81c0898000      44      12       0 r-x-- libnss_files-2.19.so
00007f81c08a3000    2044       0       0 ----- libnss_files-2.19.so
00007f81c0aa2000       4       4       4 r---- libnss_files-2.19.so
00007f81c0aa3000       4       4       4 rw--- libnss_files-2.19.so
00007f81c0aa4000      40      20       0 r-x-- libnss_nis-2.19.so
00007f81c0aae000    2044       0       0 ----- libnss_nis-2.19.so
00007f81c0cad000       4       4       4 r---- libnss_nis-2.19.so
00007f81c0cae000       4       4       4 rw--- libnss_nis-2.19.so
00007f81c0caf000      28      20       0 r-x-- libnss_compat-2.19.so
00007f81c0cb6000    2044       0       0 ----- libnss_compat-2.19.so
00007f81c0eb5000       4       4       4 r---- libnss_compat-2.19.so
00007f81c0eb6000       4       4       4 rw--- libnss_compat-2.19.so
00007f81c0eb7000       4       0       0 -----   [ anon ]
00007f81c0eb8000    8192       8       8 rw---   [ anon ]
00007f81c16b8000      84      20       0 r-x-- libnsl-2.19.so
00007f81c16cd000    2044       0       0 ----- libnsl-2.19.so
00007f81c18cc000       4       4       4 r---- libnsl-2.19.so
00007f81c18cd000       4       4       4 rw--- libnsl-2.19.so
00007f81c18ce000       8       0       0 rw---   [ anon ]
00007f81c18d0000    1668     656       0 r-x-- libc-2.19.so
00007f81c1a71000    2048       0       0 ----- libc-2.19.so
00007f81c1c71000      16      16      16 r---- libc-2.19.so
00007f81c1c75000       8       8       8 rw--- libc-2.19.so
00007f81c1c77000      16      16      16 rw---   [ anon ]
00007f81c1c7b000      88      44       0 r-x-- libgcc_s.so.1
00007f81c1c91000    2044       0       0 ----- libgcc_s.so.1
00007f81c1e90000       4       4       4 rw--- libgcc_s.so.1
00007f81c1e91000    1024     128       0 r-x-- libm-2.19.so
00007f81c1f91000    2044       0       0 ----- libm-2.19.so
00007f81c2190000       4       4       4 r---- libm-2.19.so
00007f81c2191000       4       4       4 rw--- libm-2.19.so
00007f81c2192000     944     368       0 r-x-- libstdc++.so.6.0.20
00007f81c227e000    2048       0       0 ----- libstdc++.so.6.0.20
00007f81c247e000      32      32      32 r---- libstdc++.so.6.0.20
00007f81c2486000       8       8       8 rw--- libstdc++.so.6.0.20
00007f81c2488000      84       8       8 rw---   [ anon ]
00007f81c249d000      12       8       0 r-x-- libdl-2.19.so
00007f81c24a0000    2044       0       0 ----- libdl-2.19.so
00007f81c269f000       4       4       4 r---- libdl-2.19.so
00007f81c26a0000       4       4       4 rw--- libdl-2.19.so
00007f81c26a1000      32       4       0 r-x-- libcrypt-2.19.so
00007f81c26a9000    2044       0       0 ----- libcrypt-2.19.so
00007f81c28a8000       4       4       4 r---- libcrypt-2.19.so
00007f81c28a9000       4       4       4 rw--- libcrypt-2.19.so
00007f81c28aa000     184       0       0 rw---   [ anon ]
00007f81c28d8000      36      28       0 r-x-- libwrap.so.0.7.6
00007f81c28e1000    2044       0       0 ----- libwrap.so.0.7.6
00007f81c2ae0000       4       4       4 r---- libwrap.so.0.7.6
00007f81c2ae1000       4       4       4 rw--- libwrap.so.0.7.6
00007f81c2ae2000       4       4       4 rw---   [ anon ]
00007f81c2ae3000     104      12       0 r-x-- libz.so.1.2.8
00007f81c2afd000    2044       0       0 ----- libz.so.1.2.8
00007f81c2cfc000       4       4       4 r---- libz.so.1.2.8
00007f81c2cfd000       4       4       4 rw--- libz.so.1.2.8
00007f81c2cfe000       4       4       0 r-x-- libaio.so.1.0.1
00007f81c2cff000    2044       0       0 ----- libaio.so.1.0.1
00007f81c2efe000       4       4       4 r---- libaio.so.1.0.1
00007f81c2eff000       4       4       4 rw--- libaio.so.1.0.1
00007f81c2f00000      96      84       0 r-x-- libpthread-2.19.so
00007f81c2f18000    2044       0       0 ----- libpthread-2.19.so
00007f81c3117000       4       4       4 r---- libpthread-2.19.so
00007f81c3118000       4       4       4 rw--- libpthread-2.19.so
00007f81c3119000      16       4       4 rw---   [ anon ]
00007f81c311d000     132     112       0 r-x-- ld-2.19.so
00007f81c313e000       8       0       0 rw---   [ anon ]
00007f81c3140000      20       4       0 rw-s- [aio] (deleted)
00007f81c3145000      20       4       0 rw-s- [aio] (deleted)
00007f81c314a000      20       4       0 rw-s- [aio] (deleted)
00007f81c314f000      20       4       0 rw-s- [aio] (deleted)
00007f81c3154000      20       4       0 rw-s- [aio] (deleted)
00007f81c3159000      20       4       0 rw-s- [aio] (deleted)
00007f81c315e000      20       4       0 rw-s- [aio] (deleted)
00007f81c3163000      20       4       0 rw-s- [aio] (deleted)
00007f81c3168000    1840    1840    1840 rw---   [ anon ]
00007f81c3334000       8       0       0 rw-s- [aio] (deleted)
00007f81c3336000       4       0       0 rw-s- [aio] (deleted)
00007f81c3337000      24      12      12 rw---   [ anon ]
00007f81c333d000       4       4       4 r---- ld-2.19.so
00007f81c333e000       4       4       4 rw--- ld-2.19.so
00007f81c333f000       4       4       4 rw---   [ anon ]
00007ffd2d68b000     132      68      68 rw---   [ stack ]
00007ffd2d7ad000       8       4       0 r-x--   [ anon ]
ffffffffff600000       4       0       0 r-x--   [ anon ]
---------------- ------- ------- -------
total kB          640460   89604   81708

The KBytes and RSS column’s Total at the bottom matches the VIRT and RSS figures shown by “top”.

I should emphasize that this a freshly started mysqld process. Give it a few days to run, and some extra 100 MB of virtual space is added (not clear why) plus some real RAM, depending on the setting.

I’ve marked six anonymous segments that are completely virtual (no resident memory at all) summing up to ~360 MB. This means that they are counted in as 360 MB at least once — and that’s for a process that only uses 90 MB for real.

My own anecdotal test on another machine with a 4.4.0 kernel showed that putting /proc/sys/vm/overcommit_ratio below what was actually committed (making /proc/meminfo’s CommitLimit smaller than Committed_AS) didn’t have any effect unless /proc/sys/vm/overcommit_memory was set to 2. And when I did that, the OOM wasn’t called, but instead I had a hard time running new commands:

# echo 2 > /proc/sys/vm/overcommit_memory
# cat /proc/meminfo
bash: fork: Cannot allocate memory

So this is what it looks like when memory runs out and the system refuses to play ball.

Archaeological findings have revealed that prehistoric humans buried their forefathers under the floor of their huts. Fast forward to 2018, yours truly decided to continue running the (ancient) Fedora 12 as a chroot when migrating to Linux Mint 19. That’s an eight years difference.

While a lot of Linux users are happy to just install the new system and migrate everything “automatically”, this isn’t a good idea if you’re into more than plain tasks. Upgrading is supposed to be smooth, but small changes in the default behavior, API or whatever always make things that worked before fail, and sometimes with significant damage. Of the sort of not receiving emails, backup jobs not really working as before etc. Or just a new bug.

I’ve talked with quite a few sysadmins who were responsible for computers that actually needed to work continuously and reliably, and it’s not long before the apology for their ancient Linux distribution arrived. There’s no need to apologize: Upgrading is not good for keeping the system running smoothly. If it ain’t broke, don’t fix it.

But after some time, the hardware gets old and it becomes difficult to install new software. So I had this idea to keep running the old computer, with all of its properly running services and cronjobs, as a virtual machine. And then I thought, maybe go VPS-style. And then I realized I don’t need the VPS isolation at all. So the idea is to keep the old system as a chroot inside the new one.

Some services (httpd, mail handling, dhcpd) will keep running in the chroot, and others (the desktop in particular, with new shiny GUI programs) running natively. Old and new on the same machine.

The trick is making sure one doesn’t stamp on the feet of the other. These are my insights as I managed to get this up and running.

The basics

The idea is to place the old root filesystem (only) into somewhere in the new system, and chroot into it for the sake of running services and oldschool programs:

• The old root is placed as e.g. /oldy-root/ in the new filesystem (note that oldy is a legit alternative spelling for oldie…).
• bind-mounts are used for a unified view of home directories and those containing data.
• Some services are executed from within the chroot environment. How to run them from Mint 19 (hence using systemd) is described below.
• Running old programs is also possible by chrooting from shell. This is also discussed below.

Don’t put the old root on a filesystem that contains useful data, because odds are that such file system will be bind-mounted into the chrooted filesystem, which will cause a directory tree loop. Then try to calculate disk space or backup with tar. So pick a separate filesystem (i.e. a separate partition or LVM volume), or possibly as a subdirectory of the same filesystem as the “real” root.

Bind mounting

This is where the tricky choices are made. The point is to make the old and new systems see more or less the same application data, and also allow software to communicate over /tmp. So this is the relevant part in my /etc/fstab:

# Bind mounts for oldy root: system essentials
/dev                        /oldy-root/dev none bind                0       2
/dev/pts                    /oldy-root/dev/pts none bind            0       2
/dev/shm                    /oldy-root/dev/shm none bind            0       2
/sys                        /oldy-root/sys none bind                0       2
/proc                       /oldy-root/proc none bind               0       2

# Bind mounts for oldy root: Storage
/home                       /oldy-root/home none bind               0       2
/storage                    /oldy-root/storage none bind            0       2
/tmp                        /oldy-root/tmp  none bind               0       2
/mnt                        /oldy-root/mnt  none bind               0       2
/media                      /oldy-root/media none bind              0       2

Most notable are /mnt and /media. Bind-mounting these allows temporary mounts to be visible at both sides. /tmp is required for the UNIX domain socket used for playing sound from the old system. And other sockets, I suppose.

Note that /run isn’t bind-mounted. The reason is that the tree structure has changed, so it’s quite pointless (the mounting point used to be /var/run, and the place of the runtime files tend to change with time). The motivation for bind mounting would have been to let software from the old and new software interact, and indeed, there are a few UNIX sockets there, most notably the DBus domain UNIX socket.

But DBus is a good example of how hopeless it is to bind-mount /run: Old software attempting to talk with the Console Kit on the new DBus server fails completely at the protocol level (or namespace? I didn’t really dig into that).

So just copy the old /var/run into the root filesystem and that’s it. CUPS ran smoothly, GUI programs run fairly OK, and sound is done through a UNIX domain socket as suggested in the comments of this post.

I opted out on bind mounting /lib/modules and /usr/src. This makes manipulations of kernel modules (as needed by VMware, for example) impossible from the old system. But gcc is outdated for compiling under the new Linux kernel build system, so there was little point.

/root isn’t bind-mounted either. I wasn’t so sure about that, but in the end, it’s not a very useful directory. Keeping them separate makes the shell history for the root user distinct, and that’s actually a good thing.

Make /dev/log for real

Almost all service programs (and others) send messages to the system log by writing to the UNIX domain socket /dev/log. It’s actually a misnomer, because /dev/log is not a device file. But you don’t break tradition.

WARNING: If the logging server doesn’t work properly, Linux will fail to boot, dropping you into a tiny busybox rescue shell. So before playing with this, reboot to verify all is fine, and then make the changes. Be sure to prepare yourself for reverting your changes with plain command-line utilities (cp, mv, cat) and reboot to make sure all is fine.

In Mint 19 (and forever on), logging is handled by systemd-journald, which is a godsend. However for some reason (does anyone know why? Kindly comment below), the UNIX domain socket it creates is placed at /run/systemd/journal/dev-log, and /dev/log is a symlink to it. There are a few bug reports out there on software refusing to log into a symlink.

But that’s small potatoes: Since I decided not to bind-mount /run, there’s no access to this socket from the old system.

The solution is to swap the two: Make /dev/log the UNIX socket (as it was before), and /run/systemd/journal/dev-log the symlink (I wonder if the latter is necessary). To achieve this, copy /lib/systemd/system/systemd-journald-dev-log.socket into /etc/systemd/system/systemd-journald-dev-log.socket. This will make the latter override the former (keep the file name accurate), and make the change survive possible upgrades — the file in /lib can be overwritten by apt, the one in /etc won’t be by convention.

Edit the file in /etc, in the part saying:

[Socket]
Service=systemd-journald.service
ListenDatagram=/run/systemd/journal/dev-log
Symlinks=/dev/log
SocketMode=0666
PassCredentials=yes
PassSecurity=yes

and swap the files, making it

ListenDatagram=/dev/log
Symlinks=/run/systemd/journal/dev-log

instead.

All in all this works perfectly. Old programs work well (try “logger” command line utility on both sides). This can cause problems if the program expects “the real thing” on /run/systemd/journal/dev-log. Quite unlikely.

As a side note, I had this idea to make journald listen to two UNIX domain sockets: Dropping the Symlinks assignment in the original .socket file, and copying it into a new .socket file, setting ListenDatagram to /dev/log. Two .socket files, two UNIX sockets. Sounded like a good idea, only it failed with an error message saying “Too many /dev/log sockets passed”.

Running old services

systemd’s take on sysV-style services (i.e. those init.d, rcN.d scripts) is that when systemctl is called with reference to a service, it first tries with its native services, and if none is found, it looks for a service of that name in /etc/init.d.

In order to run old services, I wrote a catch-all init.d script, /etc/init.d/oldy-chrooter. It’s intended to be symlinked to, so it tells which service it should run from the command used to call it, then chroots, and executes the script inside the old system. And guess what, systemd plays along with this.

The script follows. Note that it’s written in Perl, but it has the standard info header, which is required on init scripts. String manipulations are easier this way.

#!/usr/bin/perl
### BEGIN INIT INFO
# Required-Start:    $local_fs $remote_fs $syslog
# Required-Stop:     $local_fs $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# X-Interactive:     false
# Short-Description: Oldy root wrapper service
# Description:       Start a service within the oldy root
### END INIT INFO

use warnings;
use strict;

my $targetroot = '/oldy-root';

my ($realcmd) = ($0 =~ /\/oldy-([^\/]+)$/);

die("oldy chroot delegation script called with non-oldy command \"$0\"\n")
  unless (defined $realcmd);

chroot $targetroot or die("Failed to chroot to $targetroot\n");

exec("/etc/init.d/$realcmd", @ARGV) or
  die("Failed to execute \"/etc/init.d/$realcmd\" in oldy chroot\n");

To expose the chroot’s httpd service, make a symlink in init.d:

# cd /etc/init.d/
# ln -s oldy-chrooter oldy-httpd

And then enable with

# systemctl enable oldy-httpd
oldy-httpd.service is not a native service, redirecting to systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable oldy-httpd

which indeed runs /lib/systemd/systemd-sysv-install, a shell script, which in turn runs /usr/sbin/update-rc.d with the same arguments. The latter is a Perl script, which analyzes the init.d file, and, among others, parses the INFO header.

The result is the SysV-style generation of S01/K01 symbolic links into /etc/rcN.d. Consequently, it’s possible to start and stop the service as usual. If the service isn’t enabled (or disabled) with systemctl first, attempting to start and stop the service will result in an error message saying the service isn’t found.

It’s a good idea to install the same services on the “main” system and disable them afterwards. There’s no risk for overwriting the old root’s installation, and this allows installation and execution of programs that depend on these services (or they would complain based upon the software package database).

Running programs

Running stuff inside the chroot should be quick and easy. For this reason, I wrote a small C program, which opens a shell within the chroot when called without argument. With one argument, it executes it within the chroot. It can be called by a non-root user, and the same user is applied in the chroot.

This is compiled with

$ gcc oldy.c -o oldy -Wall -O3

and placed /usr/local/bin with setuid root:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>
#include <sys/types.h>
#include <pwd.h>

int main(int argc, char *argv[]) {
  const char jail[] = "/oldy-root/";
  const char newhome[] = "/oldy-root/home/eli/";
  struct passwd *pwd;

  if ((argc!=2) && (argc!=1)){
    printf("Usage: %s [ command ]\n", argv[0]);
    exit(1);
  }

  pwd = getpwuid(getuid());
  if (!pwd) {
    perror("Failed to obtain user name for current user(?!)");
    exit(1);
  }

  // It's necessary to set the ID to 0, or su asks for password despite the
  // root setuid flag of the executable

  if (setuid(0)) {
    perror("Failed to change user");
    exit(1);
  }

  if (chdir(newhome)) {
    perror("Failed to change directory");
    exit(1);
  }

  if (chroot(jail)) {
    perror("Failed to chroot");
    exit(1);
  }

  // oldycmd and oldyshell won't appear, as they're overridden by su

  if (argc == 1)
    execl("/bin/su", "oldyshell", "-", pwd->pw_name, (char *) NULL);
  else
    execl("/bin/su", "oldycmd", "-", pwd->pw_name, "-c", argv[1], (char *) NULL);
  perror("Execution failed");
  exit(1);
}

Notes:

• Using setuid root is a number one for security holes. I’m not sure I would have this thing on a computer used by strangers.
• getpwuid() gets the real user ID (not the effective one, as set by setuid), so the call to “su” is made with the original user (even if it’s root, of course). It will fail if that user doesn’t exist.
• … but note that the user in the chroot system is then one having the same user name as in the original one, not uid. There should be no difference, but watch it if there is (security holes…?)
• I used “su -” and not just executing bash for the sake of su’s “-” flag, which sets up the environment. Otherwise, it’s a mess.

It’s perfectly OK to run GUI programs with this trick. However it becomes extremely confusing with command line. Is this shell prompt on the old or new system? To fix this, edit /etc/bashrc in the chroot system only to change the prompt. I went for changing the line saying

[ "$PS1" = "\\s-\\v\\\$ " ] && PS1="[\u@\h \W]\\$ "

to

[ "$PS1" = "\\s-\\v\\\$ " ] && PS1="\[\e[44m\][\u@chroot \W]\[\e[m\]\\$ "

so the “\h” part, which turns into the host’s name now appears as “chroot”. But more importantly, the text background of the shell prompt is changed to blue (as opposed to nothing), so it’s easy to tell where I am.

If you’re into playing with the colors, I warmly recommend looking at this.

Lifting the user processes limit

At some point (it took a few months), I started to have failures of this sort:

$ oldy
oldyshell: /bin/bash: Resource temporarily unavailable

and even worse, some of the chroot-based utilities also failed sporadically.

Checking with ulimit -a, it turned out that the limit for the number of processes owned by my “regular” user was limited to 1024. Checking with ps, I had only about 510 processes belonging to that UID. So it’s not clear why I hit the limit. In the non-chroot environment, the limit is significantly higher.

So edit /etc/security/limits.d/90-nproc.conf (the one inside the jail), changing the line saying

-*          soft    nproc     1024

to

*          soft    nproc     65536

There’s no need for any reboot or anything of that sort, but the already running processes remain within the limit.

Desktop icons and wallpaper messup

This is a seemingly small, but annoying thing: When Nautilus is launched from within the old system, it restores the old wallpaper and sets all icons on the desktop. There are suggestions on how to fix it, but they rely on gsettings, which came after Fedora 12. Haven’t tested this, but is the common suggestion is:

$ gsettings set org.gnome.desktop.background show-desktop-icons false

So for old systems as mine, first, check the current value:

$ gconftool-2 --get /apps/nautilus/preferences/show_desktop

and if it’s “true”, fix it:

$ gconftool-2 --type bool --set /apps/nautilus/preferences/show_desktop false

The settings are stored in ~/.gconf/apps/nautilus/preferences/%gconf.xml.

Setting title in gnome-terminal

So someone thought that the possibility to set the title in the Terminal window, directly from the GUI,  is unnecessary. That happens to be one of the most useful features, if you ask me. I’d really like to know why they dropped that. Or maybe not.

After some wandering around, and reading suggestions on how to do it in various other ways, I went for the old-new solution: Run the old executable in the new system. Namely:

# cd /usr/bin
# mv gnome-terminal new-gnome-terminal
# ln -s /oldy-root/usr/bin/gnome-terminal

It was also necessary to install some library stuff:

# apt install libvte9

But then it complained that it can’t find some terminal.xml file. So

# cd /usr/share/
# ln -s /oldy-root/usr/share/gnome-terminal

And then I needed to set up the keystroke shortcuts (Copy, Paste, New Tab etc.) but that’s really no bother.

Other things to keep in mind

• Some users and groups must be migrated from the old system to the new manually. I do this always when installing a new computer to make NFS work properly etc, but in this case, some service-related users and groups need to be in sync.
• Not directly related, but if the IP address of the host changes (which it usually does), set the updated IP address in /etc/sendmail.mc, and recompile. Or get an error saying “opendaemonsocket: daemon MTA: cannot bind: Cannot assign requested address”.

For some unknown reason, possibly after an VMplayer upgrade, running any Windows Virtual machine on my Linux machine with WMware Player caused some non-stop heavy hard disk activity, even when the guest machine was effectively idle, and made had no I/O activity of its own.

Except for being surprisingly annoying, it also made the mouse pointer non-responsive and the effect was adverse on the hosting machine as well.

isolation.tools.unity.disable = "TRUE"
unity.allowCompositingInGuest = "FALSE"
unity.enableLaunchMenu = "FALSE"
unity.showBadges = "FALSE"
unity.showBorders = "FALSE"
unity.wasCapable = "FALSE"

MemTrimRate = "0"
sched.mem.pshare.enable = "FALSE"
MemAllowAutoScaleDown = "FALSE"

mks.noGL = "FALSE"

$ vncviewer :1 &

# echo never > /sys/kernel/mm/transparent_hugepage/defrag
# echo never > /sys/kernel/mm/transparent_hugepage/enabled

Linux namespaces is the foundation for container-based virtualization, which is becoming increasingly popular. Aside from the ability to isolate a shell (and the processes it generates) from the “main environment”, as is required for this kind of lightweight virtualization, namespaces is useful for overriding selected functionalities.

So I’m jotting down things I use myself.

Using an ad-hoc host name

[root@mycomputer eli]# uname -n
mycomputer.localdomain
[root@mycomputer eli]# unshare -u bash
[root@mycomputer eli]# uname -n
mycomputer.localdomain
[root@mycomputer eli]# hostname newname.localdomain
[root@mycomputer eli]# uname -n
newname.localdomain
[root@mycomputer eli]# exit
[root@mycomputer eli]# uname -n
mycomputer.localdomain

Note that unshare started a new bash shell, with a separate namespace. That’s why hostname’s effect ended when pressing CTRL-D and exiting the shell (where it says “exit”).

The truth is, that “unshare [options] bash” and just “unshare [options]” do the same thing — in the absence of a program to execute, unshare kicks off a new shell.

Hiding the network from a program

Some programs have an annoying “feature” of reporting back (hopefully anonymous) information to its vendor’s server. If it has nothing to do on the internet, it can be run from a subshell that can’t possibly see any network interface. Except for loopback, of course. For example,

# unshare -n bash
# ifconfig lo up
# ifconfig
lo        Link encap:Local Loopback 
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

I should mention that iptables allows rules per user, and there’s always SELinux for those with strong nerves, so these are alternative solutions. But a network namespace looks much simpler to me.

The loopback interface is absent unless enabled as shown above.

In a more sophisticated version, one can add a tunnel interface to the “real world” and apply iptables rules on the dedicated interface. Note that iptables is local to the new network namespace, so it’s empty. The firewall rules can therefore be written inside the network namespace so the main firewall isn’t touched, or rules can be set up on the main firewall for the tunnel interface in the “real world” network namespace.

The procedure, if you don’t want to install VMware on the computer you’re working on, is to first extract the files with something like (as a non-root user)

$ bash VMware-Player-6.0.2-1744117.x86_64.bundle --extract newplayer

which just writes the files into a new directory newplayer/.

Next step is to untar the relevant directories into a fresh working directory. May I suggest:

for i in /path/to/newplayer/vmware-vmx/lib/modules/source/*.tar ; do tar -xvf $i ; done

This creates five new directories, each containing a Makefile and kernel module sources. In principle, the goal is to type “make” in all five and not have an error.

That’s where the headache is. I’ve packed up a set of patches that took me from Player 6.0.2 (or Workstation 10.0.2) to kernel 3.12 (in a rather messy way, but hey, nothing about it is really in order): Here they are as a tarball.

Once that is done, wrap it all up with

$ for i in vmblock vmci vmmon vmnet vsock ; do tar -cf $i.tar $i-only ; done

and copy the *.tar files into /usr/lib/vmware/modules/source/ (actually, replace the existing files) in an existing installation.

And then just run VMware as usual, and behave nicely when it wants to install modules.

Or if you want to see it with your own eyes (as root):

# vmware-modconfig --console --install-all

VMCI issues

As if it wasn’t enough as is, there was a problem with running the VMCI:

Starting VMware services:
 Virtual machine monitor                                 [  OK  ]
 Virtual machine communication interface                 [FAILED]
 VM communication interface socket family                [  OK  ]
 Blocking file system                                    [  OK  ]
 Virtual ethernet                                        [  OK  ]
 VMware Authentication Daemon                            [  OK  ]

And of course, so virtual machine would agree to run this way.

After a lot of messing around, it turned out that for some reason, the module wasn’t installed. Go figure.

The solution was to go back to the vmci-only directory (one of the untarred one above), compile it with “make” (it should work by now, after all), and then copy it to the running kernel’s module repository:

# cp vmci.ko /lib/modules/$(uname -r)/kernel/drivers/misc/
# depmod -a

Or maybe just create a kernel/drivers/vmware directory and copy all *.ko files that were compiled, and depmod.

I. Never. Want. To hear. About. This. Again.

It’s sometimes desired to run an X-Windows program in a separate “screen” but not actually have another screen. The expensive way is to bring up a whole virtual server. But if it’s fine to run the program on the same computer, all we want is to have a window, in which the program is confined.

This is handy if the program has a tendency to steal focus with popups all the time.

It’s also useful for opening windows from a remote machine, and the regular X server refuses despite being generous with “xhost +”. The nested server isn’t picky with who it’s hosting.

Some installations

First, install Xnest if it’s not already installed, e.g. (as root)

# yum install Xnest

It’s also possible to install a very simple (and somewhat yucky) window manager

# yum install twm

Action

Then open a new window, which turns into a new X server:

$ Xnest -s 0 -ac :1 -geometry 1900x1020+5+0&

The dimensions given by the “geometry” flag are those making a full screen coverage on my monitor. This varies, of course.

Launch a Window Manager and a terminal window in the new X server. The former is necessary to make windows movable, resizable, etc.

$ twm -display :1
$ DISPLAY=:1 gnome-terminal &

Note that apparently nothing happens after launching the first command, because there are no clients in the Xnest window.

And then use the terminal to run applications inside the nested X-window server.

twm too yucky?

The Gnome Window Manager can be used instead of the command issuing twm:

$ DISPLAY=:1 gnome-wm &

The reason not to use Gnome’s window manager is that it allows minimizing windows. If that is done accidentally, the window is lost (unless a bottom panel is generated, which starts to get a bit messy for a simple task).

These are my own notes as I set up an OpenVZ VPS server, based upon CentOS 5.6 to function as a web and mailing list server. A $36/year 128 MB RAM machine was good enough for this.

Since there’s some criticism about the hosting provider, and it looks like they’re OK after all, I’m leaving their name out for now. The main purpose of this post is to help myself getting started again, if that is ever necessary (I sure hope it will never be).

Foul #1: Mails from hosting provider marked as spam

This is the first time it happens to me that automated emails from any service providers go to Gmail’s spam box. That includes the welcome mails as I subscribed, the payment confirmation and the message telling me the server is up. And messages about support tickets. None arrived.

Spamassassin gives these mails some points (1.3 or so) as well. I’ve hardly seen anything like this from any decent automatic mail producer. I first thought this was a major blunder, but then it turns out that machine-generated emails tend to get caught by spam filters. Since email messages that are relayed by a mailing list (mailman) don’t get caught, it looks like the spam filter checks the “received” chain of headers for the first hops of the message, and tries to figure out if that’s a decent ISP there. Just a wild guess.

Workaround: Add a filter in Gmail to never send emails from *@the-hosting-provider.com to the spam box. Simple, when you know about it.

Foul #2: 12 hours from payment to server running

Even for a low-cost service, 12 hours of “pending” is a bit too much. In particular when $36 have been paid. That alone filters out most scammers, I suppose.

Foul #3: Root password not set

Maybe I was naive to expect that the root password would be set in the server, so I tried to SSH the server with the password I had assigned during the subscription, but was consistently denied.

Workaround: Enter the VPS control panel and change the password.

Foul #4: Uncertified HTTPS link

The control panel of the VPS is accessed with a link to an IP address. Which is a bit weird, but let’s leave that alone. I mean, what about buying a domain for that purpose? To make things even worse, they supply an HTTPS link as well. Which works, but makes the browser display a scare “GET ME OUT OF HERE” message.

An uncertified HTTPS link is better than HTTP, even though cryptologists will argue that in the absence of a certificate, a man-in-the-middle attack is possible. But let’s get serious. It’s not really dangerous. It’s just yet another sign that they don’t give a shit. Setting up a domain and certifying it something you would expect from any serious company, just to avoid that scary warning message. But they didn’t.

Bump #1: Lacking yum repository

Among the first thing I did after logging in (because I’m addicted):

# yum install git
Loaded plugins: fastestmirror
Determining fastest mirrors
 * base: mirror01.th.ifl.net
 * extras: mirror01.th.ifl.net
 * updates: mirror01.th.ifl.net
base                                                    | 1.1 kB     00:00    
base/primary                                            | 967 kB     00:00    
base                                                                 2725/2725

[ ... yada yada ... ]

vz-updates/primary                                      | 1.0 kB     00:00    
vz-updates                                                                 3/3
Setting up Install Process
No package git available.
Nothing to do

Are you kidding me? Using

# rpm -qa --last | head

I got a list of packages installed, many of which were marked with “el5″, which isn’t surprising,

# cat /etc/redhat-release
CentOS release 5.6 (Final)

since it’s a CentOS 5 distro (EL = Enterprise Linux).

The list of existing repos:

# yum repolist
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror01.th.ifl.net
 * extras: mirror01.th.ifl.net
 * updates: mirror01.th.ifl.net
repo id                                repo name                                      status
base                                   CentOS-5 - Base                                enabled: 2,725
extras                                 CentOS-5 - Extras                              enabled:   286
updates                                CentOS-5 - Updates                             enabled: 1,003
vz-base                                vz-base                                        enabled:     5
vz-updates                             vz-updates                                     enabled:     3
repolist: 4,02

So where’s git? On Fedora 12, I checked where git was loaded from (for comparison) with

$ yumdb info 'git*'
Loaded plugins: presto, refresh-packagekit
git-1.7.2.3-1.fc12.x86_64
 changed_by = 1010
 checksum_data = 470af233244731e51076c6aac5007e1eebd2f73f23cd685db7cd8bd6fb2b3dd1
 checksum_type = sha256
 command_line = install git-email
 from_repo = updates
 from_repo_revision = 1291265770
 from_repo_timestamp = 1291266900
 reason = user
 releasever = 12

[ ... here comes info about git-deamon and other packages ]

So CentOS’ repository doesn’t have git? That looks odd to me. A last try:

# yum list available | less

No, git wasn’t on the list. The fix was to add Repoforge to the list of repositories on the server (following the instructions):

# wget http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.2-2.el5.rf.i386.rpm
# rpm -i rpmforge-release-0.5.2-2.el5.rf.i386.rpm

And then “yum install git” went fine.

Bump #2: Bad host name

OK, it’s partly my fault. A short host name (without a dot) isn’t good enough. At least not for sending mails. A fully-qualified host name (such as example.com, as opposed to just “example”) is needed. Or sendmail starts up very slowly and then refuses to send mails.

Bump #3: Setting up reverse DNS

For the server to be able to send emails that aren’t immediately detected as spam, its IP must have the reverse DNS set to its host name.

There is a place to edit the rDNS name in the control panel (under “Network”) but it was disabled. Contact support, it said. So I did.

Having rDNS disabled by default is somewhat understandable to at least keep an eye on spammers. On the other hand, show me a spammer paying $36 upfront.

It took support 11 hours to answer this support request, asking me to supply the rDNS record I needed for manual setting. The actual fix came an hour later, so overall this was fairly OK.

The automatic feature is simply not supported. But it’s not like decent people need to change their rDNS every day.

Bump #4: No swap

It seems like there is no way to activate a swap file on the VPS server (in particular, losetup returns with “permission denied” so there is nothing to attach the swap partition to). So there’s no choice than to make sure that the overall memory consumption doesn’t exceed the allocation of virtual RAM, which is 128 MB in my case. Or processes will just die. I can understand the commercial sense in this limitation: If users would start putting large swap files on their systems, they would buy lower-cost machines and then complain that they’re not responsive.

The figure to keep track of is the amount of free + cached memory. For example,

$ cat /proc/meminfo
MemTotal:         131072 kB
MemFree:           47516 kB
Cached:            27156 kB
[ ... ]

The free memory is 47416 + 27156 = 74672 kB, which means 131072 – 74672 = 56400 kB is used. These are the figures that appear in the Control Panel.

Installation note: Setting up sendmail

By default, sendmail doesn’t accept external connections. Edit /etc/mail/sendmail.mc, changing

DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')dnl

to

DAEMON_OPTIONS(`Port=smtp, Name=MTA')dnl

This removes the restriction that only the local address is listened to. And also from

dnl DAEMON_OPTIONS(`Port=submission, Name=MSA, M=Ea')dnl

to

DAEMON_OPTIONS(`Port=submission, Name=MSA, M=Ea')dnl

(remove the “dnl” hence uncommenting the line).

Check with the hosting provider if they supply a mail relay server. Relaying through a well-reputed server can decrease the spam score of your mails. Besides, the hosting provider can decide to block all outgoing direct connections with mail servers out of the blue, because spams were flying out from their servers.

A line like the following should be added (possibly close to where SMART_HOST is mentioned in the file).

define(`SMART_HOST',`relay.myprovider.com')dnl

And then compile the file into sendmail.cf, and restart the server as follows:

# make -C /etc/mail
# service sendmail restart

Then test the server, possibly using a script. In particular, verify that the mail server isn’t relaying (accepting messages to other domains) or the server turns into a spam machine.

Installation note: Mailman

# yum install mailman

Then customize. See instructions here and also have a look at /usr/share/doc/mailman-2.1.9/INSTALL.REDHAT. There’s a need to access the host by its domain name (as opposed to just the IP address) so the local computer’s /etc/hosts may need to be fiddled with when working on a server not yet allocated with the new address.

Note: Do not edit /usr/lib/mailman/Mailman/mm_cfg.py for changing DEFAULT_URL_HOST and DEFAULT_EMAIL_HOST, if they happen to say

DEFAULT_URL_HOST   = fqdn
DEFAULT_EMAIL_HOST = fqdn

because in this case they’re set up automagically.

First, make sure that the mailman daemon is off:

# service mailman stop

To migrate a few lists from one server to another, copy the respective lists in /var/lib/mailman/{lists,archives} into the new server. Note that the “data” directory doesn’t contain any information on the lists, so therefore just adding these directories is enough.

The lists will not appear in the web interface if there was a domain switch during the list migration, as can be observed by searching for ‘web_page_url’ in the output of

# /usr/lib/mailman/bin/dumpdb config.pck

To fix this, go (for each list)

# /usr/lib/mailman/bin/withlist -l -r fix_url the-list-name

Make sure the files are owned by mailman with

# chown -R mailman:mailman ... the copied directories ...

At this point, the list should appear on the web console. Copy the entries into /etc/aliases, more or less like this:

## listname mailing list
listname:              "|/usr/lib/mailman/mail/mailman post listname"
listname-admin:        "|/usr/lib/mailman/mail/mailman admin listname"
listname-bounces:      "|/usr/lib/mailman/mail/mailman bounces listname"
listname-confirm:      "|/usr/lib/mailman/mail/mailman confirm listname"
listname-join:         "|/usr/lib/mailman/mail/mailman join listname"
listname-leave:        "|/usr/lib/mailman/mail/mailman leave listname"
listname-owner:        "|/usr/lib/mailman/mail/mailman owner listname"
listname-request:      "|/usr/lib/mailman/mail/mailman request listname"
listname-subscribe:    "|/usr/lib/mailman/mail/mailman subscribe listname"
listname-unsubscribe:  "|/usr/lib/mailman/mail/mailman unsubscribe listname"

And finally, turn the service on again:

# service mailman start

And make it a permanent service:

# chkconfig mailman on

Changing the subscription confirmation message (to tell users to look in the spam folder): Edit /usr/lib/mailman/templates/en/subscribe.html, and remove /var/lib/mailman/lists/{listname}/en/subscribe.html (if it exists, and contains nothing special for the list).

Then restart qrunner to flush cached templates:

# /usr/lib/mailman/bin/mailmanctl restart

See these two pages for more info about this (even though they’re not very accurate).

Log files

There are definitely two files one wants to peek at every now and then

• /var/log/libvirt/qemu/{guest-name}.log (in particular when a USB device doesn’t attach)
• /var/log/audit/audit.log (SELinux audit, possibly piped to grep -E ‘^type=(AVC|SELINUX\_ERR)’ to reduce amount of junk)

Start with running virt-manager as a non-root user (it will fail with a nondescriptive message if trying to run it as root). A root password will have to be fed.

Use qcow2 disk images

If you’re going to play around with the image, and then maybe want to get rid if the changes, this is sooo simple:

#qemu-img create -F raw -b clean-hda.img -f qcow2 hda.qcow2

Note that qemu-img can also create and apply snapshots of images, which is also good.

Don’t try to use Virtual Machine manager’s clone utility for qcow2 images, since the tool will rewrite all data.

USB device passtrough

It looks like libvirt doesn’t massage the permissions for USB character devices, before adopting them, so both classic permissions and SELinux blow when trying to run passthrough. The current workaround would be to change the USB device’s classic permission manually, and run in permissive mode. Which is indeed unhealthy, but harmless given the fact that XP gives a healthy blue screen in response to the new device. I think I saw XP complaining something about USB 2.0 versus USB 1.1.

Running the same scenario with a knoppix LiveCD image, I managed to find the device (a Canon 500D camera) with lsusb. The root USB hub was shown to be a UHCI. I’m not sure whether a real PTP camera would respond to a UHCI hub, which maybe explains why Windows got confused finding such a camera under the emulated hub.

The emulator appears as /usr/bin/qemu-kvm under the SELinux domain (type) svirt_t, which is decleared in /usr/share/selinux/devel/include/services/virt.if (which you don’t want to read).

USB devices appear somewhere under /dev/bus/usb/ as character devices with SELinux type usb_device_t.

Command line

Create a guest, run it and pause it. Then dump its XML by something like (as root):

# virsh dumpxml try1 > ~eli/virt/try1.xml

Then play around with the XML file a bit, destroy the previous domain, and

# virsh create ~eli/virt/try2.xml

How can I manually manipulate the list of guests?

virsh # start demo
error: Failed to start domain demo
error: internal error unable to start guest: qemu: could not open disk image /home/eli/virt/tryxp.img: Permission denied

Reason(?): tryxp.img is of the wrong context, so SELinux prevents it to be opened…? But I ran SELinux in permissive mode. How could this happen? Or, as put in the Fedora 12 Virtualization Guide:

SELinux prevents guest images from loading if SELinux is enabled and the images are not in the correct directory. SELinux requires that all guest images are stored in /var/lib/libvirt/images.

Basically, what solved this was to move the image to the dedicated directory, and going:

# virt-install --force --name demo3 --ram 1024 --import --disk path=/var/lib/libvirt/images/tryxp.img

or even better:

# virt-install –force –name demo5 –ram 1024 –import –disk path=/var/lib/libvirt/images/hda.img

For playing around:

# virsh

“Stealing” command lines from Virtual Machine Manager

After running a machine under the GUI interface, it’s possible to do so for running on an external VNC console. Just find the command with ps aux | grep qemu-kvm. The following changes in flags apply:

• Remove the -S flag. It says that the guest should not start until commanded to do so.
• Remove the -montitor flag. We’re running on VNC only
• Change the -vnc flag’s address to point to an address known to the outside world, if necessary
• Add “-usbdevice tablet” so that the mouse is followed correctly
• Change the -net flags (two of them) to “-net nic -net user”. This is said to have a performance hit, but it’s simple and it works with an internal (fake) DHCP server

The tip of the day

In the relevant terminology, “source” refers to information seen by the host OS, while “target” to the guest OS.

Another little tip: If I try to install Windows 7, and the installation gets stuck for very long periods of time with nothing really happening, maybe it’s because the disk image is read-only? :-O

VMPlayer

Running VMplayer on a 2.6.35 kernel requires a small fix, which was published here. The thing is that some kernel symbol has changed its name, and hence the vmmon module fails to compile. How I love when people are sensitive about backward compatibility.

To make a long story short, one needs to go to where the VMplayer module sources are, and go:

$ perl -pi -e 's,_range,,' iommu.c

which is bit of a Columbus egg, I would say. Also, VMplayer repeatedly wanted to compile the modules every time I started it, because it missed vsock (which wasn’t compiled in the first place), so I followed the same page‘s hint and edited /etc/vmware/config to say

VSOCK_CONFED = "no"

By the way, I tried to figure out what this module does, and all google results tell you how to tweak and patch. Don’t people care what they do on their computers? Maybe this component is useful?

The following remark was true when this post was published, but no more:

To run VMPlayer under Fedora 12, there’s a need for a little hack, or the VMPlayer closes right after starting:

# cd /usr/lib/vmware/resources/
# mv mozilla-root-certs.crt old-mozilla-root-certs.crt

Have no idea why this is.

VMPlayer networking

The interesting stuff is at /etc/vmware/networking, which pretty much says which interface is NATed and which is host-only. To pick a certain device for bridging, additional lines configuring add_bridge_mapping should be added as explained on this page.

Also useful to play with

# vmware-networks --status
# vmware-networks --start
# vmware-networks --stop

etc. (as root)

Moving an old Linux computer

The mission: Move an old 2.4.21-kernel based computer into a VMPlayer virtual machine. The strategy was to run a LiveCD on the virtual machine, create an empty ext3 file system on it, copy the files and boot. Caveats follow.

The most important lesson learned, is that everything on the new machine has to be done with a kernel with the same or earlier version compared with the one that will be used. In simple words, the rescue disk must run a matching kernel. In particular, with a newer rescue disk, the ext3 is generated with an inode size of 256, which old kernels don’t support. Even worse, even if the file system was generated properly, newer kernels (say, 2.6.35) writes things on the disk that will confuse the old kernel. This leads to “attempt to access beyond end of device” errors during boot, from 01:00 (the ramdisk) as well as 08:01 (the root filesystem’s partition).

So fdisk, mkfs.ext3 and mkinitrd must be done with a matching kernel running. And copying the files too, of course. The rescue disk must match, as mentioned above.

The next thing to note is that all hda’s turn into sda’s. That needs to be adjusted in /etc/fstab as well as /etc/lilo.conf.

The most difficult thing to handle was the fact that SCSI drivers were not installed in the kernel by default, so the initrd image had to be adjusted. So after the file system is copied, mount it, chroot to it, and go

# mkinitrd --with=scsi_mod --with=sd_mod --with=BusLogic /boot/initrd-2.4.21-mykern.vmware.img 2.4.21-mykern

The insmods are attempted in the same order the flags appear, and the two latter modules depend on the scsi_mod. So it’s important to keep the order of the flags as above.

If these modules aren’t loaded, the generation of /dev/sda and /dev/sda1 doesn’t occur, resulting in a kernel panic with various complains (pivot_mount fails, init not found and some more).

And then fix /etc/lilo.conf and /etc/fstab and run lilo. It’s recommended to copy the /boot directory first, so that the kernel image falls within the lower 4 GB. Or pick lba32 option, as lilo will tell you.

And then boot. Mount the ISO image for VMware tools (/dev/hdc) and run ./vmware-install (going with the defaults most of time).

Converting a VMPlayer machine to VirtualBox

Make a copy of the entire directory to a new place. No point messing up things. Then create an OVF file, without converting the disks (because it takes forever, and possibly fails)

$ ovftool --noDisks CleanXP.vmx export.ovf

For some reason, ovftool doesn’t fill in the correct names of the disks (I asked not to convert them, not to ignore them). The simplest way around this is to remove the disk definitions altogether, and import them manually from Virtualbox. For a VM with three disks, the lines marked in red should be removed:

 <References>
 <File ovf:href="export-disk1.vmdk" ovf:id="file1" ovf:size="0"/>
 <File ovf:href="export-disk2.vmdk" ovf:id="file2" ovf:size="0"/>
 <File ovf:href="export-disk3.vmdk" ovf:id="file3" ovf:size="0"/>
 </References>
 <DiskSection>
 <Info>Virtual disk information</Info>
 <Disk ovf:capacity="10" ovf:capacityAllocationUnits="byte * 2^30" ovf:diskId="vmdisk1" ovf:fileRef="file1" ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" ovf:populatedSize="0"/>
 <Disk ovf:capacity="30" ovf:capacityAllocationUnits="byte * 2^30" ovf:diskId="vmdisk2" ovf:fileRef="file2" ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" ovf:populatedSize="0"/>
 <Disk ovf:capacity="40" ovf:capacityAllocationUnits="byte * 2^30" ovf:diskId="vmdisk3" ovf:fileRef="file3" ovf:format="http://www.vmware.com/interfaces/specifications/vmdk.html#streamOptimized" ovf:populatedSize="0"/>
 </DiskSection>

and also the other references to these vmdiskn:

<Item>
 <rasd:AddressOnParent>0</rasd:AddressOnParent>
 <rasd:ElementName>disk0</rasd:ElementName>
 <rasd:HostResource>ovf:/disk/vmdisk1</rasd:HostResource>
 <rasd:InstanceID>7</rasd:InstanceID>
 <rasd:Parent>6</rasd:Parent>
 <rasd:ResourceType>17</rasd:ResourceType>
 </Item>
 <Item>
 <rasd:AddressOnParent>1</rasd:AddressOnParent>
 <rasd:ElementName>disk1</rasd:ElementName>
 <rasd:HostResource>ovf:/disk/vmdisk2</rasd:HostResource>
 <rasd:InstanceID>8</rasd:InstanceID>
 <rasd:Parent>6</rasd:Parent>
 <rasd:ResourceType>17</rasd:ResourceType>
 </Item>
<Item>
 <rasd:AddressOnParent>0</rasd:AddressOnParent>
 <rasd:ElementName>disk2</rasd:ElementName>
 <rasd:HostResource>ovf:/disk/vmdisk3</rasd:HostResource>
 <rasd:InstanceID>10</rasd:InstanceID>
 <rasd:Parent>4</rasd:Parent>
 <rasd:ResourceType>17</rasd:ResourceType>
 </Item>

Delete the export.mf file. It contains export.ovf’s hash signature, and it’s wrong after editing the file.

Then in the VM VirtualBox Manager, pick File > Import Appliance… and choose export.ovf.

Then add the hard disks by choosing Storage > (Diskette Icon) > Add Hard Disk and pick Choose existing disk. Pick the .vmdk file with no number attached to it (not the e.g. *-s011.vmdk).

Enlarge the video memory to 18 MB (or more), or Virtualbox complains that it’s too little. Enable audio.

That was nice so far. Only problem is that Windows required re-activation because too much hardware changed, and the internet activation failed, probably because the NIC wasn’t detected. To install the driver, I’ll need to activate first, and so I was stuck, lost patience and left the whole thing for now.