A watchdog script for restarting Cinnamon hogging memory at startup
Introduction
August 2019 update: I’ve disactivated the service described below on my own machine, because in the end, the system didn’t recover properly from the condition it was supposed to solve. No solution, in the end.
Having installed Linux Mint 18.1 (kernel 4.4.0-53) with overlayroot on a Gigabyte Brix BACE-3160 (see other notes on this here and here) for my living room media center, I had an issue with bringing up the computer every now and then. Namely, I had a blank screen with mouse pointer only, 100% CPU on the cinnamon process, and some 50% on cinnamon-settings-daemon, eating all RAM (8 GB) gradually until the computer crashes (as it’s swapless).
The whole point with an overlayroot based media center is that it behaves like any electrical appliance. In particular, you know it will recover well from a power outage. And not crash because of some silly desktop manager messing up.
I’m not alone: There’s a bug report on this. There’s also this Git Hub issues page seems to point at the problem, in particular as it was closed pointing at a commit, which was merged into the main repository saying “Add detection for accountsservice background as it’s ubuntu only”.
It seems to be more common with SSD disks (because they’re faster).
And still there’s no solution available, almost a year later. Don’t tell me to upgrade. Upgrading means trading known bugs for new, unknown ones.
Which leaves me with making a workaround. In this case, a watchdog service, which monitors the resident memory usage of the process having the command line “cinnamon”. If it goes above 300 MiB during its first five minutes (more or less), restard the mdm service, which in turn restarts Cinnamon. Plain and disgusting.
The ironic truth is that I haven’t been able to test the watchdog for real, as I’ve failed to repeat the problem, despite a lot of power cycles. But I have verified that it works by lowering the memory threshold, and confirmed that the mdm service is indeed restarted. I also know from previous experience that restarting mdm helps.
The stuff
This should be copied to /usr/local/bin/cinnamon-watchdog.pl (executable at least by root):
#!/usr/bin/perl
use warnings;
use strict;
my $cmd = 'cinnamon';
my $limit = 200; # Maximal resident memory size in MiB
my $timeout = 300; # Approximately how long this script should run, in seconds
my $pid;
my $timer = 0;
my $rss = 0;
eval {
while (1) {
$timer++;
if ($timer >= $timeout) {
if (defined $pid) {
logger("Quitting. cinnamon process seems to behave well with $rss MiB resident memory");
} else {
logger("Quitting without finding the cinnamon process")
}
last;
}
sleep(1);
undef $pid if ((defined $pid) && !is_target($cmd, $pid));
$pid = find_target($cmd) unless (defined $pid);
next unless (defined $pid); # No target process running yet?
$rss = resident_memory($pid);
if ($rss > $limit) {
$timer = 0;
logger("Restarting cinnamon: Has reached $rss MiB resident memory");
system('/bin/systemctl', 'restart', 'mdm') == 0
or die("Failed to restart mdm service\n");
}
}
};
if ($@) {
logger("Process died: $@");
exit(1);
}
exit(0);
######## SUBROUTINES ########
sub find_target {
my $target = shift;
foreach my $entry (</proc/*>) {
my ($pid) = ($entry =~ /^\/proc\/(\d+)$/);
next unless (defined $pid);
return $pid
if (is_target($target, $pid));
}
return undef;
}
sub is_target {
my ($target, $pid) = @_;
my ($cmdline, $c);
local $/;
open (F, "/proc/$pid/cmdline") or return 0;
$cmdline = <F>;
close F;
return 0 unless (defined $cmdline);
($c) = ($cmdline =~ /^([^\x00]*)/);
return $target eq $c;
}
sub resident_memory {
my ($pid) = @_;
local $/;
open (F, "/proc/$pid/statm") or return 0;
my $mem = <F>;
close F;
return 0 unless (defined $mem);
my @entries = ($mem =~ /(\d+)/g);
return 0 unless (defined $entries[1]); # Should never happen
return int($entries[1] / 256); # Convert pages to MiBs
}
sub logger {
my $msg = shift;
system('/usr/bin/logger', "cinnamon-watchdog: $msg") == 0
or warn("Failed to call logger\n");
}
And this service unit file to /etc/systemd/system/cinnamon-watchdog.service:
[Unit] Description=Cinnamon watchdog service [Service] ExecStart=/usr/local/bin/cinnamon-watchdog.pl Type=simple [Install] WantedBy=multi-user.target
Enable service:
# systemctl enable cinnamon-watchdog Created symlink from /etc/systemd/system/multi-user.target.wants/cinnamon-watchdog.service to /etc/systemd/system/cinnamon-watchdog.service.
That’s it. The service will be active on the next reboot
A few comments on the Perl script
- The script monitors the memory consumption because it’s a simple and safe indication of the problem, not the problem itself.
- It accesses the files under /proc/ directly rather than obtaining the information from e.g. ps. I suppose the proc format is more stable than ps’ output. The ps utility obtains its information from exactly the same source.
- The script exits voluntarily after slightly more than $timeout seconds after its invocation or its last intervention. In other words, if the cinnamon process hogs memory after its first five minutes, the script won’t be there to do anything. Neither should it. Its purpose is very specific.
- The 200 MiB limit is based upon experience with my own system: It usually ends up with 123 MiB or so.
Random vaguely related notes
Before going for this ugly solution, I actually tried to find the root of the problem (and obviously failed). Below are some random pieces of info I picked up while doing so.
The mdm setup files, in this order:
- /usr/share/mdm/defaults.conf
- /usr/share/mdm/distro.conf
- /etc/mdm/mdm.conf
Things I tried that didn’t help:
- removing Background line from /var/lib/AccountsService/users/ as suggested below (didn’t make any difference)
- Replacing autologin with a timed login of 10 seconds (I suspected that the problem was that Accountservice wasn’t up when it was required to say which background to put).
Additional ramblings
It seemed to be a result of the Accountservice daemon attempting to fiddle with the background image.
So in /var/lib/AccountsService/users/ there’s a file called “eli” (my user name, obviously), which says
[User] Background=/usr/share/backgrounds/linuxmint/default_background.jpg SystemAccount=false
So what happens if I remove the Background line? Nothing. That is, the problem remains as before.
A sample .xsession-errors when things went wrong
initctl: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/u pstart: Connection refused syndaemon: no process found /etc/mdm/Xsession: Beginning session setup... localuser:eli being added to access control list ** Message: couldn't access control socket: /run/user/1010/keyring/control: No s uch file or directory ** Message: couldn't access control socket: /run/user/1010/keyring/control: No s uch file or directory SSH_AUTH_SOCK=/run/user/1010/keyring/ssh SSH_AUTH_SOCK=/run/user/1010/keyring/ssh (cinnamon-settings-daemon:2002): power-plugin-WARNING **: session inhibition not available, cinnamon-session is not available (cinnamon-settings-daemon:2002): power-plugin-WARNING **: session inhibition not available, cinnamon-session is not available