Replacing ntpd with systemd-timesyncd (Mint 18.1)
Introduction
It all began when I noted that my media center Linux machine (Linux Mint 18.1, Serena) finished a TV recording a bit earlier than expected. Logging in and typing “date” I was quite surprised to find out that the time was off by half a minute.
The first question that comes to mind is why the time synchronization didn’t work. The second is, if it didn’t work, how come I hadn’t noted this issue earlier? The computer is in use as a media center for little less than two years.
What happened
It turns out (and it wasn’t easy to tell) that the relevant daemon was ntpd.
So what’s up, ntp?
$ systemctl status ntp
● ntp.service - LSB: Start NTP daemon
Loaded: loaded (/etc/init.d/ntp; enabled; vendor preset: enabled)
Active: active (exited) since Wed 2018-12-19 12:38:06 IST; 1 months 7 days ag
Docs: man:systemd-sysv-generator(8)
Process: 1257 ExecStop=/etc/init.d/ntp stop (code=exited, status=0/SUCCESS)
Process: 1385 ExecStart=/etc/init.d/ntp start (code=exited, status=0/SUCCESS)
Dec 19 12:38:06 tv systemd[1]: Starting LSB: Start NTP daemon...
Dec 19 12:38:06 tv ntp[1385]: * Starting NTP server ntpd
Dec 19 12:38:06 tv ntp[1385]: ...done.
Dec 19 12:38:06 tv systemd[1]: Started LSB: Start NTP daemon.
Dec 19 12:38:06 tv ntpd[1398]: proto: precision = 0.187 usec (-22)
Dec 19 12:38:08 tv systemd[1]: Started LSB: Start NTP daemon.
Looks fairly OK. Maybe the logs can tell something?
$ journalctl -u ntp Dec 19 12:38:02 tv systemd[1]: Stopped LSB: Start NTP daemon. Dec 19 12:38:02 tv systemd[1]: Starting LSB: Start NTP daemon... Dec 19 12:38:02 tv ntp[1055]: * Starting NTP server ntpd Dec 19 12:38:02 tv ntpd[1074]: ntpd 4.2.8p4@1.3265-o Wed Oct 5 12:34:45 UTC 2016 (1): Starting Dec 19 12:38:02 tv ntpd[1076]: proto: precision = 0.175 usec (-22) Dec 19 12:38:02 tv ntp[1055]: ...done. Dec 19 12:38:02 tv systemd[1]: Started LSB: Start NTP daemon. Dec 19 12:38:02 tv ntpd[1076]: Listen and drop on 0 v6wildcard [::]:123 Dec 19 12:38:02 tv ntpd[1076]: Listen and drop on 1 v4wildcard 0.0.0.0:123 Dec 19 12:38:02 tv ntpd[1076]: Listen normally on 2 lo 127.0.0.1:123 Dec 19 12:38:02 tv ntpd[1076]: Listen normally on 3 lo [::1]:123 Dec 19 12:38:02 tv ntpd[1076]: Listening on routing socket on fd #20 for interface updates Dec 19 12:38:03 tv ntpd[1076]: error resolving pool 0.ubuntu.pool.ntp.org: Temporary failure in name resolution (-3) Dec 19 12:38:04 tv ntpd[1076]: error resolving pool 1.ubuntu.pool.ntp.org: Temporary failure in name resolution (-3) Dec 19 12:38:05 tv ntpd[1076]: error resolving pool 2.ubuntu.pool.ntp.org: Temporary failure in name resolution (-3) Dec 19 12:38:06 tv systemd[1]: Stopping LSB: Start NTP daemon... Dec 19 12:38:06 tv ntp[1257]: * Stopping NTP server ntpd Dec 19 12:38:06 tv ntp[1257]: ...done. Dec 19 12:38:06 tv systemd[1]: Stopped LSB: Start NTP daemon. Dec 19 12:38:06 tv systemd[1]: Stopped LSB: Start NTP daemon. Dec 19 12:38:06 tv systemd[1]: Starting LSB: Start NTP daemon... Dec 19 12:38:06 tv ntp[1385]: * Starting NTP server ntpd Dec 19 12:38:06 tv ntp[1385]: ...done. Dec 19 12:38:06 tv systemd[1]: Started LSB: Start NTP daemon. Dec 19 12:38:06 tv ntpd[1398]: proto: precision = 0.187 usec (-22) Dec 19 12:38:08 tv systemd[1]: Started LSB: Start NTP daemon.
Hmmm… There is some kind of trouble there, but it was surely resolved. Or? In fact, there was no ntpd process running, so maybe it just died?
Let’s try to restart the daemon, and see what happens. As root,
# systemctl restart ntp
after which the log went
Jan 26 20:36:46 tv systemd[1]: Stopping LSB: Start NTP daemon... Jan 26 20:36:46 tv ntp[32297]: * Stopping NTP server ntpd Jan 26 20:36:46 tv ntp[32297]: start-stop-daemon: warning: failed to kill 1398: No such process Jan 26 20:36:46 tv ntp[32297]: ...done. Jan 26 20:36:46 tv systemd[1]: Stopped LSB: Start NTP daemon. Jan 26 20:36:46 tv systemd[1]: Starting LSB: Start NTP daemon... Jan 26 20:36:46 tv ntp[32309]: * Starting NTP server ntpd Jan 26 20:36:46 tv ntp[32309]: ...done. Jan 26 20:36:46 tv systemd[1]: Started LSB: Start NTP daemon. Jan 26 20:36:46 tv ntpd[32324]: proto: precision = 0.187 usec (-22) Jan 26 20:36:46 tv ntpd[32324]: Listen and drop on 0 v6wildcard [::]:123 Jan 26 20:36:46 tv ntpd[32324]: Listen and drop on 1 v4wildcard 0.0.0.0:123 Jan 26 20:36:46 tv ntpd[32324]: Listen normally on 2 lo 127.0.0.1:123 Jan 26 20:36:46 tv ntpd[32324]: Listen normally on 3 enp3s0 10.1.1.22:123 Jan 26 20:36:46 tv ntpd[32324]: Listen normally on 4 lo [::1]:123 Jan 26 20:36:46 tv ntpd[32324]: Listen normally on 5 enp3s0 [fe80::f757:9ceb:2243:3e16%2]:123 Jan 26 20:36:46 tv ntpd[32324]: Listening on routing socket on fd #22 for interface updates Jan 26 20:36:47 tv ntpd[32324]: Soliciting pool server 118.67.200.10 Jan 26 20:36:48 tv ntpd[32324]: Soliciting pool server 210.23.25.77 Jan 26 20:36:49 tv ntpd[32324]: Soliciting pool server 211.233.40.78 Jan 26 20:36:50 tv ntpd[32324]: Soliciting pool server 43.245.49.242 Jan 26 20:36:30 tv ntpd[32324]: Soliciting pool server 45.76.187.173 Jan 26 20:36:30 tv ntpd[32324]: Soliciting pool server 46.19.96.19 Jan 26 20:36:31 tv ntpd[32324]: Soliciting pool server 210.173.160.87 Jan 26 20:36:31 tv ntpd[32324]: Soliciting pool server 119.28.206.193 Jan 26 20:36:49 tv ntpd[32324]: Soliciting pool server 133.243.238.244 Jan 26 20:36:49 tv ntpd[32324]: Soliciting pool server 91.189.89.199
Aha! So this is what a kickoff of ntpd should look like! Clearly ntpd didn’t recover all that well from the lack of internet connection (I suppose) during the media center’s bootup. Maybe it died, and was never restarted. The irony is that systemd has a wonderful mechanism for restarting failing daemons, but ntpd is still under the backward-compatible LSB interface. So the system silently remained with no time synchronization.
Go the systemd way
systemd supplies its own lightweight time synchronization mechanism, systemd-timesyncd. It makes much more sense, as it doesn’t open NTP ports as a server (like ntpd does, one may wonder what for), but just synchronizes the computer it runs on to the remote NTP server. And judging from my previous experience with systemd, in the event of multiple solutions, go for the one systemd offers. In fact, it’s sort-of enabled by default:
$ systemctl status systemd-timesyncd
● systemd-timesyncd.service - Network Time Synchronization
Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
Drop-In: /lib/systemd/system/systemd-timesyncd.service.d
└─disable-with-time-daemon.conf
Active: inactive (dead)
Condition: start condition failed at Wed 2018-12-19 12:38:01 IST; 1 months 7 days ago
ConditionFileIsExecutable=!/usr/sbin/VBoxService was not met
Docs: man:systemd-timesyncd.service(8)
Start condition failed? What’s this? Let’s look at the drop-in file:
$ cat /lib/systemd/system/systemd-timesyncd.service.d/disable-with-time-daemon.conf [Unit] # don't run timesyncd if we have another NTP daemon installed ConditionFileIsExecutable=!/usr/sbin/ntpd ConditionFileIsExecutable=!/usr/sbin/openntpd ConditionFileIsExecutable=!/usr/sbin/chronyd ConditionFileIsExecutable=!/usr/sbin/VBoxService
Oh please, you can’t be serious. Disabling the execution because of the existence of a file? If another NTP daemon is installed, does it mean it’s being enabled? In particular, if VBoxService is installed, does it mean we’re running as guests on a virtual machine? Like, seriously, someone might just install the Virtual Box client tools for no reason at all, and poof, there goes the time synchronization without any warning (note that this wasn’t the problem I had).
Moving to systemd-timesyncd
As mentioned earlier, systemd-timesyncd is enabled by default, but one may insist:
# systemctl enable systemd-timesyncd.service
(Nothing response, because it’s enabled anyhow)
However in order to make it work, remove the condition that prevents it from running:
# rm /lib/systemd/system/systemd-timesyncd.service.d/disable-with-time-daemon.conf
and then disable and stop ntpd:
# systemctl disable ntp # systemctl stop ntp
On my computer, the other two time synchronizing tools (openntpd and chrony) aren’t installed, so they are not to worry about.
And then we have timedatectl
Note directly related, and still worth mentioning
$ timedatectl
Local time: Sat 2019-01-26 21:22:57 IST
Universal time: Sat 2019-01-26 19:22:57 UTC
RTC time: Sat 2019-01-26 19:22:57
Time zone: Asia/Jerusalem (IST, +0200)
Network time on: yes
NTP synchronized: yes
RTC in local TZ: no
Systemd is here to take control of everything, obviously.