The nuisance
After trying to fix another problem with Pulseaudio, and messing up a bit with its configuration files, the option to control the volume by virtue of keyboard buttons suddenly stopped to work. The cursor would blink briefly when pressing the button, but nothing else happened.
I tried all the fairly normal things, including to restart the Pulseaudio daemon, to restart Cinnamon (with Alt-F2 + “r”), restart the session by changing the user (to the same user), disconnect and reconnect the (USB) keyboard, and a whole lot of other things.
It took me quite some time to fix this, so here’s what worked for me this time, along with some debug info.
The machine was a Linux Mint 19 with Cinnamon.
At an earlier stage, even the sound applet was gone from its desktop panel. To fix that, I right-clicked the panel, picked Troubleshoot > Looking Glass, and picked the Extension tab. There’s an applet listed called “Sound”. So I right-clicked it, and chose “Reload Code”. That got the applet back to the panel, and it worked flawlessly. But then I was stuck with the volume button problem.
The fix
What I actually did:
- Kill the process named csd-keyboard (probably irrelevant)
- Kill the process named csd-media-keys
- Restart pulseaudio with “pulseaudio -k”
This was the output of journalctl as a result of these:
May 31 12:17:03 cinnamon-session[2115]: WARNING: t+1132491.55749s: Application 'cinnamon-settings-daemon-keyboard.desktop' killed by signal 15
May 31 12:17:17 cinnamon-session[2115]: WARNING: t+1132505.78238s: Application 'cinnamon-settings-daemon-media-keys.desktop' killed by signal 15
May 31 12:17:17 rtkit-daemon[2284]: Successfully made thread 4304 of process 4304 (n/a) owned by '1010' high priority at nice level -11.
May 31 12:17:17 rtkit-daemon[2284]: Supervising 8 threads of 4 processes of 1 users.
May 31 12:17:17 pulseaudio[4304]: [pulseaudio] pid.c: Daemon already running.
May 31 12:18:04 bluetoothd[12009]: Endpoint unregistered: sender=:1.120210 path=/MediaEndpoint/A2DPSource
May 31 12:18:04 bluetoothd[12009]: Endpoint unregistered: sender=:1.120210 path=/MediaEndpoint/A2DPSink
May 31 12:18:04 rtkit-daemon[2284]: Successfully made thread 4345 of process 4345 (n/a) owned by '1010' high priority at nice level -11.
May 31 12:18:04 rtkit-daemon[2284]: Supervising 4 threads of 4 processes of 1 users.
May 31 12:18:04 pulseaudio[4345]: [pulseaudio] sink.c: Default and alternate sample rates are the same.
May 31 12:18:04 rtkit-daemon[2284]: Supervising 3 threads of 3 processes of 1 users.
May 31 12:18:04 rtkit-daemon[2284]: Successfully made thread 4347 of process 4345 (n/a) owned by '1010' RT at priority 5.
May 31 12:18:04 rtkit-daemon[2284]: Supervising 4 threads of 3 processes of 1 users.
May 31 12:18:04 rtkit-daemon[2284]: Supervising 4 threads of 3 processes of 1 users.
May 31 12:18:04 rtkit-daemon[2284]: Successfully made thread 4348 of process 4345 (n/a) owned by '1010' RT at priority 5.
May 31 12:18:04 rtkit-daemon[2284]: Supervising 5 threads of 3 processes of 1 users.
May 31 12:18:04 rtkit-daemon[2284]: Supervising 5 threads of 3 processes of 1 users.
May 31 12:18:04 rtkit-daemon[2284]: Successfully made thread 4349 of process 4345 (n/a) owned by '1010' RT at priority 5.
May 31 12:18:04 rtkit-daemon[2284]: Supervising 6 threads of 3 processes of 1 users.
May 31 12:18:05 rtkit-daemon[2284]: Supervising 6 threads of 3 processes of 1 users.
May 31 12:18:05 rtkit-daemon[2284]: Successfully made thread 4350 of process 4345 (n/a) owned by '1010' RT at priority 5.
May 31 12:18:05 rtkit-daemon[2284]: Supervising 7 threads of 3 processes of 1 users.
May 31 12:18:05 bluetoothd[12009]: Endpoint registered: sender=:1.120482 path=/MediaEndpoint/A2DPSource
May 31 12:18:05 bluetoothd[12009]: Endpoint registered: sender=:1.120482 path=/MediaEndpoint/A2DPSink
May 31 12:18:05 pulseaudio[4345]: [pulseaudio] backend-ofono.c: Failed to register as a handsfree audio agent with ofono: org.freedesktop.DBus.Error.ServiceUnknown: The name org.ofono was not provided by any .service files
May 31 12:18:05 rtkit-daemon[2284]: Successfully made thread 4352 of process 4352 (n/a) owned by '1010' high priority at nice level -11.
May 31 12:18:05 rtkit-daemon[2284]: Supervising 8 threads of 4 processes of 1 users.
May 31 12:18:05 pulseaudio[4352]: [pulseaudio] pid.c: Daemon already running.
May 31 12:18:58 gnome-keyring-daemon[2204]: asked to register item /org/freedesktop/secrets/collection/login/9, but it's already registered
I suppose that killing csd-media-keys was the part that really did the trick, because that caused a request to start Pulseaudio.
Why it worked (presumably)
The cinnamon-settings-daemon is the component responsible for passing the keyboard request to Pulseaudio. Apparently, messing up with Pulseaudio caused one of its components, csd-media-keys, to mess up as well. Killing it caused cinnamon-settings-daemon to restart it automatically. From fresh.
It’s fine to kill the csd-* processes. Nothing dramatic happens. The killed process is just restarted. But I haven’t tried to kill cinnamon-settings-daemon itself. I believe it will restart Cinnamon completely, something I wanted to avoid.
Dbus activity
I’ve already written a post mentioning how to monitor the Dbus. So here’s a different take on the same topic.
The difference is quite evident when running dbus-monitor (not as root) as follows in order to monitor the activity for the current session:
$ dbus-monitor
This is the output as a result of pressing a volume up button, when it works correctly (i.e. after fixing the problem):
method call time=1685525392.173838 sender=:1.6531 -> destination=:1.6616 serial=1431 path=/org/cinnamon/SettingsDaemon/KeybindingHandler; interface=org.cinnamon.SettingsDaemon.KeybindingHandler; member=HandleKeybinding
uint32 2
method call time=1685525392.174967 sender=:1.6616 -> destination=org.freedesktop.DBus serial=566 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=AddMatch
string "type='signal',interface='ca.desrt.dconf.Writer',path='/ca/desrt/dconf/Writer/user',arg0path='/org/cinnamon/desktop/sound/'"
method return time=1685525392.175012 sender=org.freedesktop.DBus -> destination=:1.6616 serial=382 reply_serial=566
method call time=1685525392.175026 sender=:1.6616 -> destination=:1.6531 serial=567 path=/org/Cinnamon; interface=org.Cinnamon; member=ShowOSD
array [
dict entry(
string "icon"
variant string "audio-volume-medium-symbolic"
)
dict entry(
string "level"
variant int32 34
)
]
method call time=1685525392.175170 sender=:1.6616 -> destination=org.freedesktop.DBus serial=568 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=RemoveMatch
string "type='signal',interface='ca.desrt.dconf.Writer',path='/ca/desrt/dconf/Writer/user',arg0path='/org/cinnamon/desktop/sound/'"
method return time=1685525392.175189 sender=org.freedesktop.DBus -> destination=:1.6616 serial=383 reply_serial=568
method call time=1685525392.175198 sender=:1.6616 -> destination=org.freedesktop.DBus serial=569 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=AddMatch
string "type='signal',interface='ca.desrt.dconf.Writer',path='/ca/desrt/dconf/Writer/user',arg0path='/org/cinnamon/desktop/sound/'"
method return time=1685525392.175216 sender=org.freedesktop.DBus -> destination=:1.6616 serial=384 reply_serial=569
method call time=1685525392.176330 sender=:1.6618 -> destination=org.freedesktop.DBus serial=26 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=RequestName
string "org.freedesktop.ReserveDevice1.Audio2"
uint32 5
signal time=1685525392.176383 sender=org.freedesktop.DBus -> destination=(null destination) serial=50 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=NameOwnerChanged
string "org.freedesktop.ReserveDevice1.Audio2"
string ""
string ":1.6618"
signal time=1685525392.176411 sender=org.freedesktop.DBus -> destination=:1.6618 serial=51 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=NameAcquired
string "org.freedesktop.ReserveDevice1.Audio2"
method return time=1685525392.176427 sender=org.freedesktop.DBus -> destination=:1.6618 serial=52 reply_serial=26
uint32 1
method return time=1685525392.184853 sender=:1.6616 -> destination=:1.6531 serial=570 reply_serial=1431
method call time=1685525392.184915 sender=:1.6616 -> destination=org.freedesktop.DBus serial=571 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=RemoveMatch
string "type='signal',interface='ca.desrt.dconf.Writer',path='/ca/desrt/dconf/Writer/user',arg0path='/org/cinnamon/desktop/sound/'"
method return time=1685525392.184937 sender=org.freedesktop.DBus -> destination=:1.6616 serial=385 reply_serial=571
method return time=1685525392.190757 sender=:1.6531 -> destination=:1.6616 serial=1432 reply_serial=567
Unfortunately, I didn’t save the output before the fix. But it was exactly two rows of output. As far as I recall, the first and last row from this dump were present when it didn’t work (i.e. the row involving KeybindingHandler’s request, and the return method that closed it).
The bus addresses can be listed with the following command:
$ busctl --user
In this specific session the output was:
NAME PID PROCESS USER CONNECTION UNIT SESSION DESCRIPTION
:1.0 2103 systemd eli :1.0 user@1010.service - -
:1.1014 45251 pxgsettings eli :1.1014 session-c1.scope c1 -
:1.1015 45277 scp-dbus-servic eli :1.1015 user@1010.service - -
:1.1016 45277 scp-dbus-servic eli :1.1016 user@1010.service - -
:1.12 2213 csd-a11y-keyboa eli :1.12 session-c1.scope c1 -
:1.1234 53331 nemo eli :1.1234 session-c1.scope c1 -
:1.13 2221 csd-a11y-settin eli :1.13 session-c1.scope c1 -
:1.131 16564 klauncher eli :1.131 session-c1.scope c1 -
:1.16 2232 csd-cursor eli :1.16 session-c1.scope c1 -
:1.17 2225 csd-color eli :1.17 session-c1.scope c1 -
:1.18 2238 csd-clipboard eli :1.18 session-c1.scope c1 -
:1.19 2242 csd-orientation eli :1.19 session-c1.scope c1 -
:1.20 2233 csd-wacom eli :1.20 session-c1.scope c1 -
:1.21 2245 csd-xsettings eli :1.21 session-c1.scope c1 -
:1.22 2248 csd-mouse eli :1.22 session-c1.scope c1 -
:1.24 2256 csd-print-notif eli :1.24 session-c1.scope c1 -
:1.26 2258 csd-background eli :1.26 session-c1.scope c1 -
:1.27 2260 gvfsd eli :1.27 user@1010.service - -
:1.28 2267 csd-housekeepin eli :1.28 session-c1.scope c1 -
:1.29 2274 csd-screensaver eli :1.29 session-c1.scope c1 -
:1.30 2270 csd-power eli :1.30 session-c1.scope c1 -
:1.31 2299 dconf-service eli :1.31 user@1010.service - -
:1.32 2285 csd-automount eli :1.32 session-c1.scope c1 -
:1.329 22590 xreader eli :1.329 session-c1.scope c1 -
:1.33 2306 gvfsd-fuse eli :1.33 user@1010.service - -
:1.330 22596 xreaderd eli :1.330 user@1010.service - -
:1.332 22606 WebKitNetworkPr eli :1.332 session-c1.scope c1 -
:1.35 2313 csd-printer eli :1.35 session-c1.scope c1 -
:1.37 2349 gvfs-udisks2-vo eli :1.37 user@1010.service - -
:1.38 2384 gvfs-gphoto2-vo eli :1.38 user@1010.service - -
:1.39 2277 csd-xrandr eli :1.39 session-c1.scope c1 -
:1.40 2410 gvfs-goa-volume eli :1.40 user@1010.service - -
:1.41 2414 goa-daemon eli :1.41 user@1010.service - -
:1.42 2427 goa-identity-se eli :1.42 user@1010.service - -
:1.43 2432 gvfs-mtp-volume eli :1.43 user@1010.service - -
:1.44 2436 gvfs-afc-volume eli :1.44 user@1010.service - -
:1.448 31116 gvfsd-http eli :1.448 user@1010.service - -
:1.457 31367 gvfsd-network eli :1.457 user@1010.service - -
:1.46 2472 polkit-gnome-au eli :1.46 session-c1.scope c1 -
:1.463 31401 gvfsd-dnssd eli :1.463 user@1010.service - -
:1.47 2481 cinnamon-killer eli :1.47 session-c1.scope c1 -
:1.48 2478 nemo-desktop eli :1.48 session-c1.scope c1 -
:1.49 2480 nm-applet eli :1.49 session-c1.scope c1 -
:1.5 2115 cinnamon-sessio eli :1.5 session-c1.scope c1 -
:1.50 2474 blueberry-obex- eli :1.50 session-c1.scope c1 -
:1.51 2474 blueberry-obex- eli :1.51 session-c1.scope c1 -
:1.52 2499 obexd eli :1.52 user@1010.service - -
:1.54 2506 gvfsd-trash eli :1.54 user@1010.service - -
:1.55 2512 gvfsd-metadata eli :1.55 user@1010.service - -
:1.56 2535 python2 eli :1.56 session-c1.scope c1 -
:1.5653 62813 chrome eli :1.5653 session-c1.scope c1 -
:1.5654 62813 chrome eli :1.5654 session-c1.scope c1 -
:1.5655 62813 chrome eli :1.5655 session-c1.scope c1 -
:1.5656 62813 chrome eli :1.5656 session-c1.scope c1 -
:1.5661 62813 chrome eli :1.5661 session-c1.scope c1 -
:1.57 2534 cinnamon-screen eli :1.57 session-c1.scope c1 -
:1.58 2574 gnome-terminal eli :1.58 session-c1.scope c1 -
:1.5834 52878 xreader eli :1.5834 session-c1.scope c1 -
:1.5836 52891 WebKitNetworkPr eli :1.5836 session-c1.scope c1 -
:1.5842 54509 xreader eli :1.5842 session-c1.scope c1 -
:1.5843 54522 WebKitNetworkPr eli :1.5843 session-c1.scope c1 -
:1.59 2574 gnome-terminal eli :1.59 session-c1.scope c1 -
:1.6 2115 cinnamon-sessio eli :1.6 session-c1.scope c1 -
:1.60 2579 gconfd-2 eli :1.60 user@1010.service - -
:1.61 2574 gnome-terminal eli :1.61 session-c1.scope c1 -
:1.6105 47745 xreader eli :1.6105 session-c1.scope c1 -
:1.6107 47757 WebKitNetworkPr eli :1.6107 session-c1.scope c1 -
:1.6477 31538 thunderbird-bin eli :1.6477 session-c1.scope c1 -
:1.6478 31538 thunderbird-bin eli :1.6478 session-c1.scope c1 -
:1.6531 2451 cinnamon eli :1.6531 session-c1.scope c1 -
:1.6562 69100 csd-sound eli :1.6562 session-c1.scope c1 -
:1.6592 1641 firefox-bin eli :1.6592 session-c1.scope c1 -
:1.6593 1641 firefox-bin eli :1.6593 session-c1.scope c1 -
:1.6594 1700 Web Content eli :1.6594 session-c1.scope c1 -
:1.6595 1760 WebExtensions eli :1.6595 session-c1.scope c1 -
:1.6596 1824 Web Content eli :1.6596 session-c1.scope c1 -
:1.6597 1876 Web Content eli :1.6597 session-c1.scope c1 -
:1.6598 1975 cinnamon-settin eli :1.6598 session-c1.scope c1 -
:1.6599 1975 cinnamon-settin eli :1.6599 session-c1.scope c1 -
:1.6608 2640 Web Content eli :1.6608 session-c1.scope c1 -
:1.6609 3399 Web Content eli :1.6609 session-c1.scope c1 -
:1.6615 4286 csd-keyboard eli :1.6615 session-c1.scope c1 -
:1.6616 4296 csd-media-keys eli :1.6616 session-c1.scope c1 -
:1.6617 4296 csd-media-keys eli :1.6617 session-c1.scope c1 -
:1.6618 4345 pulseaudio eli :1.6618 session-c1.scope c1 -
:1.6621 4692 xed eli :1.6621 session-c1.scope c1 -
:1.6625 5225 busctl eli :1.6625 session-c1.scope c1 -
:1.67 2860 applet.py eli :1.67 session-c1.scope c1 -
:1.7 2189 at-spi-bus-laun eli :1.7 user@1010.service - -
:1.73 3537 gnome-system-mo eli :1.73 session-c1.scope c1 -
:1.8 2196 at-spi2-registr eli :1.8 user@1010.service - -
:1.9 2204 gnome-keyring-d eli :1.9 session-c1.scope c1 -
:1.92 8884 xdg-desktop-por eli :1.92 user@1010.service - -
:1.93 8888 xdg-document-po eli :1.93 user@1010.service - -
:1.94 8891 xdg-permission- eli :1.94 user@1010.service - -
:1.95 8902 pxgsettings eli :1.95 user@1010.service - -
:1.96 8906 xdg-desktop-por eli :1.96 user@1010.service - -
ca.desrt.dconf 2299 dconf-service eli :1.31 user@1010.service - -
ca.desrt.dconf-editor - - - (activatable) - -
org.Cinnamon 2451 cinnamon eli :1.6531 session-c1.scope c1 -
org.Cinnamon.HotplugSniffer - - - (activatable) - -
org.Cinnamon.LookingGlass 2451 cinnamon eli :1.6531 session-c1.scope c1 -
org.Cinnamon.Melange - - - (activatable) - -
org.Cinnamon.Slideshow - - - (activatable) - -
org.Nemo 53331 nemo eli :1.1234 session-c1.scope c1 -
org.NemoDesktop 2478 nemo-desktop eli :1.48 session-c1.scope c1 -
org.PulseAudio1 4345 pulseaudio eli :1.6618 session-c1.scope c1 -
org.a11y.Bus 2189 at-spi-bus-laun eli :1.7 user@1010.service - -
org.bluez.obex 2499 obexd eli :1.52 user@1010.service - -
org.cinnamon.ScreenSaver 2534 cinnamon-screen eli :1.57 session-c1.scope c1 -
org.cinnamon.SettingsDaemon.KeybindingHandler 4296 csd-media-keys eli :1.6616 session-c1.scope c1 -
So it’s quite clear that the players here were cinnamon’s main socket, csd-media-keys and pulseaudio.
Overview
This is a collection of insights regarding lead-acid batteries, in particular those sealed batteries that are used in UPSes. Had it not been for my failed attempts to get a decent battery for my UPS, none of this would have been written. It was only when I asked myself why the UPS doesn’t maintain power nearly as long as I expected that I started to learn more about these batteries.
Batteries is far away from anything I do for a living. This is definitely not my expertise. Please keep that in mind. But as it turns out, there’s a lot to know about them.
Well, I think I’ll take that back: After a lot of playing with these batteries, my conclusion is that knowledge doesn’t help much. Run a test, see how long the battery holds up your computer, and that’s it. There doesn’t seem to be anything smarter to do.
There’s probably only one important thing to know about them: They should be fresh. That’s hardly new, and yet, that’s probably the bottom line. Lead-acid batteries discharge by themselves over time, and when they reach a certain level of discharge, they get damaged. Also, the discharge rate depends a lot on temperature. So a battery that has been stored in a hot place somewhere in a distant country for over a year is probably no good candidate for your UPS. The manufacturer’s timestamp is everything: It starts to seem to me that most batteries were more or less the same on the day that they left the factory. The difference is how much damage they caught during storage.
For a concise technical background, I recommend reading Power Sonic’s Technical Manual.
So without further ado, let’s go through my findings, in more or less random order.
A 7AH battery doesn’t really give 7AH
With lead-acid batteries, 7AH doesn’t meas 7A for 1 hour. Or 3.5A for 2 hours. Or any other combination, except for 0.35A for 20 hours. More about that below.
The amount of charge (and energy) that a lead-acid battery supplies until it’s discharged depends dramatically on the discharging current. The capacity printed on the battery is given for a 20-hours discharge, or using the jargon, 0.05C. That “C” is 7 taken from the 7AH figure, so 7AH are obtained if the discharge current is 0.35A. For larger currents, expect much less energy out of the battery.
This phenomenon is approximated by Peukert’s Law.
For example, my specific case: The load is 70W (at 142VA) according to the UPS itself. I’ll assume that the low power factor thing can be ignored, i.e. that the fact that the VA figure is twice the consumed power makes no difference. This low power factor is natural to switching power supplies, as they draw more current when the voltage is low, so their behavior is far from a plain resistor (unless specifically compensated to mitigate this effect). I’ll also assume that the UPS is 100% efficient on its voltage conversion, which is complete rubbish, but for the heck of it.
So for two 12V batteries in series it goes 70W/24V =~ 2.9A, which is about 0.4C (2.9 / 7 =~ 0.4). A ballpark figure can be taken from Figure 4 in Power Sonic’s Technical Manual, showing that the voltage starts to drop after about an hour, and reaches the critical value somewhere after an hour and a half. Note that I have different batteries.
Also from Table 2 of the same Manual, we have that the actual capacity of a 7AH battery, when drained with a 4.34A current, is 4.34AH (one hour). The current is higher than 2.9A, but given that the UPS isn’t really 100% efficient, it’s likely that the real discharge current is closer to 4A than to 2.9A. So that explains why the UPS said 1:12 hours when I updated the battery replacement date.
Many battery manufacturers supply datasheets with Discharge Characteristics Curves. Even better, if there are tables with Constant Current Discharge Characteristics (also referred to as F.V / Time specification). See Ritar’s datasheet for RT1270, for example. The numbers inside this table are the constant currents for each discharging scenario. Each row is the final voltage (F.V) per cell that is reached in this scenario, and each column represents the time it takes to reach this voltage.
It’s not clear at which voltage the UPS decides to stop discharging. Even though 1.60V per cell seems to be a recurring number in datasheets for the game over. And on the other hand, I’ve repeatedly seen 0.35A (i.e. 0.05C) paired with 1.75V and 20 hours discharge in those F.V / Time tables. So it’s not really clear. Anyhow, finding a number that is similar to the intended current in the table can give an idea on how long the battery is supposed to last.
Obviously, different batteries behave differently on higher currents. I couldn’t find data on my “Bulls Power” batteries. So maybe they could meet the 7AH specification for a 20-hours discharge, and then perform really poorly with higher, real-life currents. No way to know for sure.
So what is the state of the battery after it has been discharged with less than 7AH? It’s still charged, as a matter of fact. A rapid discharge, which yields less than the full capacity is healthy for the battery. It’s when the full capacity is utilized with a slow discharge pattern that the battery can be damaged if it stays in that state for a long while.
Charging a lead-acid battery
There are in fact several ways to charge a lead acid battery (for example, this), but the common way to charge a battery is to feed it with a current of 0.3C (2.1A for a 7AH battery) until the voltage rises to above a threshold. Or to set a constant voltage (with a current limit) and charge until the current goes below a certain level. Then comes the second phase, which is the slow charging to 100%.
When the battery is full, a floating voltage should be applied to the battery. Each battery specifies its own preferred floating voltage, but they all land at around 13.5-13.8V. This is the correct way to maintain a fully charged battery: A small charging current compensates for the battery’s inherent discharge (typically around 0.001C, i.e. 7 mA for a 7AH battery), and the battery remains fully charged in a way that is best for its health.
Battery manufacturers usually state something like 10% discharge for 90 days or so. This corresponds to about 0.000046C, or ~0.3 mA for a 7AH battery. But in this scenario, the battery discharges against its own voltage, and not the elevated floating voltage (does it matter?).
Knowing the battery’s charge level
How does one estimate how much energy a lead-acid battery has? The answer is unpleasant, yet simple: There is really no way to measure it from the battery electronically. After reading quite some material on the subject, that became evident to me: There are plenty of papers describing exotic algorithms for estimating a battery’s health and charge level, and their abundance and variety proves that there’s really no way to tell, except for draining it.
Maybe the first question should have been: What is consumed when the battery gets discharged? For one, sulfuric acid reacts with the lead plates and become lead sulfate. When there’s no acid left, the battery is done. On the other hand, the lead plates themselves might deteriorate in the process. This can happen relatively early if the lead plates are partly damaged because of previous use (or abuse).
There is one way to measure the charge level that is considered reliable, which is measuring the open circuit voltage (OCV) after the battery has been disconnected for a while (some say a few hours, battery manufacturers typically require 24 hours). Letting the battery rest allows it to reach a chemical equilibrium, at which point the voltage reflects its charge level. This is surely true for a fresh battery, because the OCV reflects the specific gravity of the electrolyte, that is the concentration of the sulfuric acid.
As for batteries with some history, the picture is less clear, and I haven’t managed to figure out if the OCV voltages remain the same, and if the voltage vs. charge percentage relate to the original charge capacity, or the one that is available after the battery is worn out. The OCV measurementt covers the charge level in the sense of how much acid there’s left to consume, but will the lead plates hold that long?
For example, Power Sonic claims that the OVC goes from 1.94V/cell to 2.16V/cell for 0% to 100% charge respectively. As a 12-volt battery has 6 cells, this corresponds to 11.64V to 12.96V. These figures are quite similar to those presented by another manufacturer.
But what does 100% charge mean? 7AH or as much as is left when the battery has worked for some time? My anecdotal measurement of the batteries I took out from the UPS was 12.99V after letting them rest. In other words, they presented a OCV voltage corresponding to 100% charge, even though they had much less than 7AH.
So how does a UPS estimate the remaining runtime? Well, the simple way is to let the battery run out once, and there you have a number. Clearly, Smart UPS uses this method.
Are there any alternatives? In theory, the UPS could let the battery rest for 24 hours, and measure its OCV. This is possible, because most of the time the UPS doesn’t need the battery. But even my anecdotal measurement shows that a 100% charge-like reading doesn’t mean much.
For other types of batteries (Li-ion in particular), measuring the current on the battery, in and out (Coulomb Counting), gives an idea on how much charge it contains. This doesn’t work with lead acid batteries, because the recommended way to maintain a standby battery, is to continuously float charge it. That means holding a constant voltage (say, 2.25V per cell, that is 13.5V for a 12V battery, or 27V on a battery pair, as in SmartUPS 750).
As this voltage is higher than the OCV at rest, this causes a small trickle current (said to be about 0.001C), which compensates for the battery’s self discharge. Even if it overcharges the battery slightly, the gases that are released are recycled internally in a sealed battery, so there’s no damage.
Hence the recommended strategy for charging a lead-acid battery is to charge it quickly as long as its voltage / current pair indicates that it’s far from being fully charged, and then apply a constant, known and safe voltage. This allows it to charge completely slowly, and then maintain the charge without any risk for overcharging. Odds are that this is what the UPS does.
But that makes Coulomb Counting impossible: During the float charge phase (that is, virtually all the time) the current may and may not actually charge the battery.
Measuring the discharging curve
Being quite frustrated by bad batteries, I decided to go for a more direct approach: Instead of taking my computer down and pushing the battery into the UPS each time, I thought it would be easier and more informative to run a discharging test: I used my old Mustek 600 UPS to charge the battery, and then discharged it through a (cheap) 55W light bulb for a car’s headlight (which is intended for 12V of course). With this, I ran a classic lab experiment, recording voltage and current as a function of time. Like a school lab.
The first step is to charge the battery of course. I’ve discussed this topic briefly above, but what about my Mustek’s UPS? It takes a very easy and slow approach: With a partly discharged battery, it starts with an initial charging current of 0.4–0.35 A. This current goes slowly down as the battery’s voltage rises. The voltage goes up very slowly from 12.8V and eventually stabilizes at a voltage around 13.5-13.6V (depends on the battery). Charging a battery to a decent level takes 7 hours, but I went for a 24 hours charge before running a discharging test. Just to make sure the battery was really fully charged.
By the way, the Mustek UPS charges even when not powered on (but connected to power). When disconnected from power, no charging occurs.
Now to the interesting part: The discharging of the batteries. I made several tests, and they all yielded the same results, more or less. In all of these, I made accurate voltage measurements (with a Fluke) as a function of time. This is the plot of the last experiment I did, with a logarithmic x-axis, each ‘x’ mark on the plot is a measurement:
This experiment was done on one of the Bulls Power batteries.
I measured the current throughout the session: The light bulb drew 4.00 A in the beginning (corresponding to ~48W), and the currents remained above 3.8A during the first 20 minutes. So this can be considered a 0.55C discharge.
The first thing to note is that the graph fits the discharge curve for Ritar’s RT1270 for 0.55C during the first 18-19 minutes: The voltage goes down gradually, and drops about 0.04V/cell during the first 10 minutes. On the next 8-9 minutes, the voltage drop is about 0.05V/cell, also following Ritar’s curve.
But then it’s quickly downhill, strongly diverting from the expected graph. The battery collapses quickly at this point. I haven’t found a single reference to this behavior, but I suppose that it’s a result of aging. And/or deep discharge during storage.
Is this consistent with the fact that this battery held for 9 minutes in the UPS, along with one that behaves roughly the same? Maybe. The UPS reported a power consumption of 105W in that situation, so if I’ll assume 90% power efficiency, we have a current of 105 * (1 / 0.9) / 24 = 4.86A ≈ 0.7C. That’s somewhere between the graph for 0.55C and 1C. So with the steeper discharging curve of this case, it’s actually possible that the collapse began much earlier.
Other random takeaways:
- With this battery, it doesn’t matter so much at which voltage the UPS powers itself off. The last part’s slope is so steep, so it’s a matter of 30 seconds this way or another.
- A 55W light bulb is blinding and heating. Be sure to have it fixed in a way that prevents it from heating the battery and also make sure that it’s literally out of sight, and that no visual contact will be needed with it as long as the experiment runs.
- Check the crocodiles’ or easy-hook’s resistance. In particular crocodiles can have a high an unstable resistance (0.1Ω is high in this context, because it causes a voltage drop of 0.4V).
- As it says in many information sources, lead batteries recover after a discharging session after some time of rest: After a day, I connected the battery to the bulb, and it began discharging at 11.8V, and falling rapidly.
Discharging curves, round #2
The saga went on, and I tell it in detail in the other post. The quick summary is: Being unhappy with the Bull batteries, I bought a pair of batteries from a company named Afik, and put them in the UPS. The results weren’t really impressing with these either, so I decided to replace the Afik batteries with something else. Eventually, I ended up with a pair of Vega Power batteries.
Unfortunately, I didn’t make any measurements on the Afik batteries before putting them in the UPS, mainly because I thought they would work, and end of story. So as things turned out, I obtained the discharging curve from the new Vega Power batteries before putting them in the UPS, and then I did the same thing with the Afik batteries only after they had been in the UPS for a few months.
This time I created an automatic test setting that allowed two readings per second, so the curves are more accurate and detailed.
Note to self: See new-computer/ups/lead acid batteries/discharge-tests-2/.
I used the same light 55W light bulb, which drew 4.0A at the beginning of the discharging test, and then went down slowly to about 3.5A towards the end of the test. More or less constant current, that is.
The colors of the curves in the graphs below represent which test they belong to:
- Red: Vega Power battery #1.
- Blue: Vega Power battery #2, first test.
- Magenta: Vega Power battery #2, second test.
- Green: Afik battery #1.
- Cyan: Afik battery #2.
Evidently, I obtained a curve from Vega Power battery #2 twice: Once after charging for ~24 hours (from as it came from the shop) with the Mustek UPS, and the second time after a total of ~48 hours from the state of the previous discharging session. The graphs below might imply that I got these two measurements swapped, but no, there’s no mistake about this.
Vega Power battery #1 was charged during 26 hours (from its initial state from the shop), and then 18 hours with a brief pause between the two charging sessions.
So to the first graph. Time in minutes on the X axis, and voltage per cell on the Y axis.
The curves begin one minute after discharging starts, and stop when the battery reached 10.0V (with 6 cells in the battery, that’s 1.67V / cell). I’ll show more details below about the first minute. As for choosing 1.67V as the end for discharging: The graphs for charging current around 0.55C often go this low in datasheets, so I suppose this shouldn’t damage the battery.
The first and obvious thing one can see is that the Afik batteries held shorter than the Vega Power batteries: 55 minutes vs. 80-87 minutes. That’s consistent with the fact that the Vega Power lasted longer in the UPS.
But as I’ve written in that other post, the UPS started to panic-beeping after 7:30 minutes with the Afik batteries, and then kept power up for another 15 minutes. Why did that happen? There’s nothing in the green nor cyan curves that offers a clue for the reason to the early panic. On the other hand, it’s evident that these batteries lost voltage quite rapidly after about 40 minutes, possibly going into “deep discharge”. That said, they didn’t collapse nearly as quickly as the Bull Power batteries. So there are definitely different levels of junk.
As for the Vega Power batteries, they performed surprisingly well on these tests. The shortest run was 80 minutes. Taking a gross average of 3.75A as the current, that means that the battery gave 5AH. Quite impressive. Also, according to Ritar’s datasheet for RT1270, that battery will reach 1.85V/cell after one hour if a current of 3.321A is drawn from it. Vega Power got to that region after 70 minutes with a higher current.
So the question is why the UPS started panicking after 28 minutes and cut the power very soon after that. Indeed, the current that is drawn from the batteries was higher inside the UPS. But why that sudden cut? Does it have to do with the curves’ bending at around 1.96V?
In fact, the reason that I repeated the first discharging test on Vega Power #2 (in blue) was that sharp bend. To my surprise, the bend went away on the second test (in magenta), but the battery reached the 1.67V point 7 minutes earlier (after a 48 hours charging). Why? Don’t know. Lead acid batteries are not supposed to have any memory.
In the next graph, I’ve removed the curves for the Afik batteries, and left only Vega Power’s curves. In addition, I’ve shifted the positions of these curves in the X axis. Namely, I delayed Vega #1′s curve by 9 minutes, and Vega #2′s first test’s curve 2:15 minutes. And we get this:
So it’s evident that both batteries ran according to an accurate discharging pattern for at least 45 minutes, and then things happened. Is this the start of “deep discharge”? Does this mean that discharging the batteries below 1.95V is a bad idea? Maybe this is what my UPS thought, and accordingly stopped after 28 minutes (which could be the same point, as it probably drew more current than this test).
Doesn’t the part where all three curves overlap look a little linear? I’ll come back to that below.
It also looks like the shifting I made in the X axis compensates for differences in the initial charging level. If that’s true, the charging levels of Vega Power #2 were nearly the same in both tests (slightly more charge after the second, 48-hours, charging session). But if so, why did this battery reach 1.67V faster on the second round?
And why was Vega Power #1 significantly less charged after a super-long charging session? Maybe an issue with the Mustek UPS?
The third graph is the same as the first, but with logarithmic X-axis.
The interesting thing about this representation is that it’s comparable with the curves that appear in some batteries’ datasheets. It’s quite evident that these curves are similar to those that are usually given for 0.55C.
The bends of the blue and red curves are more evident on this graph. Recall that they were obtained during the first tests of each of the Vega Power batteries. The pretty curve in magenta is the second run of Vega Power #2.
Finally, I discharged Afik battery #2 down to 1V, which is considered a damaging thing to do. But who cares, I have no plans for this battery anyhow. So here’s the curve:
Note that this is the same test and same curve as shown above, only that I display it in full here.
Transient responses
So far I’ve been looking at the curves during the actual discharging phase. But some possibly interesting things happen before and after that.
So first, what happens when the discharging begins:
Surprise, surprise, there’s an undershoot. Why? I’m tired of trying to even speculate. But it’s definitely a thing. And by the way, the initial voltages (i.e. with zero current) have no meaning in particular (?), because the batteries didn’t have enough time to stabilize after charging.
Note that the X-axis is given in seconds here. Not minutes.
And here’s a close-up on the same graph, just to make the point that the undershoot occurred on all curves:
The next thing to look at is what happens when the discharging process is stopped. I did this by disconnecting the light bulb, so the current went down abruptly from 3.5A to zero. This occurs at time zero of the following graph.
Recall that Afik #2 (cyan) went down as low as 1V/cell, and yet it recovered quite similarly Afik #1, which didn’t go through this minor trauma.
It’s a well-known fact that lead-acid batteries recover after a high-current discharge and resume their capability to provide energy, even if they aren’t charged inbetween. This graph shows this phenomenon.
It’s worth looking at the first couple of minutes of the recovery. This is just a zoom-in of the previous graph (and with time axis shown in seconds instead).
Aside from the immediate jump of the voltage when the light bulb is disconnected, there’s a smooth and yet fast rise in the voltage.
Another thing that is evident from this graph, is how much faster the Afik batteries went down towards the 1.67V level, compared with the Vega Power batteries (this refers to the time period before zero).
Is a DC test legit?
This test is made with a DC current, but the UPS discharges the battery with an alternating current: As the UPS feeds the computer with an AC voltage, the delivered power as a function of time is a more or less a sinus function (if the UPS produces a sinus wave). At any time, the energy that is delivered to the computer is drawn from the battery. The UPS barely stores any energy in its own circuits. So the current from the battery is also a sinus, more or less.
So does a test with a DC current reflect the situation with an alternating current? Is it good enough to rely on the average power, and derive a DC current from that? According to Okazaki et al. (“Influence of superimposed alternating current on capacity and cycle life for lead-acid batteries“) the rippled current doesn’t change the battery’s capacity at all. They tested a current of I=I0(1+sinωt) at 0.1Hz to 4 kHz and found only negligible differences (1%). So the DC current test seems to be accurate.
Linear discharge curve?
I’m going back to Vega Power’s second graphs, where I aligned three graphs by shifting them in the X axis. Let’s look at this graph again, but now with a dash-dotted line which represents the first-order linear regression of the segment between 20 and 40 minutes:
So there’s no doubt that the Vega Power batteries’ voltage went down linearly as a function of time for a while. Maybe it was the divergence from this linear descent that made the UPS panic after 28 minutes (once again, the UPS drew a higher current).
Looking at Afik’s curves, they don’t appear linear at any stage. Maybe that’s why the UPS became nervous very soon, but then nothing dramatic happened, so it kept running.
What about the Bulls Power battery? I didn’t show the graph with linear time scale above, but here it is:
Once again, the linear regression is shown with a dash-dotted line. There are fewer measurement points, but it’s quite clear that the voltage descends linearly for a while.
The slopes of these dash-dotted lines are -2.09 mV/minute for the Vega Power batteries, and -2.86 mV/minute for Bulls Power.
What’s the explanation for this linear behavior? Reiterating that lead-acid batteries is not my expertise, I’ll give it a shot:
At the first stage of discharging, the voltage curves seem to be governed mainly by the concentration of sulfuric acid close to the lead plates. As the acid is consumed by the discharging process, the supply of new acid depends on motion of ions between the two plates. A high current means a quick rate of consumption that works against the motion of ions, some of which depends on plain diffusion. So the result is a lower concentration of acid, hence a lower voltage. This, I suppose, explains why the curves go down faster for higher currents, and why the battery recovers quickly afterwards.
But here’s the thing: When the current is constant, the gradient of the sulfuric acid’s concentration stabilizes inside the cell after a few minutes. The motion of ions gets into a stable pace. However, as the current flows, sulfuric acid is consumed. This brings down the concentration of acid overall, even though the differences (gradients) remain the same. The current is constant, and the sulfuric acid is consumed against electrical charge that passes through the battery, so we have a linear slope.
At some point, the curve breaks off from this pattern, and begins diving faster. A possible reason is that the lead plates begin to lose effective surface area (blocked by lead sulfate?), so the current needs to go through a smaller surface. This is equivalent to a larger current, with the due implications on voltage drop.
This seems to be what is often referred to as “deep discharge”. This is a term that is loosely used with reference to drawing too much charge from a battery. Everyone seems to agree that this phase reduces the battery’s capacity on the following cycles, but when does it start? Is it when the curve diverges from the linear pattern? Or when the voltage gets to the level that the manufacturer suggests avoiding?
Academic material on the topic is also scarce (for example, this paper), but it seems like this phase occurs when the lead plates start to take their impact. As evident from the graphs shown above, this phase is rather unpredictable. Even though lead-acid batteries are said not to have a memory effect, that seems to be true until it comes to the lead plates themselves. They remember.
It looks like Smart UPS takes the linear slope approach. Or maybe it looks at the slope of the voltage as a function of time. As long as sulfuric acid is consumed, it’s OK, but when the lead plates begin to deteriorate, that’s time to beep like crazy. This strategy makes a lot of sense for a long battery life, but less beneficial when power outages are rare, so who cares if the battery gets a small hit every few months. But all this is my speculation on how the UPS makes its choices.
Afterthoughts
So what have I learned from all this messing around with the batteries? Not a whole lot of relevant information, I’m afraid. Invented in 1859, lead-acid batteries are easy to manufacture and easy to use, but there doesn’t seem to be a simple theory that accurately describes their behavior. As my own anecdotal experiments show, even the same battery won’t repeat its own discharging curve twice.
Maybe it was because I tried low-quality batteries (or are they?) and inaccurate apparatus for charging the batteries (the Mustek UPS). On the other hand, it’s difficult to find written material that describes the behavior of these batteries, even under lab conditions.
The state of car batteries is often tested by measuring the “cranking current”. This is the huge current when the car is started. This is a crude and indirect measurement of the lead plates’ resistance, which I suppose gives some indication of the battery’s condition. Does it ensure that your battery will hold for the next 12 months? Good luck with that.
Startup idea
The main conclusion from all this rambling is that the only way to know how long a battery will supply power is to fully discharge it. That is, down to a reasonable level that doesn’t harm it. On the other hand, a UPS should be able to give an accurate estimate of how long time it can hold up the computer, so that its user can decide when it’s time to replace the battery or batteries. Plus, the UPS needs to know this for the purpose of shutting down the computer gently a few minutes before it’s game over.
Smart UPS has a feature called “calibration” which does exactly that: It puts the computer on the batteries and discharges them until they reach the lowest acceptable level, and then resumes normal operation. The problem is that if the main power goes out at that moment, there’s nothing to hold up the computer with. So this has to be initiated by the user every now and then.
Solution: Make a UPS with say, four batteries, each of which is connected independently to the UPS’ power conversion circuitry. This way, the UPS can initiate a full discharge test on one of these batteries while the others remain fully charged. At the worst moment, the overall capacity will stand at 75%.
These discharge tests can be initiated automatically by the UPS for each of the four batteries every couple of weeks or so. This way, the UPS can give an accurate number for how long the UPS will hold up power.
This allows the user to decide when it’s time to replace the batteries. It also makes it possible to replace the batteries gradually, without turning off power. And the UPS can also inform the user if the new battery was worth the money.
Will this startup work? I doubt it. The vast majority of people just buy a UPS and believe it will do the work. They realize their mistake when the power goes out for more than two minutes, and buy a new UPS instead. Replacing lead-acid batteries separately? Come on.
A reminder to self: As the title implies, it’s OK to have an empty statement in Verilog. This is useful in order to make all the following clauses active only if the first condition isn’t met. This is commonly seen in C, but I wasn’t sure if it’s OK to do this in Verilog too. So it is.
So this is fine, for example:
always @(posedge clk)
if (relax)
;
else if (this_condition)
do_this <= do_this + 1;
else if (that_condition)
do_that <= do_that + 1;
This is backed up by the IEEE standard 1364-2001 (standard Verilog), which says in Syntax 9-4:
conditional_statement ::=
if ( expression )
statement_or_null [ else statement_or_null ]
| if_else_if_statement
And then defines in section A.6.4:
statement ::=
{ attribute_instance } blocking_assignment ;
| { attribute_instance } case_statement
| { attribute_instance } conditional_statement
| { attribute_instance } disable_statement
| { attribute_instance } event_trigger
| { attribute_instance } loop_statement
| { attribute_instance } nonblocking_assignment ;
| { attribute_instance } par_block
| { attribute_instance } procedural_continuous_assignments ;
| { attribute_instance } procedural_timing_control_statement
| { attribute_instance } seq_block
| { attribute_instance } system_task_enable
| { attribute_instance } task_enable
| { attribute_instance } wait_statement
statement_or_null ::=
statement | { attribute_instance } ;
Those attribute_instance are irrelevant, in particular as they are optional.
Introduction
This is a spin-off post about failing attempts to fix the problem with a webcam’s keyboard buttons. Namely, that the a shaky physical connections caused the USB device to go on and off the bus rapidly, and consequently crash X windows. The background story is in this post.
There is really nothing to learn from this post regarding how to accomplish something. The only reason I don’t trash this is that there’s some possibly useful information about udev.
What I tried to do
There is a possibility to ban a USB device from being accessed by Linux, by virtue of the “authorized” attribute. Something like this.
# cd /sys/devices/pci0000:00/0000:00:14.0/usb2/2-5/
# echo 0 > authorized
^C^Z
# echo 1 > authorized
bash: echo: write error: Invalid argument
The ^C^Z after the first command is not a mistake. The first command got stuck for several seconds.
And this can be done with udev rules as well.
But surprisingly enough, there doesn’t seem to be a way to avoid the generation of the /dev/input/event* file without ignoring the USB device completely. It’s possible to delete it early enough, but that doesn’t really help, it turns out.
ATTRS{authorized} can be set to 0 only for the entire USB device. There is no such parameter for a udev event with the “input” subsystem.
Some udev queries
While trying to figure out the ATTRS{authorized} thing, these are my little play-arounds. Nothing really useful here:
$ sudo udevadm monitor --udev --property
I got
UDEV [5662716.427855] add /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1 (usb)
ACTION=add
BUSNUM=001
DEVNAME=/dev/bus/usb/001/098
DEVNUM=098
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1
DEVTYPE=usb_device
DRIVER=usb
ID_BUS=usb
ID_MODEL=USB2.0_PC_CAMERA
ID_MODEL_ENC=USB2.0\x20PC\x20CAMERA
ID_MODEL_ID=2311
ID_REVISION=0100
ID_SERIAL=Generic_USB2.0_PC_CAMERA
ID_USB_INTERFACES=:0e0100:0e0200:
ID_VENDOR=Generic
ID_VENDOR_ENC=Generic
ID_VENDOR_FROM_DATABASE=GEMBIRD
ID_VENDOR_ID=1908
MAJOR=189
MINOR=97
PRODUCT=1908/2311/100
SEQNUM=24413
SUBSYSTEM=usb
TYPE=239/2/1
USEC_INITIALIZED=5662716427506
UDEV [5662716.430744] add /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.1 (usb)
ACTION=add
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.1
DEVTYPE=usb_interface
DRIVER=uvcvideo
ID_USB_CLASS_FROM_DATABASE=Miscellaneous Device
ID_USB_PROTOCOL_FROM_DATABASE=Interface Association
ID_VENDOR_FROM_DATABASE=GEMBIRD
INTERFACE=14/2/0
MODALIAS=usb:v1908p2311d0100dcEFdsc02dp01ic0Eisc02ip00in01
PRODUCT=1908/2311/100
SEQNUM=24420
SUBSYSTEM=usb
TYPE=239/2/1
USEC_INITIALIZED=5662716430425
UDEV [5662716.430935] add /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0 (usb)
ACTION=add
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0
DEVTYPE=usb_interface
DRIVER=uvcvideo
ID_USB_CLASS_FROM_DATABASE=Miscellaneous Device
ID_USB_PROTOCOL_FROM_DATABASE=Interface Association
ID_VENDOR_FROM_DATABASE=GEMBIRD
INTERFACE=14/1/0
MODALIAS=usb:v1908p2311d0100dcEFdsc02dp01ic0Eisc01ip00in00
PRODUCT=1908/2311/100
SEQNUM=24414
SUBSYSTEM=usb
TYPE=239/2/1
USEC_INITIALIZED=5662716430396
UDEV [5662716.433265] add /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/media5 (media)
ACTION=add
DEVNAME=/dev/media5
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/media5
MAJOR=509
MINOR=5
SEQNUM=24416
SUBSYSTEM=media
USEC_INITIALIZED=5662716433110
UDEV [5662716.435400] bind /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.1 (usb)
ACTION=bind
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.1
DEVTYPE=usb_interface
DRIVER=uvcvideo
ID_USB_CLASS_FROM_DATABASE=Miscellaneous Device
ID_USB_PROTOCOL_FROM_DATABASE=Interface Association
ID_VENDOR_FROM_DATABASE=GEMBIRD
INTERFACE=14/2/0
MODALIAS=usb:v1908p2311d0100dcEFdsc02dp01ic0Eisc02ip00in01
PRODUCT=1908/2311/100
SEQNUM=24421
SUBSYSTEM=usb
TYPE=239/2/1
USEC_INITIALIZED=5662716430425
UDEV [5662716.436539] add /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/video4linux/video0 (video4linux)
ACTION=add
COLORD_DEVICE=1
COLORD_KIND=camera
DEVLINKS=/dev/v4l/by-id/usb-Generic_USB2.0_PC_CAMERA-video-index0 /dev/v4l/by-path/pci-0000:00:14.0-usb-0:5.2.1:1.0-video-index0
DEVNAME=/dev/video0
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/video4linux/video0
ID_BUS=usb
ID_FOR_SEAT=video4linux-pci-0000_00_14_0-usb-0_5_2_1_1_0
ID_MODEL=USB2.0_PC_CAMERA
ID_MODEL_ENC=USB2.0\x20PC\x20CAMERA
ID_MODEL_ID=2311
ID_PATH=pci-0000:00:14.0-usb-0:5.2.1:1.0
ID_PATH_TAG=pci-0000_00_14_0-usb-0_5_2_1_1_0
ID_REVISION=0100
ID_SERIAL=Generic_USB2.0_PC_CAMERA
ID_TYPE=video
ID_USB_DRIVER=uvcvideo
ID_USB_INTERFACES=:0e0100:0e0200:
ID_USB_INTERFACE_NUM=00
ID_V4L_CAPABILITIES=:capture:
ID_V4L_PRODUCT=USB2.0 PC CAMERA: USB2.0 PC CAM
ID_V4L_VERSION=2
ID_VENDOR=Generic
ID_VENDOR_ENC=Generic
ID_VENDOR_ID=1908
MAJOR=81
MINOR=0
SEQNUM=24415
SUBSYSTEM=video4linux
TAGS=:seat:uaccess:
USEC_INITIALIZED=5662716436054
UDEV [5662716.436956] add /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/input/input121 (input)
ACTION=add
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/input/input121
EV=3
ID_BUS=usb
ID_FOR_SEAT=input-pci-0000_00_14_0-usb-0_5_2_1_1_0
ID_INPUT=1
ID_INPUT_KEY=1
ID_MODEL=USB2.0_PC_CAMERA
ID_MODEL_ENC=USB2.0\x20PC\x20CAMERA
ID_MODEL_ID=2311
ID_PATH=pci-0000:00:14.0-usb-0:5.2.1:1.0
ID_PATH_TAG=pci-0000_00_14_0-usb-0_5_2_1_1_0
ID_REVISION=0100
ID_SERIAL=Generic_USB2.0_PC_CAMERA
ID_TYPE=video
ID_USB_DRIVER=uvcvideo
ID_USB_INTERFACES=:0e0100:0e0200:
ID_USB_INTERFACE_NUM=00
ID_VENDOR=Generic
ID_VENDOR_ENC=Generic
ID_VENDOR_ID=1908
KEY=100000 0 0 0
MODALIAS=input:b0003v1908p2311e0100-e0,1,kD4,ramlsfw
NAME="USB2.0 PC CAMERA: USB2.0 PC CAM"
PHYS="usb-0000:00:14.0-5.2.1/button"
PRODUCT=3/1908/2311/100
PROP=0
SEQNUM=24417
SUBSYSTEM=input
TAGS=:seat:
USEC_INITIALIZED=5662716436500
UDEV [5662716.591160] add /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/input/input121/event22 (input)
ACTION=add
BACKSPACE=guess
DEVLINKS=/dev/input/by-path/pci-0000:00:14.0-usb-0:5.2.1:1.0-event /dev/input/by-id/usb-Generic_USB2.0_PC_CAMERA-event-if00
DEVNAME=/dev/input/event22
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/input/input121/event22
ID_BUS=usb
ID_INPUT=1
ID_INPUT_KEY=1
ID_MODEL=USB2.0_PC_CAMERA
ID_MODEL_ENC=USB2.0\x20PC\x20CAMERA
ID_MODEL_ID=2311
ID_PATH=pci-0000:00:14.0-usb-0:5.2.1:1.0
ID_PATH_TAG=pci-0000_00_14_0-usb-0_5_2_1_1_0
ID_REVISION=0100
ID_SERIAL=Generic_USB2.0_PC_CAMERA
ID_TYPE=video
ID_USB_DRIVER=uvcvideo
ID_USB_INTERFACES=:0e0100:0e0200:
ID_USB_INTERFACE_NUM=00
ID_VENDOR=Generic
ID_VENDOR_ENC=Generic
ID_VENDOR_ID=1908
LIBINPUT_DEVICE_GROUP=3/1908/2311:usb-0000:00:14.0-5.2
MAJOR=13
MINOR=86
SEQNUM=24418
SUBSYSTEM=input
TAGS=:power-switch:
USEC_INITIALIZED=5662716590816
XKBLAYOUT=us,il
XKBMODEL=pc105
XKBOPTIONS=grp:alt_shift_toggle,grp_led:scroll
XKBVARIANT=,
UDEV [5662716.593390] bind /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0 (usb)
ACTION=bind
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0
DEVTYPE=usb_interface
DRIVER=uvcvideo
ID_USB_CLASS_FROM_DATABASE=Miscellaneous Device
ID_USB_PROTOCOL_FROM_DATABASE=Interface Association
ID_VENDOR_FROM_DATABASE=GEMBIRD
INTERFACE=14/1/0
MODALIAS=usb:v1908p2311d0100dcEFdsc02dp01ic0Eisc01ip00in00
PRODUCT=1908/2311/100
SEQNUM=24419
SUBSYSTEM=usb
TYPE=239/2/1
USEC_INITIALIZED=5662716430396
UDEV [5662716.595836] bind /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1 (usb)
ACTION=bind
BUSNUM=001
DEVNAME=/dev/bus/usb/001/098
DEVNUM=098
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1
DEVTYPE=usb_device
DRIVER=usb
ID_BUS=usb
ID_MODEL=USB2.0_PC_CAMERA
ID_MODEL_ENC=USB2.0\x20PC\x20CAMERA
ID_MODEL_ID=2311
ID_REVISION=0100
ID_SERIAL=Generic_USB2.0_PC_CAMERA
ID_USB_INTERFACES=:0e0100:0e0200:
ID_VENDOR=Generic
ID_VENDOR_ENC=Generic
ID_VENDOR_FROM_DATABASE=GEMBIRD
ID_VENDOR_ID=1908
MAJOR=189
MINOR=97
PRODUCT=1908/2311/100
SEQNUM=24422
SUBSYSTEM=usb
TYPE=239/2/1
USEC_INITIALIZED=5662716427506
So the device I want to avoid was /dev/input/event22 this time. What’s its attributes?
$ sudo udevadm info -a -n /dev/input/event22
Udevadm info starts with the device specified by the devpath and then
walks up the chain of parent devices. It prints for every device
found, all possible attributes in the udev rules key format.
A rule to match, can be composed by the attributes of the device
and the attributes from one single parent device.
looking at device '/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/input/input121/event22':
KERNEL=="event22"
SUBSYSTEM=="input"
DRIVER==""
looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/input/input121':
KERNELS=="input121"
SUBSYSTEMS=="input"
DRIVERS==""
ATTRS{name}=="USB2.0 PC CAMERA: USB2.0 PC CAM"
ATTRS{phys}=="usb-0000:00:14.0-5.2.1/button"
ATTRS{properties}=="0"
ATTRS{uniq}==""
looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0':
KERNELS=="1-5.2.1:1.0"
SUBSYSTEMS=="usb"
DRIVERS=="uvcvideo"
ATTRS{authorized}=="1"
ATTRS{bAlternateSetting}==" 0"
ATTRS{bInterfaceClass}=="0e"
ATTRS{bInterfaceNumber}=="00"
ATTRS{bInterfaceProtocol}=="00"
ATTRS{bInterfaceSubClass}=="01"
ATTRS{bNumEndpoints}=="01"
ATTRS{iad_bFirstInterface}=="00"
ATTRS{iad_bFunctionClass}=="0e"
ATTRS{iad_bFunctionProtocol}=="00"
ATTRS{iad_bFunctionSubClass}=="03"
ATTRS{iad_bInterfaceCount}=="02"
ATTRS{interface}=="USB2.0 PC CAMERA"
ATTRS{supports_autosuspend}=="1"
looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1':
KERNELS=="1-5.2.1"
SUBSYSTEMS=="usb"
DRIVERS=="usb"
ATTRS{authorized}=="1"
ATTRS{avoid_reset_quirk}=="0"
ATTRS{bConfigurationValue}=="1"
ATTRS{bDeviceClass}=="ef"
ATTRS{bDeviceProtocol}=="01"
ATTRS{bDeviceSubClass}=="02"
ATTRS{bMaxPacketSize0}=="64"
ATTRS{bMaxPower}=="256mA"
ATTRS{bNumConfigurations}=="1"
ATTRS{bNumInterfaces}==" 2"
ATTRS{bcdDevice}=="0100"
ATTRS{bmAttributes}=="80"
ATTRS{busnum}=="1"
ATTRS{configuration}==""
ATTRS{devnum}=="98"
ATTRS{devpath}=="5.2.1"
ATTRS{idProduct}=="2311"
ATTRS{idVendor}=="1908"
ATTRS{ltm_capable}=="no"
ATTRS{manufacturer}=="Generic"
ATTRS{maxchild}=="0"
ATTRS{product}=="USB2.0 PC CAMERA"
ATTRS{quirks}=="0x0"
ATTRS{removable}=="unknown"
ATTRS{speed}=="480"
ATTRS{urbnum}=="16"
ATTRS{version}==" 2.00"
looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2':
KERNELS=="1-5.2"
SUBSYSTEMS=="usb"
DRIVERS=="usb"
ATTRS{authorized}=="1"
ATTRS{avoid_reset_quirk}=="0"
ATTRS{bConfigurationValue}=="1"
ATTRS{bDeviceClass}=="09"
ATTRS{bDeviceProtocol}=="01"
ATTRS{bDeviceSubClass}=="00"
ATTRS{bMaxPacketSize0}=="64"
ATTRS{bMaxPower}=="100mA"
ATTRS{bNumConfigurations}=="1"
ATTRS{bNumInterfaces}==" 1"
ATTRS{bcdDevice}=="0100"
ATTRS{bmAttributes}=="e0"
ATTRS{busnum}=="1"
ATTRS{configuration}==""
ATTRS{devnum}=="75"
ATTRS{devpath}=="5.2"
ATTRS{idProduct}=="7250"
ATTRS{idVendor}=="214b"
ATTRS{ltm_capable}=="no"
ATTRS{maxchild}=="4"
ATTRS{product}=="USB2.0 HUB"
ATTRS{quirks}=="0x0"
ATTRS{removable}=="unknown"
ATTRS{speed}=="480"
ATTRS{urbnum}=="409"
ATTRS{version}==" 2.00"
looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1/1-5':
KERNELS=="1-5"
SUBSYSTEMS=="usb"
DRIVERS=="usb"
ATTRS{authorized}=="1"
ATTRS{avoid_reset_quirk}=="0"
ATTRS{bConfigurationValue}=="1"
ATTRS{bDeviceClass}=="09"
ATTRS{bDeviceProtocol}=="02"
ATTRS{bDeviceSubClass}=="00"
ATTRS{bMaxPacketSize0}=="64"
ATTRS{bMaxPower}=="0mA"
ATTRS{bNumConfigurations}=="1"
ATTRS{bNumInterfaces}==" 1"
ATTRS{bcdDevice}=="0123"
ATTRS{bmAttributes}=="e0"
ATTRS{busnum}=="1"
ATTRS{configuration}==""
ATTRS{devnum}=="73"
ATTRS{devpath}=="5"
ATTRS{idProduct}=="5411"
ATTRS{idVendor}=="0bda"
ATTRS{ltm_capable}=="no"
ATTRS{manufacturer}=="Generic"
ATTRS{maxchild}=="4"
ATTRS{product}=="4-Port USB 2.0 Hub"
ATTRS{quirks}=="0x0"
ATTRS{removable}=="removable"
ATTRS{speed}=="480"
ATTRS{urbnum}=="69"
ATTRS{version}==" 2.10"
looking at parent device '/devices/pci0000:00/0000:00:14.0/usb1':
KERNELS=="usb1"
SUBSYSTEMS=="usb"
DRIVERS=="usb"
ATTRS{authorized}=="1"
ATTRS{authorized_default}=="1"
ATTRS{avoid_reset_quirk}=="0"
ATTRS{bConfigurationValue}=="1"
ATTRS{bDeviceClass}=="09"
ATTRS{bDeviceProtocol}=="01"
ATTRS{bDeviceSubClass}=="00"
ATTRS{bMaxPacketSize0}=="64"
ATTRS{bMaxPower}=="0mA"
ATTRS{bNumConfigurations}=="1"
ATTRS{bNumInterfaces}==" 1"
ATTRS{bcdDevice}=="0415"
ATTRS{bmAttributes}=="e0"
ATTRS{busnum}=="1"
ATTRS{configuration}==""
ATTRS{devnum}=="1"
ATTRS{devpath}=="0"
ATTRS{idProduct}=="0002"
ATTRS{idVendor}=="1d6b"
ATTRS{interface_authorized_default}=="1"
ATTRS{ltm_capable}=="no"
ATTRS{manufacturer}=="Linux 4.15.0-20-generic xhci-hcd"
ATTRS{maxchild}=="16"
ATTRS{product}=="xHCI Host Controller"
ATTRS{quirks}=="0x0"
ATTRS{removable}=="unknown"
ATTRS{serial}=="0000:00:14.0"
ATTRS{speed}=="480"
ATTRS{urbnum}=="454"
ATTRS{version}==" 2.00"
looking at parent device '/devices/pci0000:00/0000:00:14.0':
KERNELS=="0000:00:14.0"
SUBSYSTEMS=="pci"
DRIVERS=="xhci_hcd"
ATTRS{broken_parity_status}=="0"
ATTRS{class}=="0x0c0330"
ATTRS{consistent_dma_mask_bits}=="64"
ATTRS{d3cold_allowed}=="1"
ATTRS{dbc}=="disabled"
ATTRS{device}=="0xa2af"
ATTRS{dma_mask_bits}=="64"
ATTRS{driver_override}=="(null)"
ATTRS{enable}=="1"
ATTRS{irq}=="33"
ATTRS{local_cpulist}=="0-11"
ATTRS{local_cpus}=="0,00000000,00000fff"
ATTRS{msi_bus}=="1"
ATTRS{numa_node}=="0"
ATTRS{revision}=="0x00"
ATTRS{subsystem_device}=="0x5007"
ATTRS{subsystem_vendor}=="0x1458"
ATTRS{vendor}=="0x8086"
looking at parent device '/devices/pci0000:00':
KERNELS=="pci0000:00"
SUBSYSTEMS==""
DRIVERS==""
And what udev rules are currently in effect for this? Note that this doesn’t require root, and nothing really happens to the system:
$ udevadm test -a add $(udevadm info -q path -n /dev/input/event22)calling: test
version 237
This program is for debugging only, it does not run any program
specified by a RUN key. It may show incorrect results, because
some values may be different, or not available at a simulation run.
Load module index
Parsed configuration file /etc/systemd/network/eth1.link
Skipping empty file: /etc/systemd/network/99-default.link
Created link configuration context.
[ ... reading a lot of files ... ]
rules contain 393216 bytes tokens (32768 * 12 bytes), 39371 bytes strings
25632 strings (220044 bytes), 22252 de-duplicated (184054 bytes), 3381 trie nodes used
GROUP 104 /lib/udev/rules.d/50-udev-default.rules:29
IMPORT builtin 'hwdb' /lib/udev/rules.d/60-evdev.rules:8
IMPORT builtin 'hwdb' returned non-zero
IMPORT builtin 'hwdb' /lib/udev/rules.d/60-evdev.rules:17
IMPORT builtin 'hwdb' returned non-zero
IMPORT builtin 'hwdb' /lib/udev/rules.d/60-evdev.rules:21
IMPORT builtin 'hwdb' returned non-zero
IMPORT builtin 'input_id' /lib/udev/rules.d/60-input-id.rules:5
capabilities/ev raw kernel attribute: 3
capabilities/abs raw kernel attribute: 0
capabilities/rel raw kernel attribute: 0
capabilities/key raw kernel attribute: 100000 0 0 0
properties raw kernel attribute: 0
test_key: checking bit block 0 for any keys; found=0
test_key: checking bit block 64 for any keys; found=0
test_key: checking bit block 128 for any keys; found=0
test_key: checking bit block 192 for any keys; found=1
IMPORT builtin 'hwdb' /lib/udev/rules.d/60-input-id.rules:6
IMPORT builtin 'hwdb' returned non-zero
IMPORT builtin 'usb_id' /lib/udev/rules.d/60-persistent-input.rules:11
/sys/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0: if_class 14 protocol 0
LINK 'input/by-id/usb-Generic_USB2.0_PC_CAMERA-event-if00' /lib/udev/rules.d/60-persistent-input.rules:32
IMPORT builtin 'path_id' /lib/udev/rules.d/60-persistent-input.rules:35
LINK 'input/by-path/pci-0000:00:14.0-usb-0:5.2.1:1.0-event' /lib/udev/rules.d/60-persistent-input.rules:40
PROGRAM 'libinput-device-group /sys/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/input/input121/event22' /lib/udev/rules.d/80-libinput-device-groups.rules:7
starting 'libinput-device-group /sys/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/input/input121/event22'
'libinput-device-group /sys/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/input/input121/event22'(out) '3/1908/2311:usb-0000:00:14.0-5.2'
Process 'libinput-device-group /sys/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/input/input121/event22' succeeded.
IMPORT builtin 'hwdb' /lib/udev/rules.d/90-libinput-model-quirks.rules:46
IMPORT builtin 'hwdb' returned non-zero
IMPORT builtin 'hwdb' /lib/udev/rules.d/90-libinput-model-quirks.rules:50
IMPORT builtin 'hwdb' returned non-zero
handling device node '/dev/input/event22', devnum=c13:86, mode=0660, uid=0, gid=104
preserve permissions /dev/input/event22, 020660, uid=0, gid=104
preserve already existing symlink '/dev/char/13:86' to '../input/event22'
found 'c13:86' claiming '/run/udev/links/\x2finput\x2fby-id\x2fusb-Generic_USB2.0_PC_CAMERA-event-if00'
found 'c13:85' claiming '/run/udev/links/\x2finput\x2fby-id\x2fusb-Generic_USB2.0_PC_CAMERA-event-if00'
found 'c13:84' claiming '/run/udev/links/\x2finput\x2fby-id\x2fusb-Generic_USB2.0_PC_CAMERA-event-if00'
found 'c13:83' claiming '/run/udev/links/\x2finput\x2fby-id\x2fusb-Generic_USB2.0_PC_CAMERA-event-if00'
creating link '/dev/input/by-id/usb-Generic_USB2.0_PC_CAMERA-event-if00' to '/dev/input/event22'
preserve already existing symlink '/dev/input/by-id/usb-Generic_USB2.0_PC_CAMERA-event-if00' to '../event22'
found 'c13:86' claiming '/run/udev/links/\x2finput\x2fby-path\x2fpci-0000:00:14.0-usb-0:5.2.1:1.0-event'
found 'c13:85' claiming '/run/udev/links/\x2finput\x2fby-path\x2fpci-0000:00:14.0-usb-0:5.2.1:1.0-event'
found 'c13:84' claiming '/run/udev/links/\x2finput\x2fby-path\x2fpci-0000:00:14.0-usb-0:5.2.1:1.0-event'
found 'c13:83' claiming '/run/udev/links/\x2finput\x2fby-path\x2fpci-0000:00:14.0-usb-0:5.2.1:1.0-event'
creating link '/dev/input/by-path/pci-0000:00:14.0-usb-0:5.2.1:1.0-event' to '/dev/input/event22'
preserve already existing symlink '/dev/input/by-path/pci-0000:00:14.0-usb-0:5.2.1:1.0-event' to '../event22'
ACTION=add
BACKSPACE=guess
DEVLINKS=/dev/input/by-path/pci-0000:00:14.0-usb-0:5.2.1:1.0-event /dev/input/by-id/usb-Generic_USB2.0_PC_CAMERA-event-if00
DEVNAME=/dev/input/event22
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5.2/1-5.2.1/1-5.2.1:1.0/input/input121/event22
ID_BUS=usb
ID_INPUT=1
ID_INPUT_KEY=1
ID_MODEL=USB2.0_PC_CAMERA
ID_MODEL_ENC=USB2.0\x20PC\x20CAMERA
ID_MODEL_ID=2311
ID_PATH=pci-0000:00:14.0-usb-0:5.2.1:1.0
ID_PATH_TAG=pci-0000_00_14_0-usb-0_5_2_1_1_0
ID_REVISION=0100
ID_SERIAL=Generic_USB2.0_PC_CAMERA
ID_TYPE=video
ID_USB_DRIVER=uvcvideo
ID_USB_INTERFACES=:0e0100:0e0200:
ID_USB_INTERFACE_NUM=00
ID_VENDOR=Generic
ID_VENDOR_ENC=Generic
ID_VENDOR_ID=1908
LIBINPUT_DEVICE_GROUP=3/1908/2311:usb-0000:00:14.0-5.2
MAJOR=13
MINOR=86
SUBSYSTEM=input
TAGS=:power-switch:
USEC_INITIALIZED=5662716590816
XKBLAYOUT=us,il
XKBMODEL=pc105
XKBOPTIONS=grp:alt_shift_toggle,grp_led:scroll
XKBVARIANT=,
Unload module index
Unloaded link configuration context.
Other failed attempts
I tried the following:
# Rule for disabling bogus keyboard on webcam. It causes X-Windows to
# crash if it goes on and off too much
SUBSYSTEM=="input", ENV{ID_VENDOR_ID}=="1908", ENV{ID_MODEL_ID}=="2311", MODE:="000"
SUBSYSTEM=="input", ATTRS{name}=="USB2.0 PC CAMERA:*", ENV{LIBINPUT_IGNORE_DEVICE}="1"
(the := assignment makes this assignment final).
However none of these two rules managed to stop X from reacting.
Setting the mode to 000 made the device file inaccessible, but yet it was registered. As for the second rule, it doesn’t help, because it indeed set LIBINPUT_IGNORE_DEVICE correctly, but for the wrong udev event. That’s because the udev event that triggers libinput is based upon that the KERNEL attribute is event[0-9]*, which is executed earlier (see 80-libinput-device-groups.rules), but ATTRS{name} isn’t defined for that specific udev event (see output of udevadm info above).
I also tried RUN+=”/bin/rm /dev/input/event%n”, and that indeed removed the device node, but X still reacted, and complained with “libinput: USB2.0 PC CAMERA: USB2.0 PC CAM: Failed to create a device for /dev/input/event28″. Because it was indeed deleted.
But since it appears like X.org accesses keyboards through libinput, maybe use the example for ignoring a device, as given on this page, even though it’s quite similar to what I’ve already attempted?
So I saved this file as /etc/udev/rules.d/79-no-camera-keyboard.rules:
# Make libinput ignore webcam's button as a keyboard. As a result there's
# no event to X-Windows
ACTION=="add|change", KERNEL=="event[0-9]*", \
ENV{ID_VENDOR_ID}=="1908", \
ENV{ID_MODEL_ID}=="2311", \
ENV{LIBINPUT_IGNORE_DEVICE}="1"
And then reload:
# udevadm control --reload
but that didn’t make any apparent difference (I verified that the rule was matched).
And that’s all, folks. Recall that I didn’t promise a happy end.
Who ate my newline?
It’s 2023, Perl is ranked below COBOL, but I still consider it as my loyal working horse. But even the most loyal horse will give you a grand kick in the bottom every now and then.
So let’s jump to the problematic code:
use warnings;
use strict;
my $str = ".\n\n";
my $nonn = qr/[ \t]|(?<!\n)\n(?!\n)/;
my ($pre, $match, $post) = ($str =~ /^($nonn*)(.*?)($nonn*)$/s);
print "pre = \"$pre\"\n";
print "match = \"$match\"\n";
print "post = \"$post\"\n";
print "This doesn't add up!\n"
unless ($str eq "$pre$match$post");
For now, never mind what I tried to do here. Let’s just note that $nonn doesn’t capture anything: Those two expressions with parentheses are a lookbehind and a lookahead, and hence don’t capture.
So now let’s look at
my ($pre, $match, $post) = ($str =~ /^($nonn*)(.*?)($nonn*)$/s);
This is an enclosure between ^ and $, and everything in the middle is captured into three matches. So no matter what, the concatenation of these three matches should equal $str, shouldn’t it? Let’s give it a test run:
$ ./try.pl
pre = ""
match = ".
"
post = ""
This doesn't add up!
So $pre and $post are empty. OK, fine. Hence $match should equal $str, which is “.\n\n”. But I see only one newline. Where’s the other one?
RTFM
The one thing that I really like about Perl, is that even when it plays a dirty trick, the answer is in the plain manual. As in “man perlre”, where it says, black on white in the description of $:
Match the end of the string (or before newline at the end of the string; or before any newline if /m is used)
So there we have it. “$” can also consider the character before the last newline as the end. Note that “$” itself will not match the last newline, so even if there’s a capture on the “$” itself, as in “($)”, that last newline is still not captured. It’s a Perl quirk. One of those things that make Perl do exactly what you really want, except for when you’re surgical about it.
I’ve been using Perl a lot for 20 years, and I wasn’t aware that “$” could match anything but the end of the string (let alone the “/m” modifier).
So that’s what happened above: $ considered the character before the last newline to be the end, and one newline went up in smoke.
Use \z instead
The second thing that I really like about Perl, is that even when it’s quirky, there’s always a simple solution. The same “man perlre” also says:
\z Match only at end of string
Simple, isn’t it? From now on and until the end of time, always use \z if you really mean the end of string. Like, character-wise. And if I change “$” to “\z” in the code above, I get:
my ($pre, $match, $post) = ($str =~ /^($nonn*)(.*?)($nonn*)\z/s);
and the test run gives:
$ ./try.pl
pre = ""
match = ".
"
post = ""
The working horse is back on track again.
What I really wanted to do
Since I messed up with this regex, I should maybe explain what it does:
my $nonn = qr/[ \t]|(?<!\n)\n(?!\n)/;
First, let’s note that $nonn only matches one character (or none): It’s either a plain space, a tab or a newline. But what’s the mess with the newline?
The “(?<!\n)\n(?!\n)” part says this: Match a \n character that isn’t preceded by a \n, and isn’t followed by a \n. Or in other words, match a newline only if it isn’t part of a sequence of newlines. Only if it’s one, isolated \n.
No double \n. Or for short, “nonn”.
I needed this for a script that handles multiple newlines later on (in LaTeX, a double newline means a new paragraph, that’s the reason).
And it actually worked. The “\n\n” part in the string wasn’t matched into neither $pre nor $post. But the (.*?), which attempts to match as little as possible, sold off the last newline to $. Tricky stuff.
Oh my goodness
I’ll say this from the start: This is me babbling. You have been warned. Don’t expect anything coherent in this post. Not that I necessarily keep things very tidy in my other posts, but this one is clearly me typing at full speed, not trying to be organized or anything.
I’ll buy you a pizza if you figure out the real motivation behind this post. Well, actually I won’t. At most, I’ll answer with an emoji with a pizza. But you’re welcome to try your luck anyhow. If you bother to read this, that is. Which I’m not sure is the best way to spend your time.
So these are my thoughts after publishing a series of pages about FPGA timing and timing constraints (which begins with this page). It’s not only a relatively long series of pages, but it also involves the translation of the pages into Chinese, Japanese and Korean. I know none of these languages, so I used Google Translate. I’ve already written a post about the challenges and solutions with automatic translation of technical documents.
The translations are a huge experiment. It’s not clear at all if people who speak these languages will find them appealing. There’s clearly a readability problem with an automatic translation, no matter how hard I work to make the text digestible for an AI machine.
People who speak English and one of these languages have expressed little enthusiasm when looking at the results (to say the least). But then it boils down to the choices of people who only have that East Asian language. They will surely have a laugh or curse when reading encountering the hick-ups of automatic translations, but do they have a better alternative? Time will tell. Or more precisely, the logs will tell.
I’ve never written a behind-the-scenes post about other things that I’ve written. Maybe I do this because it’s the first time in ages that I can write English exactly the way I think, and not keeping restricting myself to write in a way that works well with the translation machinery. One of the main limitations with automatic translation to CJK languages is that the sentences must be short and concise. The ordering of words is very different in those languages, so if I use a word like “it” or “which” that refers back to something mentioned earlier in the sentence, the translation machinery is forced to decide what it referred to, and name it explicitly in the translation. And quite often, that goes wrong. When the translation is to another European language, or even Hebrew, it’s possible to retain the ambiguity in the translation, and leave it to the intelligence of the reader.
So if this post feels like a huge ramble, yes, that’s exactly what it is. Me feeling free at the keyboard. The liberation from thinking how each and every word may be mistranslated. I mean, even a work like “generation” in the sense of creating something can be translated into like generation gap. Or if I say “that’s a special case”, will that become something about a bag? I’m really tired of translating and back-translating and cutting down Chinese into single characters in order to ensure that when I write “segment” I don’t get something related to marketing.
Let’s talk about timing
I’ve been thinking about writing a guide to FPGA timing and timing constraints for quite a while. It’s one of the most important topics, if not the most important topic in the field, and somehow it seems like the explanations about this field are either very theoretic, or they’re so specific that it’s difficult to understand the whole point. It was clear to me that I needed to begin with the basics of the theory (and translate it, of course). I was however quite surprised myself how little theory there was to convey. The entire thing is about setup time, hold time and clock-to-output. That’s it. That’s the whole story.
It was quite clear where to take it from there: To discuss the most commonly used timing constraint, that is to say the period constraint. And make that international, once again.
But how do you talk about this simple constraint? I mean, the normal explanation is just “this is what you should write”. How do you explain what it really means?
So the approach that I went for throughout the series of pages was to explain it through the timing reports. It’s a bit of a tedious way, but the idea was that the correct way to understand timing constraints, and in fact one of the most essential capabilities for working with timing, is to read the timing reports. Hence I went for explaining the timing report.
Another alternative could have been to jump into explaining the meaning of the timing constraint as a clock object. This is a more technical approach, and it’s necessary to go through it at some stage. But I deferred that to later.
To PLL or not to PLL
And by the way, that’s a title that I really wanted somewhere in those pages, but it was clearly going to be lost in translation. So all you guys from East Asia out there, if you won’t read the translated pages, just know I sacrificed proper English in favor of you getting something decent to read. Kindly appreciate it.
Anyhow, it turned out that I couldn’t finish the entire topic of the period constraint in one page, so there’s a second page on the same topic, this time for the sake of explaining how a PLL changes the picture. And that brings up the whole story of clock domain crossing between related clocks. Plus I also filled in the part about hold time.
Hold time is a neglected topic regarding timing constraints. Probably because the timing constraints rarely fail on that, and still: The tools deliberately add delays to the paths in order to meet hold timing, and by doing so, they put an obstacle before themselves in the quest to attain the setup timing. It’s quite amazing to see how a simple routed net suddenly gets a delay of several ns, just because the hold timing required so. In particular when crossing clock domains between related, but unaligned clocks.
And that’s another expression I couldn’t use on those pages: Crossing clock domains. It always says “clock domain crossing”, and it’s protected against translation. Otherwise, I get crucifixion in the translation. And the protection against translation works only on nouns, not verbs. Did I just divert from the topic? Maybe I need a closure?
Timing closure
When is the right time to talk about it? Timing closure is often the reason we even start think about timing (ghosts in the FPGA is another common reason). So it’s definitely an important topic, at least to the impatient reader. But shouldn’t I put it in the end? After all, it’s the ultimate surgical operation that is done once you understand it all.
So no. I went with the people on this one. It’s the ultimate topic, yes, but I also need to realize that some people will read one page or two and then give up. So better get to the important thing ASAP. Actually, I also gave away a checklist kind of page (with trans-la-tions). Maybe that’s the biggest contradiction I made in those pages. I said all the time that timing is like a divine science, and don’t be lazy about it. Be deep and profound, learn it as an art, and then I come up with a checklist. Not only that, I repeat that crime at the end of the page series. And I do that in three additional languages, too.
To my defense (or is it?), the repeated crime had a practical and silly reason. I had that page already written. More or less. So it was a bit of low-hanging fruit. And my rule (in this blog and otherwise) has always been not to throw away food. If you have some random pieces of cheese in your fridge, make a quiche. If there is mold on the cheese, you just make a different kind of quiche.
So that last page in the series is not 100% coherent with those preceding it, but it kinda summarizes the point. Besides, I already mentioned that the whole series of pages is one big experiment…?
Tachles
That’s a word in Jiddish, literally taken from the Hebrew word תכלית. Actually, it’s the pronunciation of that word in the mouth of European Jews. The meaning of the word is “purpose”, the reason for doing something, but more in the direction of the intention, what result you want to achieve etc.
So after talking about timing closure for two pages, I finally got to the pudding: Actually talk about how to write timing constraint. But not so fast! There’s still one tedious phase to go through, namely learn how to select the logic that we want to be specific about.
This whole thing with Tcl commands, which are used in SDC format (that is, by Vivado, Quartus and several other tools) gives a certain level of compatibility. But as it turns out, there’s Synopsys’ set of capabilities, which seems to have been adopted in full by Vivado, and then there’s Quartus’ take on the same topic, which is same-same but completely different. So the simple commands work the same, but when it comes to more complicated stuff, Quartus diverges, and not for the better. In particular, Quartus lacks the “-filter” option that allows more complicated expressions for defining whether to include a logic element (actually, the object representing it) in a timing constraint. So one is limited to playing with wildcards, and that, well, sucks.
So in short, this was the point I had to admit that Vivado is superior, with a margin. I also began to suspect that Vivado is Synopsys in disguise, but that’s just a wild speculation.
It’s only after that, that I could begin to really talk about commands like set_max_delay, set_min_delay and set_false_path. That’s what I call tachles. I finally got to the point, after rambling tons.
I think that the most important point about set_false_path is that it has maximal priority. One would usually consider priority to be an advanced topic.
Ah, and here comes another messy, out-of-nowhere comment about this series of pages. The idea wasn’t to write a full and comprehensive guide that includes everything that there is possible to know. That’s what the official docs are for. The idea was to cover the things that I personally use and feel that I must be aware of, and hence the convey the minimal toolbox that one needs to carry.
Return from subroutine: I didn’t want to cover an advanced topic like priorities between timing constraints, because who really needs that? It’s quite clear that those specific timing constraints are stronger than the period constraint, and except for that, nobody should mess up with more complicated combinations of contradicting constraints.
But set_false_path is scary: It wins everything else, and it disables the timing check of the relevant paths silently. It’s literally a killer. So make a small mistake with that constraint, and your design is cooked. So I spent quite a few words on warnings about that.
Eight o’clock rock
When I first started working with FPGAs, my mentors were from the ASIC industry. For some reason, which I’m still to find out, ASIC guys like to everything with one clock, if possible. In the FPGA world, it’s more a like clock being given away like they were air. So the whole issue of timing constraints needs to be addressed in the context of crossing clock domains.
And here’s another word that I avoided all the time: Addressed. Because who knows what that will be translated into, when it comes to Chinese, Japanese and Korean.
The alternative to a lot of clocks, is multi-cycle paths, of course: This is what the ASIC guys do in order to stick to one clock for the entire design. And in order to take advantage of the relatively relaxed timing of logic that depends on a clock enable, there are timing constraints. Which is a risky business. I won’t repeat the explanation here, but let’s say that it’s generally not recommended.
Who you wanna talk to?
One of the things that I actually planned and carried out as planned (well, not really as planned) was to put the discussion about I/O constraints last. This went against the principle of important things first, but since I/O constraints are technically just a private case of constraints on internal paths (that is, paths that start and end inside the FPGA), it made a lot of sense to put this part last.
Now read this last sentence again, and realize that there’s nothing even close to that length in the pages I wrote on timing constraint. No chance a sentence like that would have been translated correctly.
So where were I? Oh yes, timing constraint for I/O. But soon enough I realized that I can’t talk about timing constraints for I/O before discussing system synchronous vs. source synchronous vs. asynchronous I/O. But it’s not that simple. One can’t just throw these terms in the air and hope for good. So all of the sudden I found myself with a small בלת”ם (Read “baltam”, meaning “unplanned” in Hebrew), which wasn’t so small.
That’s because source-synchronous I/O means both inputs and outputs, and these are implemented in different ways. So there was no choice (or was there?) to sit down and explain how to do that. How can you discuss the possibilities for timing constraints of a source-synchronous input without explaining how the I/O is implemented. There are quite a few possibilities, after all.
Source-synchronous outputs are actually a much simpler story, because it just means to generate all signals with registers.
On top of all this mess, there’s another topic which is important both for the timing of I/O and for ensuring that the design behaves in a consistent way: IOB registers.
But oops. IOB is a Xilinx word. Oddly enough, it doesn’t seem like there’s a standard term for the idea of having a flip-flop close to the I/O pin. It’s such a common thing, and yet there’s no common term. I think. So I used the term I’m used to, IOB. It’s quite apparent that I’m a Xilinx guy. I’ve worked a bit with Altera (should I call them AMD and Intel FPGA by now? It takes time to get used to that). I nevertheless tried to be vendor-neutral in those pages. They were supposed to be about all FPGAs, not only the two that were leading back in 202x, just before being acquired by two giants, leaving an unknown future ahead.
But that was quite impossible, because timing constraints are usually written in SDC, and SDC was invented by Synopsys, and Vivado are the closest that I know of to the full and accurate interpretation of the constraints, as well as the Tcl environment. That was yet another long sentence that I would never write for translation. But I actually expect that all tools will end up supporting Synopsys’ full vocabulary, so maybe a few years from now, these posts will really be vendor-neutral, and the only thing people won’t understand is what Vivado and Xilinx is, because some other FPGA vendor that we’ve never heard of has taken control of the market, while “AMD FPGA” put all they had on datacenter acceleration and ended up closing the FPGA department because, well, because.
Did I diverge again? Definitely. All I wanted to say is that I also wrote a page about IOB registers, and needless to say, that page is available in three additional languages, just like the rest.
Phew
I really wonder if anyone will read through this post from top to bottom. If you haven’t, and just scrolled down to the bottom to see if this goes anywhere, I can’t blame you. Truth to be said, this post went nowhere, and neither was it intended to do so.
Any guesses about the reason I wrote this anyhow. I still promise to send a pizza emoji in response to correct answers. Even though all answers are incorrect by the nature of this foolishness.
I’ve been a bit silent on this blog for a while, but that’s only because I’ve been busy writing on my spin-off site lately.
So here are a few posts over there which are pretty much related to what I do on this blog. These pages are also translated to Chinese, Japanese and Korean, so this is what those links in parentheses are about.
About FIFOs:
About clock domains:
Introduction
There is a widespread belief, that in order to use git send-email with Gmail, there’s a need to subscribe to Google Cloud services and obtain some credentials. Or that a two-factor authentication (2fa) is required.
This is not the case, however. If Thunderbird can manage to fetch and send emails through Google’s mail servers (as well as other OAUTH2 authenticated mail services), there’s no reason why a utility won’t be able to do the same.
The subscription to Google’s services is indeed required if the communication with Google’s server must be done without human supervision. That’s the whole point with API keys. If a human is around when the mail is dispatched, there’s no need for any special measures. And it’s quite obvious that there’s a responsive human around when a patch is being submitted.
What is actually needed, is a client ID and a client secret, and these are indeed obtained by registering to Google’s cloud service (this explains how). But here’s the thing: Someone at Mozilla has already obtained these, and hardcoded them into Thunderbird itself. So there’s no problem using these to access Gmail with another mail client. It seems like many believe that the client ID and secret must be related to the mail account to access, and therefore each and every one has to obtain their own pair. That’s a mistake that has made a lot of people angry for nothing.
This post describes how to use git send-email without any further involvement with Google, except for having a Gmail account. The same method surely applies for other mail service providers that rely on OAUTH2, but I haven’t gotten into that. It should be quite easy to apply the same idea to other services as well however.
For this to work, Thunderbird must be configured to access the same email account. This doesn’t mean that you actually have to use Thunderbird for your mail exchange. It’s actually enough to configure the Gmail server as an outgoing mail server for the relevant account. In other words, you don’t even need to fetch mails from the server with Thunderbird.
The point is to make Thunderbird set up the OAUTH2 session, and then fetch the relevant piece of credentials from it. And take it from there with Google’s servers. Thunderbird is a good candidate for taking care of the session’s setup, because the whole idea with OAUTH2 is that the user / password session (plus possible additional authentication challenges) is done with a browser. Since Thunderbird is Firefox in disguise, it integrates the browser session well into its general flow.
If you want to use another piece of software to maintain the OAUTH2 session, that’s most likely possible, given that you can get its refresh token. This will also require obtaining its client ID and client secret. Odds are that it can be found somewhere in that software’s sources, exactly as I found it for Thunderbird. Or look at the https connection it runs to get an access token (which isn’t all that easy, encryption and that).
Outline of solution
All below relates to Linux Mint 19, Thunderbird 91.10.0, git version 2.17.1, Perl 5.26 and msmtp 1.8.14. But except for Thunderbird and msmtp, I don’t think the versions are going to matter.
It’s highly recommended to read through my blog post on OAUTH2, in particular the section called “The authentication handshake in a nutshell”. You’re going to need to know the difference between an access token and a refresh token sooner or later.
So the first obstacle is the fact that git send-email relies on the system’s sendmail to send out the emails. That utility doesn’t support OAUTH2 at the time of writing this. So instead, I used msmtp, which is a drop-in replacement for sendmail, plus it supports OAUTH2 (since version 1.8.13).
msmtp identifies itself to the server by sending it an access token in the SMTP session (see a dump of a sample session below). This access token is short-lived (3600 seconds from Google as of writing this), so it can’t be fetched from Thunderbird just like that. In particular because most of the time Thunderbird doesn’t have it.
What Thunderbird does have is a refresh token. It’s a completely automatic task to ask Google’s server for the access token with the refresh token at hand. It’s also an easy task (once you’ve figured out how to do it, that is). It’s also easy to get the refresh token from Thunderbird, exactly in the same way as getting a saved password. In fact, Thunderbird treats the refresh token as a password.
msmtp allows executing an arbitrary program in order to get the password or the access token. So I wrote a Perl script (oauth2-helper.pl) that reads the refresh token from a file and gets an access token from Google’s server. This is how msmtp manages to authenticate itself.
So everything relies on this refresh token. In principle, it can change every time it’s used. In practice, as of today, Google’s servers don’t change it. It seems like the refresh token is automatically replaced every six months, but even if that’s true today, it may change.
But that doesn’t matter so much. All that is necessary is that the refresh token is correct once. If the refresh token goes out of sync with Google’s server, a simple user / password session rectifies this. And as of now, than virtually never happens.
So let’s get to the hands-on part.
Install msmtp
Odds are that your distribution offers msmtp, so it can be installed with something like
# apt install msmtp
Note however that the version needs to be at least 1.8.13, which wasn’t my case (Linux Mint 19). So I installed it from the sources. To do that, first install the TLS library, if it’s not installed already (as root):
# apt install gnutls-dev
Then clone the git repository, compile and install:
$ GIT_SSL_NO_VERIFY=true git clone http://git.marlam.de/git/msmtp.git
$ cd msmtp
$ git checkout msmtp-1.8.14
$ autoreconf -i
$ ./configure
$ make && echo Success
$ sudo make install
The installation goes to /usr/local/bin and other /usr/local/ paths, as one would expect.
I checked out version 1.8.14 because later versions failed to compile on my Linux Mint 19. OAUTH2 support was added in 1.8.13, and judging by the commit messages it hasn’t been changed since, except for commit 1f3f4bfd098, which is “Send XOAUTH2 in two lines, required by Microsoft servers”. Possibly cherry-pick this commit (I didn’t).
Once everything has been set up as described below, it’s possible to send an email with
$ msmtp -v -t < ~/email.eml
The -v flag is used only for debugging, and it prints out the entire SMTP session.
The -t flag tells msmtp to fetch the recipients from the mail’s own headers. Otherwise, the recipients need to be listed in the command line, just like sendmail. Without this flag or recipients, msmtp just replies with
msmtp: no recipients found
The -t flag isn’t necessary with git send-email, because it explicitly lists the recipients in the command line.
The oauth2-helper.pl script
As mentioned above, Thunderbird has the refresh token, but msmtp needs an access token. So the script that talks with Google’s server and grabs the access token can be downloaded from its Github repo. Save it, with execution permission to /usr/local/bin/oauth2-helper.pl (or whatever, but this is what I assume in the configurations below).
Some Perl libraries may be required to run this script. On a Debian-based system, the packages’ names are probably something like libhttp-message-perl, libwww-perl and libjson-perl.
It’s written to access Google’s token server, but can be modified easily to access a different service provider by changing the parameters at its beginning. For other email providers, check if it happens to be listed in OAuth2Providers.jsm. I don’t know how well it will work with those other providers, though.
The script reads the refresh token from ~/.oauth2_reftoken as a plain file containing the blob only. There’s an inherent security risk of having this token stored like this, but it’s basically the same risk as the fact that it can be obtained from Thunderbird’s credential files. The difference is the amount of security by obscurity. Anyhow, the reference token isn’t your password, and it can’t be derived from it. Either way, make sure that this file has a 0600 or 0400 permission, if you’re running on a multi-user computer.
The script caches the access token in ~/.oauth2_acctoken, with an expiration timestamp. As of today, it means that the script talks with the Google’s server once in 60 minutes at most.
Setting up config files
So with msmtp installed and the script downloaded into /usr/local/bin/oauth2-helper.pl, all that is left is configuration files.
First, create ~/.msmtprc as follows (put your Gmail username instead of mail.username, of course):
account default
host smtp.gmail.com
port 587
tls on
tls_starttls on
auth xoauth2
user mail.username
passwordeval /usr/local/bin/oauth2-helper.pl
from mail.username@gmail.com
And then change the [sendemail] section in ~/.gitconfig to
[sendemail]
smtpServer = /usr/local/bin/msmtp
That’s it. Only that single line. It’s however possible to use smtpServerOption in the .gitconfig to add various flags. So for example, to get the entire SMTP session shown while sending the email, it should say:
[sendemail]
smtpServer = /usr/local/bin/msmtp
smtpServerOption = -v
But really, don’t, unless there’s a problem sending mails.
Other than that, don’t keep old settings. For example, there should not be a “from=” entry in .gitconfig. Having such causes a “From:” header to be added into the mail body (so it’s visible to the reader of the mail). This header is created when there is a difference between the “From” that is generated by git send-email (which is taken from the “from=” entry) and the patch’ author, as it appears in the patch’ “From” header. The purpose of this in-body header is to tell “git am” who the real author is (i.e. not the sender of the patch). So this extra header won’t appear in the commit, but it nevertheless makes the sender of the message look somewhat clueless.
So in short, no old junk.
Sending a patch
Unless it’s the first time, I suggest just trying to send the patch to your own email address, and see if it works. There’s a good chance that the refresh token from the previous time will still be good, so it will just work, and no point hassling more.
Actually, it’s fine to try like this even on the first time, because the Perl script will fail to grab the access token and then tell you what to do to fix it, namely:
- Make sure that Thunderbird has access to the mail account itself, possibly by attempting to send an email through Gmail’s server.
- Go to Thunderbird’s Preferences > Privacy & Security and click on Saved Passwords. Look for the account, where the Provider start with oauth://. Right-click that line and choose “Copy Password”.
- Create or open ~/.oauth2_reftoken, and paste the blob into that file, so it contains only that string. No need to be uptight with newlines and whitespaces: They are ignored.
And then go, as usual:
$ git send-email --to 'my@test.mail' 0001-my.patch
I’ve added the output of a successful session (with the -v flag) below.
Room for improvements
It would have been nicer to fetch the refresh token automatically from Thunderbird’s credentials store (that is from logins.json, based upon the decryption key that is kept in key4.db), but the available scripts for that are written in Python. And to me Python is equal to “will cause trouble sooner or later”. Anyhow, this tutorial describes the mechanism (in the part about Firefox).
Besides, it could have been even nicer if the script was completely standalone, and didn’t depend on Thunderbird at all. That requires doing the whole dance with the browser, something I have no motivation to get into.
A successful session
This is what it looks like when a patch is properly sent, with the smtpServerOption = -v line in .gitignore (so msmtp produces verbose output):
Send this email? ([y]es|[n]o|[q]uit|[a]ll): y
ignoring system configuration file /usr/local/etc/msmtprc: No such file or directory
loaded user configuration file /home/eli/.msmtprc
falling back to default account
Fetching access token based upon refresh token in /home/eli/.oauth2_reftoken...
using account default from /home/eli/.msmtprc
host = smtp.gmail.com
port = 587
source ip = (not set)
proxy host = (not set)
proxy port = 0
socket = (not set)
timeout = off
protocol = smtp
domain = localhost
auth = XOAUTH2
user = mail.username
password = *
passwordeval = /usr/local/bin/oauth2-helper.pl
ntlmdomain = (not set)
tls = on
tls_starttls = on
tls_trust_file = system
tls_crl_file = (not set)
tls_fingerprint = (not set)
tls_key_file = (not set)
tls_cert_file = (not set)
tls_certcheck = on
tls_min_dh_prime_bits = (not set)
tls_priorities = (not set)
tls_host_override = (not set)
auto_from = off
maildomain = (not set)
from = mail.username@gmail.com
set_from_header = auto
set_date_header = auto
remove_bcc_headers = on
undisclosed_recipients = off
dsn_notify = (not set)
dsn_return = (not set)
logfile = (not set)
logfile_time_format = (not set)
syslog = (not set)
aliases = (not set)
reading recipients from the command line
<-- 220 smtp.gmail.com ESMTP m8-20020a7bcb88000000b003c6d21a19a0sm3316430wmi.29 - gsmtp
--> EHLO localhost
<-- 250-smtp.gmail.com at your service, [109.186.183.118]
<-- 250-SIZE 35882577
<-- 250-8BITMIME
<-- 250-STARTTLS
<-- 250-ENHANCEDSTATUSCODES
<-- 250-PIPELINING
<-- 250-CHUNKING
<-- 250 SMTPUTF8
--> STARTTLS
<-- 220 2.0.0 Ready to start TLS
TLS session parameters:
(TLS1.2)-(ECDHE-ECDSA-SECP256R1)-(CHACHA20-POLY1305)
TLS certificate information:
Subject:
CN=smtp.gmail.com
Issuer:
C=US,O=Google Trust Services LLC,CN=GTS CA 1C3
Validity:
Activation time: Mon 26 Sep 2022 11:22:04 AM IDT
Expiration time: Mon 19 Dec 2022 10:22:03 AM IST
Fingerprints:
SHA256: 53:F3:CA:1D:37:F2:1F:ED:2C:67:40:A2:A2:29:C2:C8:E8:AF:9E:60:7A:01:92:EC:F0:2A:11:E8:37:A5:88:F3
SHA1 (deprecated): D4:69:6E:59:2D:75:43:59:02:74:25:67:E7:57:40:E0:28:43:A8:62
--> EHLO localhost
<-- 250-smtp.gmail.com at your service, [109.186.183.118]
<-- 250-SIZE 35882577
<-- 250-8BITMIME
<-- 250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH
<-- 250-ENHANCEDSTATUSCODES
<-- 250-PIPELINING
<-- 250-CHUNKING
<-- 250 SMTPUTF8
--> AUTH XOAUTH2 dXNlcj1lbGkuYmlsbGF1ZXIBYXV0aD1CZWFyZXIgeWEyOS5hMEFhNHhyWE1GM1gtOTJMVWNidjE4MFdVOBROENRcUdSbk5KaUFSY0VSckVaXzdzbDlHMTNpdFIyUTk0NjlKWG45aHVGLQVRBU0FSTVXJpSjRqMjBLcWh6WU9GekxlcU5BYVpFNUU4WXRhNjdLUXpCRm1HRDg3dFgzeHJ4amNPTnRVTkZFVWdESXhsUlcxOFhVT0pqQ1hPSlFwZlNGUUVqRHZMOWw4RExkTjlKZlNbGRTazNNbFNMNjVfQWFDZ1lLVVF2Y0luOWNSSUEwMTY2AQE=
<-- 235 2.7.0 Accepted
--> MAIL FROM:<mail.username@gmail.com>
--> RCPT TO:<test@mail.com>
--> RCPT TO:<mail.username@gmail.com>
--> DATA
<-- 250 2.1.0 OK m8-20020a7bcb88000000b003c6d21a19a0sm3316430wmi.29 - gsmtp
<-- 250 2.1.5 OK m8-20020a7bcb88000000b003c6d21a19a0sm3316430wmi.29 - gsmtp
<-- 250 2.1.5 OK m8-20020a7bcb88000000b003c6d21a19a0sm3316430wmi.29 - gsmtp
<-- 354 Go ahead m8-20020a7bcb88000000b003c6d21a19a0sm3316430wmi.29 - gsmtp
--> From: Eli Billauer <mail.username@gmail.com>
--> To: test@mail.com
--> Cc: Eli Billauer <mail.username@gmail.com>
--> Subject: [PATCH v8] Gosh! Why don't you apply this patch already!
--> Date: Sun, 30 Oct 2022 07:01:14 +0200
--> Message-Id: <20221030050114.49299-1-mail.username@gmail.com>
--> X-Mailer: git-send-email 2.17.1
-->
[ ... email body comes here ... ]
--> --
--> 2.17.1
-->
--> .
<-- 250 2.0.0 OK 1667106108 m8-20020a7bcb88000000b003c6d21a19a0sm3316430wmi.29 - gsmtp
--> QUIT
<-- 221 2.0.0 closing connection m8-20020a7bcb88000000b003c6d21a19a0sm3316430wmi.29 - gsmtp
OK. Log says:
Sendmail: /usr/local/bin/msmtp -v -i test@mail.com mail.username@gmail.com
From: Eli Billauer <mail.username@gmail.com>
To: test@mail.com
Cc: Eli Billauer <mail.username@gmail.com>
Subject: [PATCH v8] Gosh! Why don't you apply this patch already!
Date: Sun, 30 Oct 2022 07:01:14 +0200
Message-Id: <20221030050114.49299-1-mail.username@gmail.com>
X-Mailer: git-send-email 2.17.1
Result: OK
Ah, and the fact that the access token can be copied from here is of course meaningless, as it has expired long ago.
Thunderbird debug notes
These are some random notes I made while digging in Thunderbird’s guts to find out what’s going on.
So this is Thunderbird’s official git repo. Not that I used it.
To get logging info from Thunderbird: Based upon this page, go to Thunderbird’s preferences > General and click the Config Editor button. Set mailnews.oauth.loglevel to All (was Warn). Same with mailnews.smtp.loglevel. Then open the Error Console with Ctrl+Shift+J.
The cute thing about these logs is that the access code is written in the log. So it’s possible to skip the Perl script, and use the access code from Thunderbird’s log. Really inconvenient, but possible.
The OAuth2 token requests is implemented in Oauth2.jsm. It’s possible to make a breakpoint in this module by through Tools > Developer Tools > Developer Toolbox, and once it opens (after requesting permission for external connection), go to the debugger.
Find Oauth2.jsm in the sources pane to the left (of the Debugger tab), under resource:// modules > sessionstore. Add a breakpoint in requestAccessToken() so that the clientID and consumerSecret properties can be revealed.
Sending a patch from Thunderbird directly
This is a really bad idea. But if you have Thunderbird, and need to send a patch right now, this is a quick, dirty and somewhat dangerous procedure for doing that.
Why is it dangerous? Because at some point, it’s easy to pick “Send now” instead of “Send later”, and boom, a junk patch is mailed to the whole world.
The problem with Thunderbird is that it makes small changes into the patch’ body. So to work around this, there’s a really silly procedure. I used it once, and I’m not proud of that.
So here we go.
First, a very simple script that outputs the patch mail into a file. Say that I called it dumpit (should be executable, of course):
#!/bin/bash
cat > /home/eli/Desktop/git-send-email.eml
Then change ~/.gitconfig, so it reads something like this in the [sendemail] section:
[sendemail]
from = mail.username@gmail.com
smtpServer = /home/eli/Desktop/dumpit
So basically it uses the silly script as a mail server, and the content goes out to a plain file.
Then run git send-email as usual. The result is a git-send-email.eml as a file.
And now comes the part of making Thunderbird send it.
- Close Thunderbird. All windows.
- Change directory to where Thunderbird keeps its profile files, to under Mail/Local Folders
- Remove “Unsent Messages” and “Unsent Messages.msf”
- Open Thunderbird again
- Inside Thunderbird, go to Hamburger Icon > File > Open > Saved Message… and select git-send-email.eml. The email message should appear.
- Right-Click somewhere in the message’s body, and pick Edit as New Message…
- Don’t send this message as is! It’s completely messed up. In particular, there are some indentations in the patch itself, which renders it useless.
- Instead, pick File > Send Later.
- Once again, close Thunderbird. All windows.
- Remove “Unsent Messages.msf” (only)
- Edit “Unsent Messages” as follows: Everything under the “Content-Transfer-Encoding: 7bit” part is the mail’s body. So remove the “From:” line after it, and paste the email’s body from git-send-email.eml instead.
- Note that there are normally two blank lines after the mail’s body. Retain them.
- Open Thunderbird again. Verify that those indentations are away.
- Look at the mail inside Outbox, and verify that it’s OK now. These are the three things to look for in particular:
- The “From:” part at the beginning of the message is gone.
- At the end of the message, there’s a “–” and git’s version number. These should be in separate lines.
- Look at the mail’s source. The “+” and “-” signs of the diffs must not be indented.
- If all is fine, right-click Outbox, and pick “Send unsent messages”. And hope for good.
Are you sure you want to do this?
This is a failure
This post is a messy collection of things I wrote down as I tried to make a simple Java command-line (as in adb shell) utility for making a change in the device’s config settings. It’s one of several attempts I’ve made to stop the automatic hibernation of unused app, as discussion on this other post of mine.
Except for a lot of conclusions about what is impossible, this post shows how to compile and run a plain Java program that can access Android’s API functions. But be warned: This is most likely not very useful, because the better part of that API can’t be used. More on this below.
So I did manage to get the program running. It’s just that I learned that it’s useless.
So if you’re reading this because you want to do something quick and dirty utility for Android, my message is: Don’t. Fetch the source code of some boilerplate app somewhere, and take it from there.
Let’s be honest: Stopping the hibernation is by itself not reason enough for all the efforts I’ve put in, but it’s a good way to get acquainted with Android in a way that will hopefully help me to solve issues in the future.
And this is a good place to mention that neither Java nor Android is not what I do for living, so no wonder I get into nonsense.
I am however well-versed with Linux machines in general, which is why I’m inclined towards plain command-line programs. Which turns out to be a bad idea with Android, as evident from below.
The strategies
As mentioned in that post, the device’s config settings are more than apparently stored in an XML file, which isn’t really an XML file, but rather a file in ABX format, which is some special Android thing with currently no reliable toolset for manipulation.
Android’s own “settings” command-line utility allows manipulation of these settings, but specifically not the “config” namespace. So if the built-in utility won’t do that for me, how about writing a little thingy of my own?
Strategy #1 was to use the API methods of android.provider.Settings to change the setting values directly. But alas, the methods that manipulate the “config” namespace are not exposed to the API level. So they are accessible only from within Android’s core classes. Once again, mr. Android tells me not touch “config”.
Strategy #2 was to follow the SettingService class. What this service actually does, is to do the work as requested by the “settings” command-line utility. More on that below. Anyhow, the trick is that the actual access to the settings database (XML / ABX files, actually) is through the SettingsProvider class, which extends ContentProvider.
OK, so a quick word on Android Content Providers. This is an Android-specific IPC API that allows Android components (including apps) to expose information to just other components, with a noSQL-like set of methods. So the Content Provider is expected to have some kind of database with row/column tables, just like good old SQL. The most common use for this is the contact list (as in a phonebook).
The side that wants to access the data calls getContentResolver() with a URI that represents the requested database, and the ContentResolver object that is returned has methods like insert(), query(), update(), and most interestingly, call(). So by using the methods of the ContentResolver object, the requests are passed on the Content Provider on the other side, which fulfills the request and conveys the result.
So I thought I should connect with the SettingsProvider as a Content Provider, and use the same calls as in SettingService. The URI of this Content Provider isn’t given in SettingService, because it’s apparently given to the object constructor directly. A wild guess, based upon what I found in the dumpsys output, is “content://com.android.providers.settings/.SettingsProvider”. Based upon the Settings.java it’s more like just “content://settings”. Never got to try this however.
The reason I never got that far is that getContentResolver() is a method of the Context class. And that’s not a small technicality. Because a command line program of the sort I was playing around with doesn’t have a context. That’s something that apps and services have. There’s no way to create one ad-hoc, as far as I know. In the absence of a Context object, there’s no way to call getContentResolver(), and that means that there’s no access to the Content Provider. Game over.
This obstacle is probably not a coincidence either: In order to access data, the system wants to know who you are. A plain Java command-line utility has no such identity. In the context of the “config” class, it’s even more evident: Setting’s plain API doesn’t expose it, and the Content Provider API wants to know who fiddles with these variables.
The conclusion is hence that odds are that any valuable and interesting functionality will be impossible from a simple command-line utility. For this to have a chance to work, there must be an app or a service, which is what I wanted to avoid.
How does “settings” run
The “settings” command is in fact /system/bin/settings which consists of the following:
#!/system/bin/sh
cmd settings "$@"
So what is “cmd”? Let’s just try:
$ adb shell cmd
cmd: No service specified; use -l to list all running services. Use -w to start and wait for a service.
And using “cmd -l” indeed enlists “settings” as a service.
Same goes for “pm”, but with the “package” service instead.
So these important utilities don’t perform the action themselves. A bit like Linux’ systemd commands, they talk with a service instead. The rationale is probably the same: Not to expose the API that can touch the sensitive stuff, so only a service can do that.
Not directly related, but dumpsys is /system/bin/dumpsys, which is an ELF:
dumpsys: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /system/, BuildID[md5/uuid]=eb94567e9085216f43594a212afd2160, stripped
Installing the minimal SDK for compilation
Downloaded commandlinetools-linux-8512546_latest.zip from Android Studio’s download page. Android Studio can be downloaded on the same page, but I’m a little fan of IDEs, and what I want to do is small and non-standard (Gradle is often mentioned in the context of building from command line, but that’s for full-blown apps).
Unzip the file, and move the lib and bin directories into the following directory structure:
├── sdk
│ └── cmdline-tools
│ └── latest
│ ├── bin
│ └── lib
Or sdkmanager will complain that
Error: Could not determine SDK root.
Error: Either specify it explicitly with --sdk_root= or move this package into its expected location: <sdk>/cmdline-tools/latest/
A little silly obstacle to prepare you for what’s ahead.
Change directory to sdk/cmdline-tools/latest and get a list of installable packages:
$ bin/sdkmanager --list
And then, to download the build tools, go something like:
$ bin/sdkmanager 'build-tools;28.0.0'
Note that the exact version is specified (they were all listed before) and that the single-quote is required because of the semicolon. Expect a long download.
Remember that old build tools are better if you don’t need the features of the new ones. Generally speaking, old tools build things that are compatible to a wider range of targets.
Congrats, the sdk/ directory now consumes 1.1 GB, most of which is under emulator/ (just delete that directory).
dx is in build-tools/28.0.0/.
Next up, obtain android.jar, which contains the API stubs. This is needed by javac, or else it refuses to import Android libraries. First, figure out which API level I need. My phone runs Android 12, so I’ll go for API level 31, according to the list at the top of this page.
Hence:
$ bin/sdkmanager 'platforms;android-31'
The SDK just grew to 1.3 GB. Nice. The important part is platforms/android-31/android.jar, and it’s there.
Like any jar file, it’s possible to peek inside with
$ jar xvf android.jar
and that allows seeing what is exposed in each class by using e.g. to see what android.provider.Settings imports:
javap android/provider/Settings.class
Note that classes defined inside classes are put in separate .class files, e.g. Settings$System.class. The first line in javap’s output says which .java file the .class file was compiled from.
Compile command line utility for Android
You know that this is probably useless, right? You’ve read the introduction above?
So let’s take this simple program as hello.java:
import java.io.*;
public class hello {
public static void main(String[] args) {
PrintStream p = java.lang.System.out;
p.println("Hello, world");
}
}
The only thing worth mentioning is that @p is used instead of java.lang.System.out. Just System.out is ambiguous, so it has to be this long.
Compile with
./compile.sh
where compile.sh is
#!/bin/bash
set -e
TARGET=hello
SDK=/path/to/sdk
DX=$SDK/build-tools/28.0.0/dx
JAR=$SDK/platforms/android-31/android.jar
rm -f $TARGET.class $TARGET.dex
javac -source 1.7 -target 1.7 -bootclasspath $JAR -source 1.7 -target 1.7 hello.java $TARGET.java
$DX --dex --output=$TARGET.dex $TARGET.class
The -source 1.7 -target 1.7 thing in javac’s arguments is to create Java 7 bytecode. Which is the reason for this warning, which can be ignored safely:
warning: [options] bootstrap class path not set in conjunction with -source 7
This is a good time to mention that Java is deprecated, and Android embraces Kotlin. As if they didn’t learn the lesson with those .NET languages.
If everything goes fine, send it to the phone (delme-tmp should be created first, of course):
$ adb push hello.dex /storage/emulated/0/delme-tmp/
Execution: Change directory to /storage/emulated/0/delme-tmp and go
$ dalvikvm -Djava.class.path=./hello.dex hello
Hello, world
Yey. It works. But that’s hardly worth anything, is it? How about using some Android API? So let’s change hello.java to this. The most basic thing I could think about (and in fact, I failed to come up with anything to do that didn’t somehow require a context):
import java.io.*;
import android.os.Build;
public class hello {
public static void main(String[] args) {
PrintStream p = java.lang.System.out;
String manufacturer = Build.MANUFACTURER;
p.println("Manufacturer is " + manufacturer);
}
}
But that gave
$ dalvikvm -Djava.class.path=./hello.dex hello
Exception in thread "main" java.lang.UnsatisfiedLinkError: No implementation found for java.lang.String android.os.SystemProperties.native_get(java.lang.String, java.lang.String) (tried Java_android_os_SystemProperties_native_1get and Java_android_os_SystemProperties_native_1get__Ljava_lang_String_2Ljava_lang_String_2)
at android.os.SystemProperties.native_get(Native Method)
at android.os.SystemProperties.get(SystemProperties.java:165)
at android.os.Build.getString(Build.java:1434)
at android.os.Build.<clinit>(Build.java:54)
at hello.main(hello.java:8)
So compiling against the libraries is fine, but actually getting it to do something is a different story. There is probably a solution for this, but given the prospects of getting something useful done with this method, I didn’t bother going any further.
Sources on compilation:
- See the last answer on this page on how to compile something that runs with DalvikVM.
- See this page on how to compile a command-line program, albeit with a shell wrapper for “app_process”. This page also mentions that both dalvikvm as well as app_process lack context.
General notes
Most important: There’s Android Code Search for looking up stuff in the code base.
The sources for the Android core functions can be fetched with
git clone https://github.com/aosp-mirror/platform_frameworks_base.git
It’s 4.6 GB, so it takes quite some time to download.
It’s a good idea to check out a tag that corresponds to the same API as in the android.jar you’re working with. For the sake of seeing the actual implementation of the API functions, as well as how they’re really used, when the API was in that stage. I haven’t found a clear-cut way to connect between the API level and a tag in the git repo, so I went by the dates of the commits.
- The root of the core classes is at platform_frameworks_base/core/java
- There’s a direct connection between the class name and the directory it resides under.
- The package directive at the beginning of Java files are just a way to allow wildcards for importing several classes at once.
- Command-line utilities tend to extend the ShellCommand class. Looking for this is a good way to find the utilities and test programs in the source code.
- However a common way to implement a command-line utility is through the “cmd” executable, which invokes the onShellCommand() public method of a service.
Dissection notes
Random pieces of info as I went along (towards failure, that is):
- setSyncDisabledMode() is defined in core/java/android/provider/Settings.java, inside the Config subclass (search for “class Config”) in the file.
- The command-line utility “settings” is implemented in packages/SettingsProvider/src/com/android/providers/settings/SettingsService.java.
- The Settings functionality is implemented in packages/SettingsProvider/src/com/android/providers/settings/SettingsProvider.java. The most interesting part is the list of methods it supports, under the “call” method, which includes constants such as Settings.CALL_METHOD_LIST_GLOBAL (accessed by the “settings” utility) but also Settings.CALL_METHOD_LIST_CONFIG (which isn’t accessed by the same utility).
- The names of the XML files are listed in packages/SettingsProvider/src/com/android/providers/settings/SettingsProvider.java (e.g. SETTINGS_FILE_CONFIG = “settings_config.xml”).
It’s quite apparent that the command-line utility (SettingsService.java) uses the public method calls (e.g. Settings.CALL_METHOD_GET_SYSTEM) and it doesn’t seem like there’s another way to access the registery (i.e. the XML files) because calls a private method for implementing this call (getSystemSetting() in this case).
No solution, yet
This post summarizes my failed attempt to get rid of the said nuisance. Actually, I was successful, but the success was short-lived: The fix that I suggest here gets reverted by the system soon enough.
I expect to get back to this, in particular if I find a way to do either of the following three:
- Manipulate an ABX file (that’s Android’s compressed XML format)
- Permanently disable a service.
- Write a utility that changes the required setting by virtue of Android’s API. A failed attempt to write a command-line utility is described here.
If anyone knows how to do this, beyond anything I’ve written here, please comment below.
Note Gene Cash’ suggestion below in the comments, which is based upon turning off a specific permissions for each app. It’s an interesting direction, however it requires revoking a permission for each new app that we install.
So here comes the long story.
“For your protection”
Yet another “feature” for “my own good”: If an app isn’t used for 90 days, Android 12 automatically revokes the apps permissions, and deletes its temporary data. Which sounds harmless, but if that app involves a registration, well, you’ll have to do that again when you launch it next time. Which can be extremely annoying if you need a taxi quickly because of an issue with your car. And of course you don’t need the taxi app so often…
The common term for this brilliant feature is “removing permissions from unused apps”, but the internal term is “hibernation”.
I guess the people who though about this feature also throw away everything in their homes if they haven’t used them for 90 days.
Anyhow, I like my smartphone repeatable. If I open an app after a long time, I want it to work like it did before. It’s a weird expectation in these days of upgrades every day, I know, but that’s what I want.
It seems like there’s a simple solution. If you have adb access to your phone, that is (or some other type of shell). Or maybe someone will come up with an app that reverts all the silly stuff that new versions of Android come up with. Just be sure to run it every 90 days.
I should mention, that it’s possible to opt out this “feature” for each App individually: Go to each app’s configuration, and opt out “Remove permissions and free up space” under “Unused apps”. But I didn’t find a way to do this globally. Probably because we’re not considered responsible enough to make such a decision.
The short-lived fix
Hibernation of apps is officially documented on this page.
According to that page, the timeout before hibernation is set as a parameter, auto_revoke_unused_threshold_millis2. So
$ adb shell device_config get permissions auto_revoke_unused_threshold_millis2
7776000000
7776000000 milliseconds equals 7776000 seconds, which equals 90 days. Looks like it, doesn’t it?
So the apparent workaround is to make the hibernation time very long, so it never happens. Let’s say 20 times 365 days? Good enough? That’s 630720000000 milliseconds. Yes, this number exceeds 232, but so does the original threshold. It’s a 64-bit machine, after all.
So just go (this doesn’t require rooting the phone):
$ adb shell device_config put permissions auto_revoke_unused_threshold_millis2 630720000000
And then verify that the new number is in effect:
$ adb shell device_config get permissions auto_revoke_unused_threshold_millis2
630720000000
Yey. Does it solve the problem? Nope, that didn’t work. A couple of weeks later I got the same notification, and this time a couple of other apps were hibernated.
Taking a closer look, it turned out that the value of auto_revoke_unused_threshold_millis2 had returned to 7776000000 (after a couple of weeks). How and why, I don’t know. I tried to change it back to the desired value, and ran a reboot, and the updated value survived that. So the values that are set with device_config are permanent. But another reboot later, it was back to 7776000000. So I don’t know what’s going on.
Maybe it’s because I didn’t change the parameter with the same name, but under the app_hibernation namespace?
$ adb shell device_config get app_hibernation auto_revoke_unused_threshold_millis2
7776000000
And maybe the easiest way is to to turn off hibernation altogether. There happens to be a parameter with an interesting name:
$ adb shell device_config get app_hibernation app_hibernation_enabled
true
Googling for this parameter name, I found absolutely nothing. But if it does what its name implies, maybe this will help?
$ adb shell device_config put app_hibernation app_hibernation_enabled false
I can’t even tell if this would help. The parameter got changed back soon enough to “true”.
Other ideas
Truth to be said, I don’t really know why and when the values of these parameters change back to their original values.
But here are a few clues: device_config has a “set_sync_disabled_for_tests” option, so I suppose
adb shell device_config set_sync_disabled_for_tests persistent
could prevent the configuration to reset all the time, but I didn’t try that.
It more than appears that the source of the default values for these parameters are in a permanent setting file, /data/system/users/0/settings_config.xml (needless to say, root access is required to invoke the 0/ directory). There are several similar .xml files in the 0/ directory, as listed in the definition of the class that processes these XML files, SettingsRegistry.
But here’s the crux: Despite the .xml suffix, this isn’t an XML file, but ABX, which is a format that is new in Android 12. It’s a condensed form of XML, which can’t be edited with just a text editor. Since this format is new and specific, there’s currently no reliable toolkit for making pinpoint changes. The only thing I found today (October 2022) is described below, along with its shortcomings.
An entry like this in settings_config.xml:
<setting id="3247" name="app_hibernation/app_hibernation_enabled" package="com.google.android.gms" preserve_in_restore="true" value="true" />
appears with a row like this in the output of “dumpsys settings”.
_id:3247 name:app_hibernation/app_hibernation_enabled pkg:com.google.android.gms value:true
In other words, the ABX file is clearly maintained by the “settings” service (this is what the second argument to dumpsys means). There’s a “settings” command-line utility too, but it doesn’t give access to this attribute (I’m not sure how related it is to the “settings” service).
An alternative approach could be to turn off the Hibernation service itself.
The services can be listed in Settings > System > Developer Options > Running Services, but these are only services that are linked with apps, it seems like. So not there.
On the other hand, the service can be found with dumpsys, and it’s called com.android.server.apphibernation.AppHibernationService. Unfortunately it can’t be disabled with pm disable-user as suggested on this page, be the argument to this command must be a package. The service’s name isn’t recognized by pm for this purpose.
I have no idea how to just disable a service. I’m not even sure it’s possible.
Additional trivia
In case this doesn’t work, here are some random pieces of info that might help. First of all, there’s a service called com.android.server.apphibernation.AppHibernationService. So maybe disable it somehow?
Another thing is that by looking at the output of “adb shell dumpsys package”, it appears like there’s a package called android.intent.action.MANAGE_UNUSED_APPS, which has an Intent named com.google.android.permissioncontroller/com.android.permissioncontroller.permission.ui.ManagePermissionsActivity. Maybe disable it, in the spirit of this post? It’s not clear if that’s the Activity that disables the app, or the one that allows enabling it back.
And then we have this snippet from the (adb shell dumpsys):
logs: "1663488654709 HibernationPolicy:i unused app com.google.android.apps.podcasts - last used on Thu Jan 01 02:00:00 GMT+02:00 1970 "
logs: "1663488654710 HibernationPolicy:i unused app com.google.android.apps.authenticator2 - last used on Thu Jan 01 02:00:00 GMT+02:00 1970 "
logs: "1663488654711 HibernationPolicy:i unused app com.gettaxi.android - last used on Thu Jan 01 02:00:00 GMT+02:00 1970 "
logs: " Globally hibernating apps [com.google.android.apps.podcasts, com.google.android.apps.authenticator2, com.gettaxi.android] "
The sources for the hibernation code:
git clone https://android.googlesource.com/platform/packages/modules/Permission
and the code is under PermissionController/src/com/android/permissioncontroller/. In particular, look at permission/utils/Utils.java for the list of parameters and their internal representation.
The source for the device_config utility is here. The implementation of DeviceConfig’s core methods is here.
Just in case it helps someone (future self in particular).
ABX manipulation
The only reference I found for ABX was in CCL Solution Group’s website. The said post describes the format, and also points at a git repository, which includes a couple of utilities for converting from XML to ABX and back.
The problem with this is that when I tried it on my settings_config.xml file, the back-and-forth conversion didn’t end with exactly the same binary. And I’m not sure if the difference matters. So if I can’t be sure that everything that isn’t related to the little change I want to make stays the same, I prefer not touch it at all.
I suppose there will be better tools in the future (or maybe I’ll write something like that myself, but beware, I will do it in Perl).
So this is how to work with the existing tool:
Downloading the pair of utilities from the git repo:
$ git clone https://github.com/cclgroupltd/android-bits
The last commit in the repository that I refer to below is 7031d0b, made in January 2022.
Change directory to ccl_abx, copy the desired ABX (.xml) file into there, and go
$ python3.6 ccl_abx.py settings_config.xml -mr > opened.xml
(python3.6, as it implies, is Python 3.6 on my machine. There are so many versions, and it didn’t work on earlier ones).
This creates an XML file with the content of the ABX file.
Now, manually edit opened.xml and remove the <root> and </root> tags at the beginning and end. Not clear why they were put there to begin with.
As for back-conversion to ABX, here comes some Java.
ABX manipulation II
This is an update on 27.2.23, following a comment I got to this post: It turns out that there are utilities for ABX conversion on the phone itself, which are wrappers to a Java utility:
$ cat /system/bin/abx2xml
#!/system/bin/sh
export CLASSPATH=/system/framework/abx.jar
exec app_process /system/bin com.android.commands.abx.Abx "$0" "$@"
$ cat /system/bin/xml2abx
#!/system/bin/sh
export CLASSPATH=/system/framework/abx.jar
exec app_process /system/bin com.android.commands.abx.Abx "$0" "$@"
And they work:
$ abx2xml settings_config.xml settings_config.xml.unabx
$ xml2abx settings_config.xml.unabx settings_config.xml.reabx
$ abx2xml settings_config.xml.reabx settings_config.xml.todiff
Unfortunately, settings_config.xml.reabx didn’t end up identical with settings_config.xml. Making a comparison between the outputs of the UNIX “strings” utility, I found a “true3″ string in each setting in the .reabx file, which wasn’t in the original.
Even weirder, the .unabx wasn’t identical to .todiff. But this made them identical (on a regular Linux machine):
$ dos2unix settings_config.xml.unabx
Say what? The ABX was converted into a text file with CR-LF on the first time, and only with LF on the second? I guess the answer to this is somewhere in the Java code.
But maybe it’s OK. I’ll give this a go sometime.
Ugh. Java stuff
Java is definitely not my cup of tea, neither my expertise. So don’t learn anything from me on how to work with Java. The only thing I can say is that it worked.
First, I installed the Java compiler (Linux mint 19):
# apt install default-jdk
Then change directory to makeabx/src.
Compile everything (this will probably make Java-savvy guys laugh, but hey, it worked and created .class files in the respective dirtectories).
$ javac $(find . -iname \*.java)
There will be a few notes about deprecated API, but that’s fine.
Then run the program with
$ java com.ccl.abxmaker.Main path/to/opened.xml
If this fails, saying something like “Could not find or load main class src.com.ccl.abxmaker.Main”, it’s because I tried to run the program from the wrong directory. Even though shell’s autocomplete filled in the class nicely, it’s still wrong. This has to be done from the src/ directory (at least in the way I clumsily compiled this stuff).
This creates a file with the same name and path as the input, but adds an .abx suffix.
The ABX file that I obtained from back-conversion wasn’t identical to the original, however: For example, the first XML element is <settings version=”-1″>. In the original ABX, it’s encoded as a 32-bit integer (0xffffffff), but in the opened.xml.abx it’s encoded as a string (“-1″).
Does it even matter? I don’t know. If the pair of utilities doesn’t pass this kind of regression test, am I ready to inject the result into my phone’s configuration system? Well, no.