This is a failure
This post is a messy collection of things I wrote down as I tried to make a simple Java command-line (as in adb shell) utility for making a change in the device’s config settings. It’s one of several attempts I’ve made to stop the automatic hibernation of unused app, as discussion on this other post of mine.
Except for a lot of conclusions about what is impossible, this post shows how to compile and run a plain Java program that can access Android’s API functions. But be warned: This is most likely not very useful, because the better part of that API can’t be used. More on this below.
So I did manage to get the program running. It’s just that I learned that it’s useless.
So if you’re reading this because you want to do something quick and dirty utility for Android, my message is: Don’t. Fetch the source code of some boilerplate app somewhere, and take it from there.
Let’s be honest: Stopping the hibernation is by itself not reason enough for all the efforts I’ve put in, but it’s a good way to get acquainted with Android in a way that will hopefully help me to solve issues in the future.
And this is a good place to mention that neither Java nor Android is not what I do for living, so no wonder I get into nonsense.
I am however well-versed with Linux machines in general, which is why I’m inclined towards plain command-line programs. Which turns out to be a bad idea with Android, as evident from below.
The strategies
As mentioned in that post, the device’s config settings are more than apparently stored in an XML file, which isn’t really an XML file, but rather a file in ABX format, which is some special Android thing with currently no reliable toolset for manipulation.
Android’s own “settings” command-line utility allows manipulation of these settings, but specifically not the “config” namespace. So if the built-in utility won’t do that for me, how about writing a little thingy of my own?
Strategy #1 was to use the API methods of android.provider.Settings to change the setting values directly. But alas, the methods that manipulate the “config” namespace are not exposed to the API level. So they are accessible only from within Android’s core classes. Once again, mr. Android tells me not touch “config”.
Strategy #2 was to follow the SettingService class. What this service actually does, is to do the work as requested by the “settings” command-line utility. More on that below. Anyhow, the trick is that the actual access to the settings database (XML / ABX files, actually) is through the SettingsProvider class, which extends ContentProvider.
OK, so a quick word on Android Content Providers. This is an Android-specific IPC API that allows Android components (including apps) to expose information to just other components, with a noSQL-like set of methods. So the Content Provider is expected to have some kind of database with row/column tables, just like good old SQL. The most common use for this is the contact list (as in a phonebook).
The side that wants to access the data calls getContentResolver() with a URI that represents the requested database, and the ContentResolver object that is returned has methods like insert(), query(), update(), and most interestingly, call(). So by using the methods of the ContentResolver object, the requests are passed on the Content Provider on the other side, which fulfills the request and conveys the result.
So I thought I should connect with the SettingsProvider as a Content Provider, and use the same calls as in SettingService. The URI of this Content Provider isn’t given in SettingService, because it’s apparently given to the object constructor directly. A wild guess, based upon what I found in the dumpsys output, is “content://com.android.providers.settings/.SettingsProvider”. Based upon the Settings.java it’s more like just “content://settings”. Never got to try this however.
The reason I never got that far is that getContentResolver() is a method of the Context class. And that’s not a small technicality. Because a command line program of the sort I was playing around with doesn’t have a context. That’s something that apps and services have. There’s no way to create one ad-hoc, as far as I know. In the absence of a Context object, there’s no way to call getContentResolver(), and that means that there’s no access to the Content Provider. Game over.
This obstacle is probably not a coincidence either: In order to access data, the system wants to know who you are. A plain Java command-line utility has no such identity. In the context of the “config” class, it’s even more evident: Setting’s plain API doesn’t expose it, and the Content Provider API wants to know who fiddles with these variables.
The conclusion is hence that odds are that any valuable and interesting functionality will be impossible from a simple command-line utility. For this to have a chance to work, there must be an app or a service, which is what I wanted to avoid.
How does “settings” run
The “settings” command is in fact /system/bin/settings which consists of the following:
#!/system/bin/sh
cmd settings "$@"
So what is “cmd”? Let’s just try:
$ adb shell cmd
cmd: No service specified; use -l to list all running services. Use -w to start and wait for a service.
And using “cmd -l” indeed enlists “settings” as a service.
Same goes for “pm”, but with the “package” service instead.
So these important utilities don’t perform the action themselves. A bit like Linux’ systemd commands, they talk with a service instead. The rationale is probably the same: Not to expose the API that can touch the sensitive stuff, so only a service can do that.
Not directly related, but dumpsys is /system/bin/dumpsys, which is an ELF:
dumpsys: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /system/, BuildID[md5/uuid]=eb94567e9085216f43594a212afd2160, stripped
Installing the minimal SDK for compilation
Downloaded commandlinetools-linux-8512546_latest.zip from Android Studio’s download page. Android Studio can be downloaded on the same page, but I’m a little fan of IDEs, and what I want to do is small and non-standard (Gradle is often mentioned in the context of building from command line, but that’s for full-blown apps).
Unzip the file, and move the lib and bin directories into the following directory structure:
├── sdk
│ └── cmdline-tools
│ └── latest
│ ├── bin
│ └── lib
Or sdkmanager will complain that
Error: Could not determine SDK root.
Error: Either specify it explicitly with --sdk_root= or move this package into its expected location: <sdk>/cmdline-tools/latest/
A little silly obstacle to prepare you for what’s ahead.
Change directory to sdk/cmdline-tools/latest and get a list of installable packages:
$ bin/sdkmanager --list
And then, to download the build tools, go something like:
$ bin/sdkmanager 'build-tools;28.0.0'
Note that the exact version is specified (they were all listed before) and that the single-quote is required because of the semicolon. Expect a long download.
Remember that old build tools are better if you don’t need the features of the new ones. Generally speaking, old tools build things that are compatible to a wider range of targets.
Congrats, the sdk/ directory now consumes 1.1 GB, most of which is under emulator/ (just delete that directory).
dx is in build-tools/28.0.0/.
Next up, obtain android.jar, which contains the API stubs. This is needed by javac, or else it refuses to import Android libraries. First, figure out which API level I need. My phone runs Android 12, so I’ll go for API level 31, according to the list at the top of this page.
Hence:
$ bin/sdkmanager 'platforms;android-31'
The SDK just grew to 1.3 GB. Nice. The important part is platforms/android-31/android.jar, and it’s there.
Like any jar file, it’s possible to peek inside with
$ jar xvf android.jar
and that allows seeing what is exposed in each class by using e.g. to see what android.provider.Settings imports:
javap android/provider/Settings.class
Note that classes defined inside classes are put in separate .class files, e.g. Settings$System.class. The first line in javap’s output says which .java file the .class file was compiled from.
Compile command line utility for Android
You know that this is probably useless, right? You’ve read the introduction above?
So let’s take this simple program as hello.java:
import java.io.*;
public class hello {
public static void main(String[] args) {
PrintStream p = java.lang.System.out;
p.println("Hello, world");
}
}
The only thing worth mentioning is that @p is used instead of java.lang.System.out. Just System.out is ambiguous, so it has to be this long.
Compile with
./compile.sh
where compile.sh is
#!/bin/bash
set -e
TARGET=hello
SDK=/path/to/sdk
DX=$SDK/build-tools/28.0.0/dx
JAR=$SDK/platforms/android-31/android.jar
rm -f $TARGET.class $TARGET.dex
javac -source 1.7 -target 1.7 -bootclasspath $JAR -source 1.7 -target 1.7 hello.java $TARGET.java
$DX --dex --output=$TARGET.dex $TARGET.class
The -source 1.7 -target 1.7 thing in javac’s arguments is to create Java 7 bytecode. Which is the reason for this warning, which can be ignored safely:
warning: [options] bootstrap class path not set in conjunction with -source 7
This is a good time to mention that Java is deprecated, and Android embraces Kotlin. As if they didn’t learn the lesson with those .NET languages.
If everything goes fine, send it to the phone (delme-tmp should be created first, of course):
$ adb push hello.dex /storage/emulated/0/delme-tmp/
Execution: Change directory to /storage/emulated/0/delme-tmp and go
$ dalvikvm -Djava.class.path=./hello.dex hello
Hello, world
Yey. It works. But that’s hardly worth anything, is it? How about using some Android API? So let’s change hello.java to this. The most basic thing I could think about (and in fact, I failed to come up with anything to do that didn’t somehow require a context):
import java.io.*;
import android.os.Build;
public class hello {
public static void main(String[] args) {
PrintStream p = java.lang.System.out;
String manufacturer = Build.MANUFACTURER;
p.println("Manufacturer is " + manufacturer);
}
}
But that gave
$ dalvikvm -Djava.class.path=./hello.dex hello
Exception in thread "main" java.lang.UnsatisfiedLinkError: No implementation found for java.lang.String android.os.SystemProperties.native_get(java.lang.String, java.lang.String) (tried Java_android_os_SystemProperties_native_1get and Java_android_os_SystemProperties_native_1get__Ljava_lang_String_2Ljava_lang_String_2)
at android.os.SystemProperties.native_get(Native Method)
at android.os.SystemProperties.get(SystemProperties.java:165)
at android.os.Build.getString(Build.java:1434)
at android.os.Build.<clinit>(Build.java:54)
at hello.main(hello.java:8)
So compiling against the libraries is fine, but actually getting it to do something is a different story. There is probably a solution for this, but given the prospects of getting something useful done with this method, I didn’t bother going any further.
Sources on compilation:
- See the last answer on this page on how to compile something that runs with DalvikVM.
- See this page on how to compile a command-line program, albeit with a shell wrapper for “app_process”. This page also mentions that both dalvikvm as well as app_process lack context.
General notes
Most important: There’s Android Code Search for looking up stuff in the code base.
The sources for the Android core functions can be fetched with
git clone https://github.com/aosp-mirror/platform_frameworks_base.git
It’s 4.6 GB, so it takes quite some time to download.
It’s a good idea to check out a tag that corresponds to the same API as in the android.jar you’re working with. For the sake of seeing the actual implementation of the API functions, as well as how they’re really used, when the API was in that stage. I haven’t found a clear-cut way to connect between the API level and a tag in the git repo, so I went by the dates of the commits.
- The root of the core classes is at platform_frameworks_base/core/java
- There’s a direct connection between the class name and the directory it resides under.
- The package directive at the beginning of Java files are just a way to allow wildcards for importing several classes at once.
- Command-line utilities tend to extend the ShellCommand class. Looking for this is a good way to find the utilities and test programs in the source code.
- However a common way to implement a command-line utility is through the “cmd” executable, which invokes the onShellCommand() public method of a service.
Dissection notes
Random pieces of info as I went along (towards failure, that is):
- setSyncDisabledMode() is defined in core/java/android/provider/Settings.java, inside the Config subclass (search for “class Config”) in the file.
- The command-line utility “settings” is implemented in packages/SettingsProvider/src/com/android/providers/settings/SettingsService.java.
- The Settings functionality is implemented in packages/SettingsProvider/src/com/android/providers/settings/SettingsProvider.java. The most interesting part is the list of methods it supports, under the “call” method, which includes constants such as Settings.CALL_METHOD_LIST_GLOBAL (accessed by the “settings” utility) but also Settings.CALL_METHOD_LIST_CONFIG (which isn’t accessed by the same utility).
- The names of the XML files are listed in packages/SettingsProvider/src/com/android/providers/settings/SettingsProvider.java (e.g. SETTINGS_FILE_CONFIG = “settings_config.xml”).
It’s quite apparent that the command-line utility (SettingsService.java) uses the public method calls (e.g. Settings.CALL_METHOD_GET_SYSTEM) and it doesn’t seem like there’s another way to access the registery (i.e. the XML files) because calls a private method for implementing this call (getSystemSetting() in this case).
No solution, yet
This post summarizes my failed attempt to get rid of the said nuisance. Actually, I was successful, but the success was short-lived: The fix that I suggest here gets reverted by the system soon enough.
I expect to get back to this, in particular if I find a way to do either of the following three:
- Manipulate an ABX file (that’s Android’s compressed XML format)
- Permanently disable a service.
- Write a utility that changes the required setting by virtue of Android’s API. A failed attempt to write a command-line utility is described here.
If anyone knows how to do this, beyond anything I’ve written here, please comment below.
Note Gene Cash’ suggestion below in the comments, which is based upon turning off a specific permissions for each app. It’s an interesting direction, however it requires revoking a permission for each new app that we install.
So here comes the long story.
“For your protection”
Yet another “feature” for “my own good”: If an app isn’t used for 90 days, Android 12 automatically revokes the apps permissions, and deletes its temporary data. Which sounds harmless, but if that app involves a registration, well, you’ll have to do that again when you launch it next time. Which can be extremely annoying if you need a taxi quickly because of an issue with your car. And of course you don’t need the taxi app so often…
The common term for this brilliant feature is “removing permissions from unused apps”, but the internal term is “hibernation”.
I guess the people who though about this feature also throw away everything in their homes if they haven’t used them for 90 days.
Anyhow, I like my smartphone repeatable. If I open an app after a long time, I want it to work like it did before. It’s a weird expectation in these days of upgrades every day, I know, but that’s what I want.
It seems like there’s a simple solution. If you have adb access to your phone, that is (or some other type of shell). Or maybe someone will come up with an app that reverts all the silly stuff that new versions of Android come up with. Just be sure to run it every 90 days.
I should mention, that it’s possible to opt out this “feature” for each App individually: Go to each app’s configuration, and opt out “Remove permissions and free up space” under “Unused apps”. But I didn’t find a way to do this globally. Probably because we’re not considered responsible enough to make such a decision.
The short-lived fix
Hibernation of apps is officially documented on this page.
According to that page, the timeout before hibernation is set as a parameter, auto_revoke_unused_threshold_millis2. So
$ adb shell device_config get permissions auto_revoke_unused_threshold_millis2
7776000000
7776000000 milliseconds equals 7776000 seconds, which equals 90 days. Looks like it, doesn’t it?
So the apparent workaround is to make the hibernation time very long, so it never happens. Let’s say 20 times 365 days? Good enough? That’s 630720000000 milliseconds. Yes, this number exceeds 232, but so does the original threshold. It’s a 64-bit machine, after all.
So just go (this doesn’t require rooting the phone):
$ adb shell device_config put permissions auto_revoke_unused_threshold_millis2 630720000000
And then verify that the new number is in effect:
$ adb shell device_config get permissions auto_revoke_unused_threshold_millis2
630720000000
Yey. Does it solve the problem? Nope, that didn’t work. A couple of weeks later I got the same notification, and this time a couple of other apps were hibernated.
Taking a closer look, it turned out that the value of auto_revoke_unused_threshold_millis2 had returned to 7776000000 (after a couple of weeks). How and why, I don’t know. I tried to change it back to the desired value, and ran a reboot, and the updated value survived that. So the values that are set with device_config are permanent. But another reboot later, it was back to 7776000000. So I don’t know what’s going on.
Maybe it’s because I didn’t change the parameter with the same name, but under the app_hibernation namespace?
$ adb shell device_config get app_hibernation auto_revoke_unused_threshold_millis2
7776000000
And maybe the easiest way is to to turn off hibernation altogether. There happens to be a parameter with an interesting name:
$ adb shell device_config get app_hibernation app_hibernation_enabled
true
Googling for this parameter name, I found absolutely nothing. But if it does what its name implies, maybe this will help?
$ adb shell device_config put app_hibernation app_hibernation_enabled false
I can’t even tell if this would help. The parameter got changed back soon enough to “true”.
Other ideas
Truth to be said, I don’t really know why and when the values of these parameters change back to their original values.
But here are a few clues: device_config has a “set_sync_disabled_for_tests” option, so I suppose
adb shell device_config set_sync_disabled_for_tests persistent
could prevent the configuration to reset all the time, but I didn’t try that.
It more than appears that the source of the default values for these parameters are in a permanent setting file, /data/system/users/0/settings_config.xml (needless to say, root access is required to invoke the 0/ directory). There are several similar .xml files in the 0/ directory, as listed in the definition of the class that processes these XML files, SettingsRegistry.
But here’s the crux: Despite the .xml suffix, this isn’t an XML file, but ABX, which is a format that is new in Android 12. It’s a condensed form of XML, which can’t be edited with just a text editor. Since this format is new and specific, there’s currently no reliable toolkit for making pinpoint changes. The only thing I found today (October 2022) is described below, along with its shortcomings.
An entry like this in settings_config.xml:
<setting id="3247" name="app_hibernation/app_hibernation_enabled" package="com.google.android.gms" preserve_in_restore="true" value="true" />
appears with a row like this in the output of “dumpsys settings”.
_id:3247 name:app_hibernation/app_hibernation_enabled pkg:com.google.android.gms value:true
In other words, the ABX file is clearly maintained by the “settings” service (this is what the second argument to dumpsys means). There’s a “settings” command-line utility too, but it doesn’t give access to this attribute (I’m not sure how related it is to the “settings” service).
An alternative approach could be to turn off the Hibernation service itself.
The services can be listed in Settings > System > Developer Options > Running Services, but these are only services that are linked with apps, it seems like. So not there.
On the other hand, the service can be found with dumpsys, and it’s called com.android.server.apphibernation.AppHibernationService. Unfortunately it can’t be disabled with pm disable-user as suggested on this page, be the argument to this command must be a package. The service’s name isn’t recognized by pm for this purpose.
I have no idea how to just disable a service. I’m not even sure it’s possible.
Additional trivia
In case this doesn’t work, here are some random pieces of info that might help. First of all, there’s a service called com.android.server.apphibernation.AppHibernationService. So maybe disable it somehow?
Another thing is that by looking at the output of “adb shell dumpsys package”, it appears like there’s a package called android.intent.action.MANAGE_UNUSED_APPS, which has an Intent named com.google.android.permissioncontroller/com.android.permissioncontroller.permission.ui.ManagePermissionsActivity. Maybe disable it, in the spirit of this post? It’s not clear if that’s the Activity that disables the app, or the one that allows enabling it back.
And then we have this snippet from the (adb shell dumpsys):
logs: "1663488654709 HibernationPolicy:i unused app com.google.android.apps.podcasts - last used on Thu Jan 01 02:00:00 GMT+02:00 1970 "
logs: "1663488654710 HibernationPolicy:i unused app com.google.android.apps.authenticator2 - last used on Thu Jan 01 02:00:00 GMT+02:00 1970 "
logs: "1663488654711 HibernationPolicy:i unused app com.gettaxi.android - last used on Thu Jan 01 02:00:00 GMT+02:00 1970 "
logs: " Globally hibernating apps [com.google.android.apps.podcasts, com.google.android.apps.authenticator2, com.gettaxi.android] "
The sources for the hibernation code:
git clone https://android.googlesource.com/platform/packages/modules/Permission
and the code is under PermissionController/src/com/android/permissioncontroller/. In particular, look at permission/utils/Utils.java for the list of parameters and their internal representation.
The source for the device_config utility is here. The implementation of DeviceConfig’s core methods is here.
Just in case it helps someone (future self in particular).
ABX manipulation
The only reference I found for ABX was in CCL Solution Group’s website. The said post describes the format, and also points at a git repository, which includes a couple of utilities for converting from XML to ABX and back.
The problem with this is that when I tried it on my settings_config.xml file, the back-and-forth conversion didn’t end with exactly the same binary. And I’m not sure if the difference matters. So if I can’t be sure that everything that isn’t related to the little change I want to make stays the same, I prefer not touch it at all.
I suppose there will be better tools in the future (or maybe I’ll write something like that myself, but beware, I will do it in Perl).
So this is how to work with the existing tool:
Downloading the pair of utilities from the git repo:
$ git clone https://github.com/cclgroupltd/android-bits
The last commit in the repository that I refer to below is 7031d0b, made in January 2022.
Change directory to ccl_abx, copy the desired ABX (.xml) file into there, and go
$ python3.6 ccl_abx.py settings_config.xml -mr > opened.xml
(python3.6, as it implies, is Python 3.6 on my machine. There are so many versions, and it didn’t work on earlier ones).
This creates an XML file with the content of the ABX file.
Now, manually edit opened.xml and remove the <root> and </root> tags at the beginning and end. Not clear why they were put there to begin with.
As for back-conversion to ABX, here comes some Java.
ABX manipulation II
This is an update on 27.2.23, following a comment I got to this post: It turns out that there are utilities for ABX conversion on the phone itself, which are wrappers to a Java utility:
$ cat /system/bin/abx2xml
#!/system/bin/sh
export CLASSPATH=/system/framework/abx.jar
exec app_process /system/bin com.android.commands.abx.Abx "$0" "$@"
$ cat /system/bin/xml2abx
#!/system/bin/sh
export CLASSPATH=/system/framework/abx.jar
exec app_process /system/bin com.android.commands.abx.Abx "$0" "$@"
And they work:
$ abx2xml settings_config.xml settings_config.xml.unabx
$ xml2abx settings_config.xml.unabx settings_config.xml.reabx
$ abx2xml settings_config.xml.reabx settings_config.xml.todiff
Unfortunately, settings_config.xml.reabx didn’t end up identical with settings_config.xml. Making a comparison between the outputs of the UNIX “strings” utility, I found a “true3″ string in each setting in the .reabx file, which wasn’t in the original.
Even weirder, the .unabx wasn’t identical to .todiff. But this made them identical (on a regular Linux machine):
$ dos2unix settings_config.xml.unabx
Say what? The ABX was converted into a text file with CR-LF on the first time, and only with LF on the second? I guess the answer to this is somewhere in the Java code.
But maybe it’s OK. I’ll give this a go sometime.
Ugh. Java stuff
Java is definitely not my cup of tea, neither my expertise. So don’t learn anything from me on how to work with Java. The only thing I can say is that it worked.
First, I installed the Java compiler (Linux mint 19):
# apt install default-jdk
Then change directory to makeabx/src.
Compile everything (this will probably make Java-savvy guys laugh, but hey, it worked and created .class files in the respective dirtectories).
$ javac $(find . -iname \*.java)
There will be a few notes about deprecated API, but that’s fine.
Then run the program with
$ java com.ccl.abxmaker.Main path/to/opened.xml
If this fails, saying something like “Could not find or load main class src.com.ccl.abxmaker.Main”, it’s because I tried to run the program from the wrong directory. Even though shell’s autocomplete filled in the class nicely, it’s still wrong. This has to be done from the src/ directory (at least in the way I clumsily compiled this stuff).
This creates a file with the same name and path as the input, but adds an .abx suffix.
The ABX file that I obtained from back-conversion wasn’t identical to the original, however: For example, the first XML element is <settings version=”-1″>. In the original ABX, it’s encoded as a 32-bit integer (0xffffffff), but in the opened.xml.abx it’s encoded as a string (“-1″).
Does it even matter? I don’t know. If the pair of utilities doesn’t pass this kind of regression test, am I ready to inject the result into my phone’s configuration system? Well, no.
Android is not my profession
I just want to say this first: I do completely different things for living. I’m not an Android expert, the last time I wrote code in Java was 20 years ago, and what’s written below is really not professional information nor advice. This is the stuff I found out while trying to get my phone under control.
When the phone doesn’t take “no” for an answer
This is an increasing phenomenon: Software treating us like kids, not asking what to do, not suggesting, but telling us what it’s about to do. Or just does it.
So let’s assume for a second that we’re grown-ups who can make our own decisions, and one of those decisions was not to update our phone’s software. Let’s leave the question if this is clever or not. It’s about who has control over the piece of electronics that you consider yours.
For the record, I would have been perfectly fine with having security patches applied frequently. The problem with updates is that there are “improvements” along with the security fixes, so there’s always something that suddenly doesn’t work after them. Because of a bug that will be fixed on the next update, of course.
The bad news is that to really stop the updates, you need a rooted phone, and if you haven’t done that yet, odds are that the price for doing that (i.e. wiping the phone completely) is much worse than an update itself.
Plus you need adb installed and know how to work with it (I discuss adb briefly in this post). Maybe that will change in the future, if there will be a simple app performing the necessary operations. Or maybe this will be integrated into the app that assists with rooting?
In case you want to get right to the point, jump to “Disabling activities” below. That’s where it says what to do.
All said below relates to Android 12, build SQ1D.220105.007 on a Google Pixel 6 Pro (and it quite appears like it’s going to stay that way).
That nagging popup window
After rooting the phone, there are two immediate actions that are supposed to turn off automatic system updates. Not that it helped much in my case, but here they are:
- Under Developer Options (which are enabled for unlocking anyhow). Turn off “Automatic system updates”.
- Go to Settings > Notifications > App Notifications > Google Play Settings and turn off System Update (to silence the “System update paused” notification)
And after some time, the famous “Install update to keep device secure” popup appears. And again, and again:
Which is a nuisance, and there’s always the risk of mistakenly tap “Install now”. But then it escalated to “Install now to control when your device updates”:
This is a simple ultimatum, made by the machine I’m supposed to own: Either you do the update yourself, or I do it anyhow. And if that turns out to blow up your phone bill, your problem. How cute.
Who’s behind the popup window?
The first step is to figure out what package initiates the request for an update.
But even before that, it’s necessary to understand how Intents and Activities work together to put things on the screen.
This page explains Activities and their relations with Intents, and this page shows a simple code example on this matter.
Activities are (primarily?) foreground tasks that represent a piece of GUI interaction, that are always visible on the screen, and they get paused as soon as the user chooses something else on the GUI.
Then there’s a thing called Intent in Android, which is an abstraction for performing a certain operation. This allows any software component to ask for a certain task to be completed, and that translates into an Activity. So the idea is that any software components requests an operation as an Intent, and some other software component (or itself) listens for these requests, and carries out the matching Activity in response.
For example, WhatsApp (like many other apps) has an Intent published for sharing files (ACTION_SEND), so when any application wants to share something, it opens a menu of candidates for sharing (which is all Intents published for that purpose), and when the user selects WhatsApp to share with, the application calls the Activity that is registered by WhatsApp for that Intent. Which file to share is given as an “extra” to the Intent (which simply means that the call has an argument). Note that what actually happens is that WhatApp takes over the screen completely, which is exactly the idea behind Activities.
Now some hands-on. When the popup appears, go
$ adb shell dumpsys window windows > dump.txt
That produces a lot of output, but there was this segment:
Window #10 Window{28d0746 u0 com.google.android.gms/com.google.android.gms.update.phone.PopupDialog}:
mDisplayId=0 rootTaskId=26 mSession=Session{55a992b 9996:u0a10146} mClient=android.os.BinderProxy@95ec721
mOwnerUid=10146 showForAllUsers=false package=com.google.android.gms appop=NONE
mAttrs={(0,0)(wrapxwrap) sim={adjust=pan forwardNavigation} ty=BASE_APPLICATION fmt=TRANSLUCENT wanim=0x10302f4 surfaceInsets=Rect(112, 112 - 112, 112)
fl=DIM_BEHIND SPLIT_TOUCH HARDWARE_ACCELERATED
pfl=USE_BLAST INSET_PARENT_FRAME_BY_IME
bhv=DEFAULT
fitTypes=STATUS_BARS NAVIGATION_BARS CAPTION_BAR}
Requested w=1120 h=1589 mLayoutSeq=4826
mBaseLayer=21000 mSubLayer=0 mToken=ActivityRecord{577992f u0 com.google.android.gms/.update.phone.PopupDialog t26}
mActivityRecord=ActivityRecord{577992f u0 com.google.android.gms/.update.phone.PopupDialog t26}
mAppDied=false drawnStateEvaluated=true mightAffectAllDrawn=true
mViewVisibility=0x0 mHaveFrame=true mObscured=false
mGivenContentInsets=[0,0][0,0] mGivenVisibleInsets=[0,0][0,0]
mFullConfiguration={1.0 425mcc1mnc [en_US,iw_IL,ar_PS] ldltr sw411dp w411dp h834dp 560dpi nrml long hdr widecg port night finger -keyb/v/h -nav/h winConfig={ mBounds=Rect(0, 0 - 1440, 3120) mAppBounds=Rect(0, 130 - 1440, 3064) mMaxBounds=Rect(0, 0 - 1440, 3120) mWindowingMode=fullscreen mDisplayWindowingMode=fullscreen mActivityType=standard mAlwaysOnTop=undefined mRotation=ROTATION_0} as.2 s.1 fontWeightAdjustment=0}
mLastReportedConfiguration={1.0 425mcc1mnc [en_US,iw_IL,ar_PS] ldltr sw411dp w411dp h834dp 560dpi nrml long hdr widecg port night finger -keyb/v/h -nav/h winConfig={ mBounds=Rect(0, 0 - 1440, 3120) mAppBounds=Rect(0, 130 - 1440, 3064) mMaxBounds=Rect(0, 0 - 1440, 3120) mWindowingMode=fullscreen mDisplayWindowingMode=fullscreen mActivityType=standard mAlwaysOnTop=undefined mRotation=ROTATION_0} as.2 s.1 fontWeightAdjustment=0}
mHasSurface=true isReadyForDisplay()=true mWindowRemovalAllowed=false
Frames: containing=[0,145][1440,3064] parent=[0,145][1440,3064] display=[0,145][1440,3064]
mFrame=[160,810][1280,2399] last=[160,810][1280,2399]
surface=[112,112][112,112]
WindowStateAnimator{9e9cbdf com.google.android.gms/com.google.android.gms.update.phone.PopupDialog}:
mSurface=Surface(name=com.google.android.gms/com.google.android.gms.update.phone.PopupDialog)/@0x223f02c
Surface: shown=true layer=0 alpha=1.0 rect=(0.0,0.0) transform=(1.0, 0.0, 0.0, 1.0)
mDrawState=HAS_DRAWN mLastHidden=false
mEnterAnimationPending=false mSystemDecorRect=[0,0][0,0]
mForceSeamlesslyRotate=false seamlesslyRotate: pending=null finishedFrameNumber=0
mDrawLock=WakeLock{a82daf5 held=false, refCount=5}
isOnScreen=true
isVisible=true
Also, in the output of
$ adb shell dumpsys > all.txt
there was a much more to-the-point section saying (pretty much at the beginning of this huge file):
Display 4619827677550801152 HWC layers:
---------------------------------------------------------------------------------------------------------------------------------------------------------------
Layer name
Z | Window Type | Comp Type | Transform | Disp Frame (LTRB) | Source Crop (LTRB) | Frame Rate (Explicit) (Seamlessness) [Focused]
---------------------------------------------------------------------------------------------------------------------------------------------------------------
Wallpaper BBQ wrapper#0
rel 0 | 2013 | DEVICE | 0 | 0 0 1440 3120 | 65.0 142.0 1375.0 2978.0 | [ ]
---------------------------------------------------------------------------------------------------------------------------------------------------------------
com.google.android.apps.nexuslaunche[...]exuslauncher.NexusLauncherActivity#0
rel 0 | 1 | DEVICE | 0 | 0 0 1440 3120 | 0.0 0.0 1440.0 3120.0 | [ ]
---------------------------------------------------------------------------------------------------------------------------------------------------------------
Dim Layer for - WindowedMagnification:0:31#0
rel -1 | 0 | DEVICE | 0 | 0 0 1440 3120 | 0.0 0.0 0.0 0.0 | [ ]
---------------------------------------------------------------------------------------------------------------------------------------------------------------
com.google.android.gms/com.google.android.gms.update.phone.PopupDialog#0
rel 0 | 1 | DEVICE | 0 | 48 698 1392 2511 | 0.0 0.0 1344.0 1813.0 | [*]
---------------------------------------------------------------------------------------------------------------------------------------------------------------
StatusBar#0
rel 0 | 2000 | DEVICE | 0 | 0 0 1440 145 | 0.0 0.0 1440.0 145.0 | [ ]
---------------------------------------------------------------------------------------------------------------------------------------------------------------
NavigationBar0#0
rel 0 | 2019 | DEVICE | 0 | 0 2952 1440 3120 | 0.0 0.0 1440.0 168.0 | [ ]
---------------------------------------------------------------------------------------------------------------------------------------------------------------
ScreenDecorOverlay#0
rel 0 | 2024 | DEVICE | 0 | 0 0 1440 176 | 0.0 0.0 1440.0 176.0 | [ ]
---------------------------------------------------------------------------------------------------------------------------------------------------------------
ScreenDecorOverlayBottom#0
rel 0 | 2024 | DEVICE | 0 | 0 2944 1440 3120 | 0.0 0.0 1440.0 176.0 | [ ]
---------------------------------------------------------------------------------------------------------------------------------------------------------------
This is much better, because the window in focus is clearly marked. No need to guess.
Another place to look at is
$ adb shell dumpsys activity recents > recent.txt
Where it said:
* Recent #0: Task{51b5cb8 #26 type=standard A=10146:com.google.android.gms U=0 visible=true mode=fullscreen translucent=true sz=1}
userId=0 effectiveUid=u0a146 mCallingUid=u0a146 mUserSetupComplete=true mCallingPackage=com.google.android.gms mCallingFeatureId=com.google.android.gms.ota_base
affinity=10146:com.google.android.gms
intent={act=android.settings.SYSTEM_UPDATE_COMPLETE flg=0x10848000 pkg=com.google.android.gms cmp=com.google.android.gms/.update.phone.PopupDialog}
mActivityComponent=com.google.android.gms/.update.phone.PopupDialog
rootWasReset=false mNeverRelinquishIdentity=true mReuseTask=false mLockTaskAuth=LOCK_TASK_AUTH_PINNABLE
Activities=[ActivityRecord{577992f u0 com.google.android.gms/.update.phone.PopupDialog t26}]
askedCompatMode=false inRecents=true isAvailable=true
taskId=26 rootTaskId=26
hasChildPipActivity=false
mHasBeenVisible=true
mResizeMode=RESIZE_MODE_RESIZEABLE_VIA_SDK_VERSION mSupportsPictureInPicture=false isResizeable=true
lastActiveTime=232790582 (inactive for 23s)
This is interesting, as it says which Intent and Activity stand behind the popup, just by asking what the last Activity requests were. Even more important, if the popup was dismissed or disappeared for any other reason, it can still be found here.
So no doubt, it’s com.google.android.gms that stands behind this popup. That’s Google Mobile Service, and it’s a package that is responsible for a whole lot. So disabling it is out of the question (and uninstalling impossible).
Under the section “ACTIVITY MANAGER PENDING INTENTS (dumpsys activity intents)” there was
#8: PendingIntentRecord{50a35f1 com.google.android.gms/com.google.android.gms.ota_base broadcastIntent}
uid=10146 packageName=com.google.android.gms featureId=com.google.android.gms.ota_base type=broadcastIntent flags=0x2000000
requestIntent=act=com.google.android.chimera.IntentOperation.TARGETED_INTENT dat=chimeraio:/com.google.android.gms.chimera.GmsIntentOperationService/com.google.android.gms.update.INSTALL_UPDATE pkg=com.google.android.gms (has extras)
sent=true canceled=false
which I suppose indicates that com.google.android.gms has requested to run its own .update.INSTALL_UPDATE at a later stage. In other words, this is the indication of the recurring request to run the INSTALL_UPDATE intent.
Disabling activities
The trick to disable the popup, as well as the update itself, is to disable certain Android Activities. This is inspired by this post in XDA forums.
First, find all activities in the system:
$ adb shell dumpsys package > dumpsys-package.txt
Then look for those specific to the package:
$ grep com.google.android.gms dumpsys-package.txt | less
Actually, one can narrow it down even more to those having the “.update.” substring:
$ grep com.google.android.gms dumpsys-package.txt | grep -i \\.update\\. | less
And eventually, disable what appears to be relevant activities (adb shell commands as root):
# pm disable com.google.android.gms/.update.SystemUpdateActivity
# pm disable com.google.android.gms/.update.SystemUpdateService
# pm disable com.google.android.gms/.update.SystemUpdateGcmTaskService
And possibly also the popup itself:
# pm disable com.google.android.gms/.update.phone.PopupDialog
# pm disable com.google.android.gms/.update.OtaSuggestionActivity
I’m not sure if all of these are necessary. The list might change across different versions of Android.
For each command, pm responds with a confirmation, e.g.
# pm disable com.google.android.gms/.update.SystemUpdateActivity
Component {com.google.android.gms/com.google.android.gms.update.SystemUpdateActivity} new state: disabled
And then reboot (not sure it’s necessary, but often “disable” doesn’t mean “stop”).
Are the changes in effect? Make a dump of the package:
$ adb shell pm dump com.google.android.gms > gms-dump.txt
and search for e.g. SystemUpdateActivity in the file. These features should appear under “disabledComponents:”.
However running “adb shell dumpsys activity intents”, it’s evident that the com.google.android.gms.update.INSTALL_UPDATE intent is still active. So this Intent will still be requested in the future. See below what happens with that.
So it’s quite clear that the popup can be disabled, but it’s less obvious what happens if the system wants to update itself when the relevant activity is disabled. Will it or will it not prevent the update? The proof is in the pudding.
So here’s the pudding
To begin with, the phone didn’t bug me again on updating, and neither has it done anything in that direction. Regardless (or not), there’s no problem updating all apps on the phone, and neither does it provoke any unwanted stuff. I’ve seen some speculations on the web, that System Update was somehow related to Google Play, and given my experience, I don’t think this is the case.
So disabling the Activities did the trick. It’s also possible to see exactly what happened by looking at the output of
$ adb shell logcat -d > all-log.txt
where it says this somewhere around the time it should have started messing around:
08-22 18:39:19.063 1461 1972 I ActivityTaskManager: START u0 {act=android.settings.SYSTEM_UPDATE_COMPLETE flg=0x10048000 pkg=com.google.android.gms (has extras)} from uid 10146
--------- beginning of crash
08-22 18:39:19.069 2838 7279 E AndroidRuntime: FATAL EXCEPTION: [com.google.android.gms.chimera.container.intentoperation.GmsIntentOperationChimeraService-Executor] idle
08-22 18:39:19.069 2838 7279 E AndroidRuntime: Process: com.google.android.gms, PID: 2838
08-22 18:39:19.069 2838 7279 E AndroidRuntime: android.content.ActivityNotFoundException: No Activity found to handle Intent { act=android.settings.SYSTEM_UPDATE_COMPLETE flg=0x10048000 pkg=com.google.android.gms (has extras) }
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at android.app.Instrumentation.checkStartActivityResult(Instrumentation.java:2087)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at android.app.Instrumentation.execStartActivity(Instrumentation.java:1747)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at android.app.ContextImpl.startActivity(ContextImpl.java:1085)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at android.app.ContextImpl.startActivity(ContextImpl.java:1056)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at android.content.ContextWrapper.startActivity(ContextWrapper.java:411)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at com.google.android.gms.update.reminder.UpdateReminderDialogIntentOperation.a(:com.google.android.gms@222615044@22.26.15 (190400-461192076):67)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at com.google.android.gms.update.reminder.UpdateReminderDialogIntentOperation.onHandleIntent(:com.google.android.gms@222615044@22.26.15 (190400-461192076):10)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at com.google.android.chimera.IntentOperation.onHandleIntent(:com.google.android.gms@222615044@22.26.15 (190400-461192076):2)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at uzr.onHandleIntent(:com.google.android.gms@222615044@22.26.15 (190400-461192076):4)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at ffi.run(:com.google.android.gms@222615044@22.26.15 (190400-461192076):3)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at ffh.run(:com.google.android.gms@222615044@22.26.15 (190400-461192076):11)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at cfvf.run(:com.google.android.gms@222615044@22.26.15 (190400-461192076):2)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1137)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:637)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at java.lang.Thread.run(Thread.java:1012)
08-22 18:39:19.092 1461 4210 I DropBoxManagerService: add tag=system_app_crash isTagEnabled=true flags=0x2
08-22 18:39:19.094 1461 1552 W BroadcastQueue: Background execution not allowed: receiving Intent { act=android.intent.action.DROPBOX_ENTRY_ADDED flg=0x10 (has extras) } to com.google.android.gms/.stats.service.DropBoxEntryAddedReceiver
08-22 18:39:19.094 1461 1552 W BroadcastQueue: Background execution not allowed: receiving Intent { act=android.intent.action.DROPBOX_ENTRY_ADDED flg=0x10 (has extras) } to com.google.android.gms/.chimera.GmsIntentOperationService$PersistentTrustedReceiver
08-22 18:39:19.270 1461 4215 I ActivityManager: Process com.google.android.gms (pid 2838) has died: fg SVC
08-22 18:39:19.271 1461 4215 W ActivityManager: Scheduling restart of crashed service com.google.android.gms/.chimera.GmsIntentOperationService in 1000ms for start-requested
08-22 18:39:20.273 1461 1552 D CompatibilityChangeReporter: Compat change id reported: 135634846; UID 10146; state: DISABLED
08-22 18:39:20.274 1461 1552 D CompatibilityChangeReporter: Compat change id reported: 177438394; UID 10146; state: DISABLED
08-22 18:39:20.274 1461 1552 D CompatibilityChangeReporter: Compat change id reported: 135772972; UID 10146; state: DISABLED
08-22 18:39:20.274 1461 1552 D CompatibilityChangeReporter: Compat change id reported: 135754954; UID 10146; state: ENABLED
08-22 18:39:20.275 1461 1553 D CompatibilityChangeReporter: Compat change id reported: 143937733; UID 10146; state: ENABLED
08-22 18:39:20.296 1461 1553 I ActivityManager: Start proc 7305:com.google.android.gms/u0a146 for service {com.google.android.gms/com.google.android.gms.chimera.GmsIntentOperationService}
Clearly, disabling the Activities made them ineligible for handling the SYSTEM_UPDATE_COMPLETE Intent, so an ActivityNotFoundException was thrown. Surprisingly enough, this exception wasn’t caught, so the com.google.android.gms process simply died and was quickly restarted by the system.
I also found these later on:
08-24 07:20:00.085 16126 25006 I SystemUpdate: [Execution,InstallationIntentOperation] Received intent: Intent { act=com.google.android.gms.update.INSTALL_UPDATE cat=[targeted_intent_op_prefix:.update.execution.InstallationIntentOperation] pkg=com.google.android.gms cmp=com.google.android.gms/.chimera.GmsIntentOperationService }.
08-24 07:20:00.114 16126 25006 W GmsTaskScheduler: com.google.android.gms.update.SystemUpdateGcmTaskService is not available. This may cause the task to be lost.
08-24 07:20:00.118 16126 25006 W GmsTaskScheduler: com.google.android.gms.update.SystemUpdateGcmTaskService is not available. This may cause the task to be lost.
08-24 07:20:00.119 16126 25006 W GmsTaskScheduler: com.google.android.gms.update.SystemUpdateGcmTaskService is not available. This may cause the task to be lost.
[ ... ]
08-24 07:20:00.198 16126 25006 I SystemUpdate: [Control,InstallationControl] Installation progress updated to (0x413, -1.000).
08-24 07:20:00.273 16126 25006 I SystemUpdate: [Control,ChimeraGcmTaskService] Scheduling task: DeviceIdle.
08-24 07:20:00.273 16126 25006 W GmsTaskScheduler: com.google.android.gms.update.SystemUpdateGcmTaskService is not available. This may cause the task to be lost.
08-24 07:20:00.274 16126 25006 I SystemUpdate: [Execution,ExecutionManager] Action streaming-apply executed for 0.17 seconds.
08-24 07:20:00.283 16126 25006 I SystemUpdate: [Execution,ExecutionManager] Action fixed-delay-execution executed for 0.01 seconds.
Bottom line: Disabling the activity causes the GMS service to die and restart every time it thinks about updating the system and/or nagging about it. So it won’t happen. Mission accomplished.
Actually, I’m not sure it’s possible to update the phone now, even if I wanted to.
It wasn’t obvious that this would work. Disabling the activity could have canceled just the GUI part of the update process. But it did.
It would have been more elegant to add another package, that supplies an Activity for the SYSTEM_UPDATE_COMPLETE Intent, that runs instead of the original, disabled one. This would have avoided this recurring crash. I don’t know if this is possible though.
Or even better, to disable the recurring call to this Intent. Ideas are welcome.
Or did it really work?
A couple of months later, and I noted something that looked like a huge download. Using “top” on “adb shell” revealed that the update_engine consumed some 60% CPU. Using logcat as above revealed several entries like these:
10-29 20:43:38.677 18745 24156 I SystemUpdate: [Control,InstallationControl] Installation progress updated to (0x111, 0.704).
10-29 20:43:38.717 18745 24156 I SystemUpdate: [Control,InstallationControl] Update engine status updated to 0x003.
10-29 20:43:38.832 1106 1106 I update_engine: [INFO:delta_performer.cc(115)] Completed 1691/2292 operations (73%), 1429588024/2027780609 bytes downloaded (70%), overall progress 71%
So there is definitely some download activity going on. The question is what will happen when it reaches 100%. And even more important, if there’s a way to prevent this from happening.
And maybe I’m just a bit too uptight. Using (as root)
# find /mnt/installer/0/emulated/ -cmin -1 2>/dev/null
to find newly update files, it appears like the activity is in /mnt/installer/0/emulated/0/Android/data/com.google.android.googlequicksearchbox/files/download_cache, and that the downloaded files are like soda-en-US-v7025-3.zip, which appears to be Speech On Device Access. And I’m fine with that.
Or did it really work II?
I owe this part to Namelesswonder’s very useful comment below. Indeed, the logs are at /data/misc/update_engine_log/, and there are indeed some files there from October, which is when there was some update activity. The progress of the download, in percents and bytes, is written in these logs.
The first log has an interesting row:
[1003/115142.712102] [INFO:main.cc(55)] A/B Update Engine starting
[1003/115142.720019] [INFO:boot_control_android.cc(68)] Loaded boot control hidl hal.
[1003/115142.738330] [INFO:update_attempter_android.cc(1070)] Scheduling CleanupPreviousUpdateAction.
[1003/115142.747361] [INFO:action_processor.cc(51)] ActionProcessor: starting CleanupPreviousUpdateAction
Any idea how to schedule CleanupPreviousUpdateAction manually? Sounds like fun to me.
The logs also name the URL of the downloaded package, https://ota.googlezip.net/packages/ota-api/package/d895ce906129e5138db6141ec735740cd1cd1b07.zip which is about 1.9 GB, so it’s definitely a full-blown update.
What is even more interesting from the logs is this line, which appears every time before starting or resuming an OTA download:
[1029/201649.018352] [INFO:delta_performer.cc(1009)] Verifying using certificates: /system/etc/security/otacerts.zip
It’s a ZIP file, which contains a single, bare-bone self-signed certificate in PEM format, which contains no fancy information.
It seems quite obvious that this certificate is used to authenticate the update file. What happens if this file is absent? No authentication, no update. Probably no download. So the immediate idea is to rename this file to something else. But it’s not that easy:
# mv otacerts.zip renamed-otacerts.zip
mv: bad 'otacerts.zip': Read-only file system
That’s true. /system/etc/security/ is on the root file system, and that happens to be a read-only one. But it’s a good candidate for a Magisk override, I guess.
Another thing that was mentioned in the said comment, is that the data goes to /data/ota_package/. That’s interesting, because in my system this directory is nearly empty, but the files’ modification timestamp is slightly before I neutralized the update activities, as mentioned above.
So it appears like the downloaded data goes directly into a raw partition, rather than a file.
There’s also /data/misc/update_engine/prefs/, which contains the current status of the download and other metadata. For example, update-state-next-data-offset apparently says how much has been downloaded already. What happens if this directory is nonexistent? Is it recreated, or is this too much for the updater to take?
As an initial approach, I renamed the “prefs” directory to “renamed-prefs”. A few days later, a new “prefs” directory was created, with just boot-id, previous-slot and previous-version files. Their timestamp matched a new log entry in update_engine_log/, which consisted of a CleanupPreviousUpdateAction. So apparently, the previous download was aborted and restarted.
So this time I renamed update_engine into something else, and added update_engine as a plain file. Before doing this, I tried to make the directory unreadable, but as root it’s accessible anyhow (in Android, not a normal Linux system). So the idea is to make it a non-directory, and maybe that will cause an irrecoverable failure.
Update 8.12.22: Nope, that didn’t help. A new log entry appeared in update_engine_log/, not a single word about the failure to create update_engine/prefs/, but a whole lot of rows indicating the download’s progress. /data/misc/update_engine remained as a plain file, so nothing could be stored in the correct place for the prefs/ subdirectory. I failed to find files with the names of those that are usually in prefs/ anywhere in the filesystem, so I wonder how the download will resume after it’s interrupted.
Update 8.5.24: For a long time, the phone hasn’t attempted to upgrade itself. Maybe because of the version gap. So in the end (?), I got some peace and quiet.
Maybe this is related?
/etc/init/ contains the services to be run during boot. One of them is called update_engine.rc, and another update_verifier.rc. Maybe delete these two files? I haven’t done this, because why mess with a phone that is OK now?
These are a few unrelated topics, which didn’t turn out helpful. So they’re left here, just in case.
Downloading the source
I had the idea of grabbing the sources for the GMS package, and see what went on there.
The list of Git repos is here (or the GitHub mirror here). I went for the this one:
git clone https://android.googlesource.com/platform/frameworks/base
or the GitHub mirror:
git clone https://github.com/aosp-mirror/platform_frameworks_base.git
however this doesn’t seem to include the relevant parts.
Permissions
Another approach I had in mind was to turn off the permission to make a system update. I don’t know if there actually is such.
Android’s permission system allows granting and revoking permissions as required. Just typing “pm” (package manager) on adb shell returns help information
To get a list of all permissions in the system:
$ adb shell pm list permissions -g > list.txt
But that covers everything, so the trick is to search for the appearances of the specific package of interest. Something like
$ grep com.google.android.gms list.txt | sort | less
and look at that list. I didn’t anything that appears to related to system update.
Introduction
Due to an incident that is beyond the scope of this blog, I wanted to put a 24/7 camera that watched a certain something, just in case that incident repeated itself.
Having a laptop that I barely use, and a cheap e-bay web camera, I thought I set up something and let ffmpeg do the job.
I’m not sure if a Raspberry Pi would be up for this job, even when connected to an external hard disk through USB. It depends much on how well ffmpeg performs on that platform. Haven’t tried. The laptop’s clear advantage is when there’s a brief power outage.
Overall verdict: It’s as good as the stability of the USB connection with the camera.
Note to self: I keep this in the misc/utils git repo, under surveillance-cam/.
Warming up
Show the webcam’s image on screen, the ffmpeg way:
$ ffplay -f video4linux2 /dev/video0
Let ffmpeg list the formats:
$ ffplay -f video4linux2 -list_formats all /dev/video0
Or with a dedicated tool:
# apt install v4l-utils
and then
$ v4l2-ctl --list-formats-ext -d /dev/video0
Possibly also use “lsusb -v” on the device: It lists the format information, not necessarily in a user-friendly way, but that’s the actual source of information.
Get all parameters that can be tweaked:
$ v4l2-ctl --all
See an example output for this command at the bottom of this post.
If control over the exposure time is available, it will be listed as “exposure_absolute” (none of the webcams I tried had this). The exposure time is given in units of 100µs (see e.g. the definition of V4L2_CID_EXPOSURE_ABSOLUTE).
Get a specific parameter, e.g. brightness
$ v4l2-ctl --get-ctrl=brightness
brightness: 137
Set the control (can be done while the camera is capturing video)
$ v4l2-ctl --set-ctrl=brightness=255
Continuous capturing
This is a simple bash script that creates .mp4 files from the captured video:
#!/bin/bash
OUTDIR=/extra/videos SRC=/dev/v4l/by-id/usb-Generic*
DURATION=3600
while [ 1 ]; do
TIME=`date +%F-%H%M%S`
if ! ffmpeg -f video4linux2 -i $SRC -t $DURATION -r 10 $OUTDIR/video-$TIME.mp4 < /dev/null ; then
echo 2-2 | sudo tee /sys/bus/usb/drivers/usb/unbind
echo 2-2 | sudo tee /sys/bus/usb/drivers/usb/bind
sleep 5;
fi
done
Comments on the script:
- To make this a real surveillance application, there must be another script that deletes old files, so that the disk isn’t full. My script on this matter is so hacky, that I left it out here.
- The real problem I encountered was occasional USB errors. They happened every now and then, without any specific pattern. Sometimes the camera disconnected briefly and reconnected right away, sometimes it failed to come back for a few minutes. Once in a week or so, it didn’t come back at all, and only a lot of USB errors appeared in the kernel log, so a reboot was required. This is most likely some kind of combination of cheap hardware, a long and not so good USB cable and maybe hardware + kernel driver issues. I don’t know. This wasn’t important enough to solve in a bulletproof way.
- Because of these USB errors, those two “echo 2-2″ commands attempt to reset the USB port if ffmpeg fails, and then sleep 5 seconds. The “2-2″ is the physical position of the USB port to which the USB camera was connected. Ugly hardcoding, yes. I know for sure that these commands were called occasionally, but whether this helped, I’m not sure.
- Also because of these disconnections, the length of the videos wasn’t always 60 minutes as requested. But this doesn’t matter all that much, as long as the time between the clips is short. Which it usually was (less than 5 seconds, the result of a brief disconnection).
- Note that the device file for the camera is found using a /dev/v4l/by-id/ path rather than /dev/video0, not just to avoid mixing between the external and built-in webcam: There were sporadic USB disconnections after which the external webcam ended up as /dev/video2. And then back to /dev/video1 after the next disconnection. The by-id path remained constant in the sense that it could be found with the * wildcard.
- Frame rate is always a dilemma, as it ends up influencing the file’s size, and hence how long back videos are stored. At 5 fps, an hour long .mp4 took about 800 MB for daytime footage, and much less than so during night. At 10 fps, it got up to 1.1 GB, so by all means, 10 fps is better.
- Run the recording on a text console, rather than inside a terminal window inside X-Windows (i.e. use Ctrl-Alt-F1 and Ctrl-Alt-F7 to go back to X). This is because the graphical desktop crashed at some point — see below on why. So if this happens again, the recording will keep going.
- For the purpose of running ffmpeg without a console (i.e. run in the background with an “&” and then log out), note that the ffmpeg command has a “< /dev/null”. Otherwise ffmpeg expects to be interactive, meaning it does nothing if it runs in the background. There’s supposed to be a -nostdin flag for this, and ffmpeg recognized it on my machine, but expected a console nevertheless. So I went for the old method.
How a wobbling USB camera crashes X-Windows
First, the spoiler: I solved this problem by putting a physical weight on the USB cable, close to the plug. This held the connector steady in place, and the vast majority of the problems were gone.
I also have a separate post about how I tried to make Linux ignore the offending bogus keyboard from being. Needless to say, that failed (because either you ban the entire USB device or you don’t ban at all).
This is the smoking gun in /var/log/Xorg.0.log: Lots of
[1194182.076] (II) config/udev: Adding input device USB2.0 PC CAMERA: USB2.0 PC CAM (/dev/input/event421)
[1194182.076] (**) USB2.0 PC CAMERA: USB2.0 PC CAM: Applying InputClass "evdev keyboard catchall"
[1194182.076] (II) Using input driver 'evdev' for 'USB2.0 PC CAMERA: USB2.0 PC CAM'
[1194182.076] (**) USB2.0 PC CAMERA: USB2.0 PC CAM: always reports core events
[1194182.076] (**) evdev: USB2.0 PC CAMERA: USB2.0 PC CAM: Device: "/dev/input/event421"
[1194182.076] (--) evdev: USB2.0 PC CAMERA: USB2.0 PC CAM: Vendor 0x1908 Product 0x2311
[1194182.076] (--) evdev: USB2.0 PC CAMERA: USB2.0 PC CAM: Found keys
[1194182.076] (II) evdev: USB2.0 PC CAMERA: USB2.0 PC CAM: Configuring as keyboard
[1194182.076] (EE) Too many input devices. Ignoring USB2.0 PC CAMERA: USB2.0 PC CAM
[1194182.076] (II) UnloadModule: "evdev"
and at some point the sad end:
[1194192.408] (II) config/udev: Adding input device USB2.0 PC CAMERA: USB2.0 PC CAM (/dev/input/event423)
[1194192.408] (**) USB2.0 PC CAMERA: USB2.0 PC CAM: Applying InputClass "evdev keyboard catchall"
[1194192.408] (II) Using input driver 'evdev' for 'USB2.0 PC CAMERA: USB2.0 PC CAM'
[1194192.408] (**) USB2.0 PC CAMERA: USB2.0 PC CAM: always reports core events
[1194192.408] (**) evdev: USB2.0 PC CAMERA: USB2.0 PC CAM: Device: "/dev/input/event423"
[1194192.445] (EE)
[1194192.445] (EE) Backtrace:
[1194192.445] (EE) 0: /usr/bin/X (xorg_backtrace+0x48) [0x564128416d28]
[1194192.445] (EE) 1: /usr/bin/X (0x56412826e000+0x1aca19) [0x56412841aa19]
[1194192.445] (EE) 2: /lib/x86_64-linux-gnu/libpthread.so.0 (0x7f6e4d8b4000+0x10340) [0x7f6e4d8c4340]
[1194192.445] (EE) 3: /usr/lib/xorg/modules/input/evdev_drv.so (0x7f6e45c4c000+0x39f5) [0x7f6e45c4f9f5]
[1194192.445] (EE) 4: /usr/lib/xorg/modules/input/evdev_drv.so (0x7f6e45c4c000+0x68df) [0x7f6e45c528df]
[1194192.445] (EE) 5: /usr/bin/X (0x56412826e000+0xa1721) [0x56412830f721]
[1194192.446] (EE) 6: /usr/bin/X (0x56412826e000+0xb731b) [0x56412832531b]
[1194192.446] (EE) 7: /usr/bin/X (0x56412826e000+0xb7658) [0x564128325658]
[1194192.446] (EE) 8: /usr/bin/X (WakeupHandler+0x6d) [0x5641282c839d]
[1194192.446] (EE) 9: /usr/bin/X (WaitForSomething+0x1bf) [0x5641284142df]
[1194192.446] (EE) 10: /usr/bin/X (0x56412826e000+0x55771) [0x5641282c3771]
[1194192.446] (EE) 11: /usr/bin/X (0x56412826e000+0x598aa) [0x5641282c78aa]
[1194192.446] (EE) 12: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0xf5) [0x7f6e4c2f3ec5]
[1194192.446] (EE) 13: /usr/bin/X (0x56412826e000+0x44dde) [0x5641282b2dde]
[1194192.446] (EE)
[1194192.446] (EE) Segmentation fault at address 0x10200000adb
[1194192.446] (EE)
Fatal server error:
[1194192.446] (EE) Caught signal 11 (Segmentation fault). Server aborting
[1194192.446] (EE)
The thing is that webcam presents itself as a keyboard, among others. I guess the chipset has inputs for control buttons (which the specific webcam doesn’t have), so as the USB device goes on and off, X windows registers the nonexistent keyboard on and off, and eventually some bug causes it to crash (note that number of the event device is 423, so there were quite a few on and offs). It might very well be that the camera camera connected, started some kind of connection event handler, which didn’t finish its job before it disconnected. Somewhere in the code, the handler fetched information that didn’t exist, it got a bad pointer instead (NULL?) and used it. Boom. Just a wild guess, but this is the typical scenario.
The crash can be avoided by making X windows ignore this “keyboard”. I did this by adding a new file named /usr/share/X11/xorg.conf.d/10-nocamera.conf as follows:
# Ignore bogus button on webcam
Section "InputClass"
Identifier "Blacklist USB webcam button as keyboard"
MatchUSBID "1908:2311"
Option "Ignore" "on"
EndSection
This way, X windows didn’t fiddle with the bogus buttons, and hence didn’t care if they suddenly went away.
Anyhow, it’s a really old OS (Ubuntu 14.04.1) so this bug might have been solved long ago.
Accumulation of /dev/input/event files
Another problem with this wobbling is that /dev/input/ becomes crowded with a lot of eventN files:
$ ls /dev/input/event*
/dev/input/event0 /dev/input/event267 /dev/input/event295
/dev/input/event1 /dev/input/event268 /dev/input/event296
/dev/input/event10 /dev/input/event269 /dev/input/event297
/dev/input/event11 /dev/input/event27 /dev/input/event298
/dev/input/event12 /dev/input/event270 /dev/input/event299
/dev/input/event13 /dev/input/event271 /dev/input/event3
/dev/input/event14 /dev/input/event272 /dev/input/event30
/dev/input/event15 /dev/input/event273 /dev/input/event300
/dev/input/event16 /dev/input/event274 /dev/input/event301
/dev/input/event17 /dev/input/event275 /dev/input/event302
/dev/input/event18 /dev/input/event276 /dev/input/event303
/dev/input/event19 /dev/input/event277 /dev/input/event304
/dev/input/event2 /dev/input/event278 /dev/input/event305
/dev/input/event20 /dev/input/event279 /dev/input/event306
/dev/input/event21 /dev/input/event28 /dev/input/event307
/dev/input/event22 /dev/input/event280 /dev/input/event308
/dev/input/event23 /dev/input/event281 /dev/input/event309
/dev/input/event24 /dev/input/event282 /dev/input/event31
/dev/input/event25 /dev/input/event283 /dev/input/event310
/dev/input/event256 /dev/input/event284 /dev/input/event311
/dev/input/event257 /dev/input/event285 /dev/input/event312
/dev/input/event258 /dev/input/event286 /dev/input/event313
/dev/input/event259 /dev/input/event287 /dev/input/event314
/dev/input/event26 /dev/input/event288 /dev/input/event315
/dev/input/event260 /dev/input/event289 /dev/input/event316
/dev/input/event261 /dev/input/event29 /dev/input/event4
/dev/input/event262 /dev/input/event290 /dev/input/event5
/dev/input/event263 /dev/input/event291 /dev/input/event6
/dev/input/event264 /dev/input/event292 /dev/input/event7
/dev/input/event265 /dev/input/event293 /dev/input/event8
/dev/input/event266 /dev/input/event294 /dev/input/event9
Cute, huh? And this is even before there was a problem. So what does X windows make of this?
$ xinput list
⎡ Virtual core pointer id=2 [master pointer (3)]
⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)]
⎜ ↳ ELAN Touchscreen id=9 [slave pointer (2)]
⎜ ↳ SynPS/2 Synaptics TouchPad id=13 [slave pointer (2)]
⎣ Virtual core keyboard id=3 [master keyboard (2)]
↳ Virtual core XTEST keyboard id=5 [slave keyboard (3)]
↳ Power Button id=6 [slave keyboard (3)]
↳ Video Bus id=7 [slave keyboard (3)]
↳ Power Button id=8 [slave keyboard (3)]
↳ Lenovo EasyCamera: Lenovo EasyC id=10 [slave keyboard (3)]
↳ Ideapad extra buttons id=11 [slave keyboard (3)]
↳ AT Translated Set 2 keyboard id=12 [slave keyboard (3)]
↳ USB 2.0 PC Cam id=14 [slave keyboard (3)]
↳ USB 2.0 PC Cam id=15 [slave keyboard (3)]
↳ USB 2.0 PC Cam id=16 [slave keyboard (3)]
↳ USB 2.0 PC Cam id=17 [slave keyboard (3)]
↳ USB 2.0 PC Cam id=18 [slave keyboard (3)]
↳ USB 2.0 PC Cam id=19 [slave keyboard (3)]
Now, let me assure you that there were not six webcams connected when I did this. Actually, not a single one.
Anyhow, I didn’t dig further into this. The real problem is that all of these /dev/input/event files have the same major. Which means that when there are really a lot of them, the system runs out of minors. So if the normal kernel log for plugging in the webcam was this,
usb 2-2: new high-speed USB device number 22 using xhci_hcd
usb 2-2: New USB device found, idVendor=1908, idProduct=2311
usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 2-2: Product: USB2.0 PC CAMERA
usb 2-2: Manufacturer: Generic
uvcvideo: Found UVC 1.00 device USB2.0 PC CAMERA (1908:2311)
uvcvideo 2-2:1.0: Entity type for entity Processing 2 was not initialized!
uvcvideo 2-2:1.0: Entity type for entity Camera 1 was not initialized!
input: USB2.0 PC CAMERA: USB2.0 PC CAM as /devices/pci0000:00/0000:00:14.0/usb2/2-2/2-2:1.0/input/input274
after all minors ran out, I got this:
usb 2-2: new high-speed USB device number 24 using xhci_hcd
usb 2-2: New USB device found, idVendor=1908, idProduct=2311
usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 2-2: Product: USB2.0 PC CAMERA
usb 2-2: Manufacturer: Generic
uvcvideo: Found UVC 1.00 device USB2.0 PC CAMERA (1908:2311)
uvcvideo 2-2:1.0: Entity type for entity Processing 2 was not initialized!
uvcvideo 2-2:1.0: Entity type for entity Camera 1 was not initialized!
media: could not get a free minor
And then immediately after:
systemd-udevd[4487]: Failed to apply ACL on /dev/video2: No such file or directory
systemd-udevd[4487]: Failed to apply ACL on /dev/video2: No such file or directory
Why these eventN files aren’t removed is unclear. The kernel is pretty old, v4.14, so maybe this has been fixed since.
Sample output of v412-all
This is small & junky webcam. Clearly no control over exposure time.
$ v4l2-ctl --all -d /dev/v4l/by-id/usb-Generic_USB2.0_PC_CAMERA-video-index0
Driver Info (not using libv4l2):
Driver name : uvcvideo
Card type : USB2.0 PC CAMERA: USB2.0 PC CAM
Bus info : usb-0000:00:14.0-2
Driver version: 4.14.0
Capabilities : 0x84200001
Video Capture
Streaming
Device Capabilities
Device Caps : 0x04200001
Video Capture
Streaming
Priority: 2
Video input : 0 (Camera 1: ok)
Format Video Capture:
Width/Height : 640/480
Pixel Format : 'YUYV'
Field : None
Bytes per Line: 1280
Size Image : 614400
Colorspace : Unknown (00000000)
Custom Info : feedcafe
Crop Capability Video Capture:
Bounds : Left 0, Top 0, Width 640, Height 480
Default : Left 0, Top 0, Width 640, Height 480
Pixel Aspect: 1/1
Selection: crop_default, Left 0, Top 0, Width 640, Height 480
Selection: crop_bounds, Left 0, Top 0, Width 640, Height 480
Streaming Parameters Video Capture:
Capabilities : timeperframe
Frames per second: 30.000 (30/1)
Read buffers : 0
brightness (int) : min=0 max=255 step=1 default=128 value=128
contrast (int) : min=0 max=255 step=1 default=130 value=130
saturation (int) : min=0 max=255 step=1 default=64 value=64
hue (int) : min=-127 max=127 step=1 default=0 value=0
gamma (int) : min=1 max=8 step=1 default=4 value=4
power_line_frequency (menu) : min=0 max=2 default=1 value=1
sharpness (int) : min=0 max=15 step=1 default=13 value=13
backlight_compensation (int) : min=1 max=5 step=1 default=1 value=1
Introduction
I bought a Google Pixel 6 Pro (P6P henceforth) a few months ago, and as always, I write down what I do as I do it. The result is below. I hope this will never be useful for myself, because if it does, it means I had the phone completely reset, and not I’m starting from scratch.
I’m not a great fan of mobile phones in general, and even less playing around with upgrading and setting them up. I know that to a lot of people, that is a bit of a technology kick, but being an Electrical Engineer, I don’t need another toy to play with.
So I had my previous smartphone for six year, and it ran Android 5.5 to its last day. Jumping to Android 12 is quite a leap, and it’s quite possible that things that I write about below are related to this jump, rather than the Pixel itself.
Generally speaking, this is an iPhone-wannabe phone. The user interface has taken a sharp turn towards sweet talking, and I can’t say I like it very much. It’s clearly aimed for plain people (to put it gently). Everything is automatic and “helpful”, but it’s difficult to figure out how to get control over the machine. In short, annoying.
Another thing is that the software is designed to have Wifi available most of the time. Or at least a very good Internet connection over the cellular network. Everything is backed up to the cloud, run on a cloud server and whatnot. Without Internet, the phone is crippled.
Google does however live up to my expectation from them in the sense that there is a way to mute all these annoyances. It just requires knocking them down one by one.
Notes while setting up
The most important thing: If you want to root your phone, do it immediately as you get it, before staring to set it up. As I’ve explained in a separate post on rooting, you will have to reset the phone to root the phone. So all data and setup will be lost.
- Do the initial startup properly and with Wifi present. Alternatively, go to Settings > System > Reset options > Erase all data (factory reset) to start over again. This doesn’t necessarily erase downloaded SIMs.
- There’s a “Pixel Tips” app, which is useful for getting information on what the phone can do. Something to keep me busy in relatively short periods of otherwise wasted time.
- WhatsApp’s backup on Google Cloud is perfectly fine for restoring all conversations. However it happens only on installation, so I had to reinstall the app to get the restoration kicked off. Worked like a charm, though.
- SIM: LG G4 uses a Micro SIM, and P6P is with a Nano SIM or eSIM. The latter is non-removable on the phone, and the idea is to download the information into it. Unfortunately, Partner (my cellular provider) doesn’t support eSIM on phones, so I used a regular SIM. There’s a tool to pick out the SIM tray from the phone in the phone’s kit, but any needle would probably do likewise.
- It’s possible to connect keyboard, mouse and even a USB stick.
- As for the USB stick, it requested to format the driver, and I agreed. Maybe because it had two partitions? Anyhow, it formatted it into plain VFAT, but then I couldn’t access the files nor copy files into it. To be investigated.
- Tap on the home screen (that is, on the wallpaper), pick “Home Settings” and disable “Swipe to access Google app” or else a silly news feed appears from the home screen.
- At an early stage, enter Google Photos with no network connectivity and tap the icon at the top right. Set the app to work without an account. Otherwise, all junk that has been uploaded into the Photos account appears in the app.
- I opted out Chat Features. SMS is old, and I don’t want any hassle with that.
- Also opted out: Settings > Internet > Wi-fi > Network preferences > Notify for public networks (I don’t want the phone to look for them at all).
- As for the ringtone, nothing like the LG’s good old Little Spheres.
- I initially disallowed apps to scan for Wifi, even when Wifi is turned off: Settings > Location > Location services > Wi-Fi scanning. However it turns out that some apps (Facebook Messenger, for example) fail to start in this situation. So I turned it on again.
- Turn off Assistant’s voice response: Settings > Apps > Default Apps > Digital assistant app > settings icon to top right (flywheel) > Spoken results. Hands-free searches only.
- Apps to uninstall: Google One (no backups, thanks).
- First prize for unnecessary notification: Need time to focus? Open Focus mode to pause distracting apps. So I opened the Settings > Notifications > App settings and turn off notification for Digital Wellbeing. Digital wellbeing is, among others, not being bothered by silly notifications.
- Improving the fingerprint detection I: Settings > Display > Increase Touch Sensitivity. Doesn’t help.
- Improving the fingerprint detection II: Press firmly, in particular in the training session. That gives the sensor more area to work with. It also appears like using different fingers each time doesn’t help.
- Turn off autocorrect (correction while typing on keyboard). It’s better to have a lot of typos than a text that is perfectly spelled but meaning something else than intended.
- Turn off revocation and removal of temporary files of apps that aren’t used for 90 days.
- Limit the battery’s charging level (see below).
Disable right edge swipe for “back”
The point is to disable the feature that a swipe from the right means “back”. Swiping from the left edge is enough for me, thanks.
The official suggestion is going to Settings > System > Gestures > System navigation > Gesture Settings, set Back Sensitivity for Right Edge to Low. But that merely reduces the sensitivity, not turning it off completely.
There’s the possibility to return to three-button navigation on that menu, but left edge sweep is actually good. It’s the other edge I wanted to get rid of.
So use adb shell to hack the setting, as suggested here:
$ adb shell
raven:/ $ settings put secure back_gesture_inset_scale_right 0
This worked. Amazing, huh? Power to command-line even on Android.
And rooting the phone is not required for this one. Developer mode and knowing a thing or two with computers, definitely yes.
Copying data from old phone
At some point in the setup process, the phone asked me to connect to my previous phone. So I took the little USB C to USB A adapter that came with the phone, and connected it to the same USB cable that I usually connect with a computer.
Some information was copied to the new phone: Photos and videos (if requested, this is slow), apps, contacts and my SMSes. But no application information: Instagram and Facebook didn’t know which user I was.. Not to mention WhatsApp conversations (but that can be obtained from the cloud backup).
But most important: The Google ID moves along to the new phone.
Prepare for a long session of app downloads, installations and upgrades after this copying.
Note that this copying can’t be repeated later on, unless the phone is reset per request.
“Room for improvement”
- There’s apparently a need to reboot the phone before using Waze, or else the voice instructions are missing or partial. I suppose that will fix itself somehow over time with a Waze upgrade.
- In screensaver mode, the phone usually displays the time with thin and dim numbers. However when a notification is canceled, it may display gibberish or even the wrong time (that is, the displayed time is stuck).
- The fingerprint detection is indeed not good. It works most of the time. I speculate that there’s a suspicion level factor involved, so if the phone thinks the overall situation is somewhat fishy, it becomes extra picky with matching my fingerprint.
Charging
An anecdotal set of measurements of currents, with the Ampere app: Starts at ~4540 mA, drops to ~2150 mA at 50%. But then it may wander towards ~3100 mA at 66%. Eventually it drops to ~1600 mA at 89%. At 98% it’s down at 500 mA.
When connected to a USB 2.0 port, it charges at ~600 mA, but only when the phone is in screensaver mode (probably because there’s not enough current when it’s on).
When connected to a QC fast charger, it ran at 1400 mA (at high charging percentages).
As a lot of people (on the web) have pointed out, the phone is indeed picky with charging cables, and I’m not sure yet what makes it accept a cable to charge rapidly with.
As for USB A to C charging cables, it’s a bit of a lottery. A random cable that I got directly from China (Ebay) connects the phone as USB 3.0, and charges the phone at fairly rapid speed even with QC (it was used in the experiments above).
But then I have two USB 2.0 cables: One is a Wesdar T38 which didn’t charge neither from the computer nor a charger, and a Miracase USB Type C cable, which charges the phone from both. The peculiar thing is that the phone was detected by the computer as a High Speed device (USB 2.0), but there was no charging initiated with the former. And this cable failed to charge with a various of sources I tried, while the latter was successful with all. Go figure.
May 2024 update: The battery is a balloon
After almost exactly two years of having this phone, I noted that the phone had opened by itself on its right side. Something was pushing the screen upwards. Having seen this happening on my previous LG G4, I immediately realized it was the battery that was inflated, most likely due to overcharging.
In fact, when I looked for a new phone a couple of years ago, the first criterion was a replaceable battery. It’s only when I realized that no such phone was available that I somehow convinced myself that if the battery is hard to replace, it must not be a problem anymore. Wrong.
Maybe it’s because nobody is really supposed to have an iPhone-wannabe phone for more than a couple of years, but the reason is more likely that my phone is charging all the time. It’s merely a habit of negligence, that I don’t use my phone and don’t care about it so much. I let it charge when I don’t use it, so it’s 100% when I’m leaving home. Once again, wrong. This phone wasn’t designed for people who work from home and have better toys to play with.
And of course I had adaptive charging enabled. Indeed, it did limit the charging level to 80% occasionally, giving me the false confidence that the the battery wouldn’t become a balloon. So here’s what my battery looked like when it was eventually taken out from my phone, dedicated to all of you out there who doubt the need to keep the charging level below 100%.
(click to enlarge)
It may not look very inflated, but believe me, the plastic wrapping was puffy and a bit scary to touch.
The battery’s part number of this battery is G63QN, by the way.
Having the battery replaced was a little quest of its own, as I live in Israel, and not close to Tel-Aviv. I bought the battery from Gomobile, who were kind enough to offer me to hand over my phone for repair, and get it back a week afterwards, hopefully. They also offered a replacement phone.
Had I lived in the Tel-Aviv area, I would have had the option of going to their lab, and having the replacement done on spot.
I called a whole lot of shops in my area (Haifa), and the answer was somewhere between “we don’t do Google Pixel” to “it’s impossible to get a suitable battery in Israel”. A few of them told me to get a battery on Aliexpress and they would do the installation.
In the end, I talked with Gomobile’s lab, who were kind enough to agree to sell me a battery at 350 NIS. The lab guy emphasized that they would install the battery too for the same price, but I went for buying. I brought the battery and the phone to a shop called SuperLink at Haifa’s Grand Mall (גראנד קניון), and a guy worked hard for an hour to made the replacement. Taking the phone apart was surprisingly hard, and also to remove the battery. But it also looked like this was the first time this guy replaced a battery on this phone. The advantage of this shop is that it’s so small that they don’t have any lab to hide in. So I was there during the entire thing. Just to be sure the guy was gentle (he was). That cost me another 100 NIS, which is really little for all the work put in.
Limiting the charge level
So keeping the charge level at 100% seems to be harmful, even though there are plenty of speculations out there if it’s really 100% or if the phone only says it’s 100%, but holds a lower level. It’s actually quite simple to determine with any battery monitoring app: Check the voltage and compare with the battery’s spec. Personally, I don’t care.
But as someone who barely uses the phone and has already seen one battery turning into a bag of chips, lowering the battery level to 80% sounds like a good idea. Since my life doesn’t revolve around my phone, monitoring the charging level would turn into a chore.
There are plenty of apps out there that sound an alarm when the battery level exceeds a certain percentage, so it can be unplugged. That rather ridiculous solution is common because there is no way for an app to control the battery charging unless the phone is rooted (see my notes on how to root the phone) and the app has been granted root privileges. And of course, it’s a legit method if you’re hugging your phone 24/7.
But my phone is luckily rooted, so no problem with that. I’ve written a separate post about that.
Introduction
I had some really annoying bots on one of my websites. Of the sort that make a million requests (like really, a million) per month, identifying themselves as a browser.
So IP blocking it is. I went for a minimalistic DIY approach. There are plenty of tools out there, but my experience with things like this is that in the end, it’s me and the scripts. So I might as well write them myself.
The IP set feature
Iptables has an IP set module, which allows feeding it with a set of random IP addresses. Internally, it creates a hash with these addresses, so it’s an efficient way to keep track of multiple addresses.
IP sets has been in the kernel since ages, but it has to be opted in the kernel with CONFIG_IP_SET. Which it most likely is.
The ipset utility may need to be installed, with something like
# apt install ipset
There seems to be a protocol mismatch issue with the kernel, which apparently is a non-issue. But every time something goes wrong with ipset, there’s a warning message about this mismatch, which is misleading. So it looks something like this.
# ipset [ ... something stupid or malformed ... ]
ipset v6.23: Kernel support protocol versions 6-7 while userspace supports protocol versions 6-6
[ ... some error message related to the stupidity ... ]
So the important thing is to be aware of is that odds are that the problem isn’t the version mismatch, but between chair and keyboard.
Hello, world
A quick session
# ipset create testset hash:ip
# ipset add testset 1.2.3.4
# iptables -I INPUT -m set --match-set testset src -j DROP
# ipset del testset 1.2.3.4
Attempting to add an IP address that is already in the list causes a warning, and the address isn’t added. So no need to check if the address is already there. Besides, there the -exist option, which is really great.
List the members of the IP set:
# ipset -L
Timeout
An entry can have a timeout feature, which works exactly as one would expect: The rule vanishes after the timeout expires. The timeout entry in ipset -L counts down.
For this to work, the set must be created with a default timeout attribute. Zero means that timeout is disabled (which I chose as a default in this example).
# ipset create testset hash:ip timeout 0
# ipset add testset 1.2.3.4 timeout 10
The ‘-exist’ flag causes ipset to re-add an existing entry, which also resets its timeout. So this is the way to keep the list fresh.
Don’t put the DROP rule first
It’s tempting to put the DROP rule with –match-set first, because hey, let’s give those intruders the boot right away. But doing that, there might be TCP connections lingering, because the last FIN packet is caught by the firewall as the new rule is added. Given that adding an IP address is the result of a flood of requests, this is a realistic scenario.
The solution is simple: There’s most likely a “state RELATED,ESTABLISHED” rule somewhere in the list. So push it to the top. The rationale is simple: If a connection has begun, don’t chop it in the middle in any case. It’s the first packet that we want killed.
Persistence
The rule in iptables must refer to an existing set. So if the rule that relies on the set is part of the persistent firewall rules, it must be created before the script that brings up iptables runs.
This is easily done by adding a rule file like this as /usr/share/netfilter-persistent/plugins.d/10-ipset
#!/bin/sh
IPSET=/sbin/ipset
SET=mysiteset
case "$1" in
start|restart|reload|force-reload)
$IPSET destroy
$IPSET create $SET hash:ip timeout 0
;;
save)
echo "ipset-persistent: The save option does nothing"
;;
stop|flush)
$IPSET flush $SET
;;
*)
echo "Usage: $0 {start|restart|reload|force-reload|save|flush}" >&2
exit 1
;;
esac
exit 0
The idea is that the index 10 in the file’s name is smaller than the rule that sets up iptables, so it runs first.
This script is a dirty hack, but hey, it works. There’s a small project on this, for those who like to do it properly.
The operating system in question is systemd-based, but this old school style is still in effect.
Maybe block by country?
Since all offending requests came from the same country (cough, cough, China, from more than 4000 different IP addresses) I’m considering to block them in one go. A list of 4000+ IP addresses that I busted in August 2022 with aggressive bots (all from China) can be downloaded as a simple compressed text file.
So the idea is going something like
ipset create foo hash:net
ipset add foo 192.168.0.0/24
ipset add foo 10.1.0.0/16
ipset add foo 192.168.0/24
and download the per-country IP ranges from IP deny. That’s a simple and crude tool for denial by geolocation. The only thing that puts me down a bit is that it’s > 7000 rules, so I wonder if that doesn’t put a load on the server. But what really counts is the number of sizes of submasks, because each submask size has its own hash. So if the list covers all possible sizes, from a full /32 down to say, 16/, there are 17 hashes to look up for each packet arriving.
On the other hand, since the rule should be after the “state RELATED,ESTABLISHED” rule, it only covers SYN packets. And if this whole thing is put as late as possible in the list of rules, it boils down to handling only packets that are intended for the web server’s ports, or those that are going to be dropped anyhow. So compared with the CPU cycles of handling the http request, even 17 hashes isn’t all that much.
The biggest caveat is however if other websites are colocated on the server. It’s one thing to block offending IPs, but blocking a whole country from all sites, that’s a bit too much.
Note to self: In the end, I wrote a little Perl-XS module that says if the IP belongs to a group. Look for byip.pm.
The blacklisting script
The Perl script that performs the blacklisting is crude and inaccurate, but simple. This is the part to tweak and play with, and in particular adapt to each specific website. It’s all about detecting abnormal access.
Truth to be told, I replaced this script with a more sophisticated mechanism pretty much right away on my own system. But what’s really interesting is the calls to ipset.
This script reads through Apache’s access log file, and analyzes each minute in time (as in 60 seconds). In other words, all accesses that have the same timestamp, with the seconds part ignored. Note that the regex part that captures $time in the script ignores the last part of :\d\d.
If the same IP address appears more than 50 times, that address is blacklisted, with a timeout of 86400 seconds (24 hours). Log file that correspond to page requisites and such (images, style files etc.) are skipped for this purpose. Otherwise, it’s easy to reach 50 accesses within a minute with legit web browsing.
There are several imperfections about this script, among others:
- Since it reads through the entire log file each time, it keeps relisting each IP address until the access file is rotated away, and a new one is started. This causes an update of the timeout, so effectively the blacklisting takes place for up to 48 hours.
- Looking in segments of accesses that happen to have the same minute in the timestamp is quite inaccurate regarding which IPs are caught and which aren’t.
The script goes as follows:
use warnings;
use strict;
my $logfile = '/var/log/mysite.com/access.log';
my $limit = 50;
my $timeout = 86400;
open(my $in, "<", $logfile)
or die "Can't open $logfile for read: $!\n";
my $current = '';
my $l;
my %h;
my %blacklist;
while (defined ($l = <$in>)) {
my ($ip, $time, $req) = ($l =~ /^([^ ]+).*?\[(.+?):\d\d[ ].*?\"\w+[ ]+([^\"]+)/);
unless (defined $ip) {
next;
}
next
if ($req =~ /^\/(?:media\/|robots\.txt)/);
unless ($time eq $current) {
foreach my $k (sort keys %h) {
$blacklist{$k} = 1
if ($h{$k} >= $limit);
}
%h = ();
$current = $time;
}
$h{$ip}++;
}
close $in;
foreach my $k (sort keys %blacklist) {
system('/sbin/ipset', 'add', '-exist', 'mysiteset', $k, 'timeout', $timeout);
}
It has to be run as root, of course. Most likely as a cronjob.
Introduction
These post contains a few technical notes of using Google Translate for translating LaTeX documents into Chinese, Japanese and Korean. The insights on the language-related issues are written down in a separate post.
Text vs. HTML
Google’s cloud translator can be fed with either plain text or HTML, and it returns the same format. Plain text format is out of the question for anything but translating short sentences, as it becomes impossible to maintain the text’s formatting. So I went for the HTML interface.
The thing with HTML is that whitespaces can take different forms and shapes, and they are redundant in many situations. For example, a newline is often equivalent to a plain space, and neither make any difference between two paragraphs that are enclosed by <p> tags.
Google Translate takes this notion to the extreme, and typically removes all newlines from the original text. OK, that’s understandable. But it also adds and removes whitespaces where it had no business doing anything, in particular around meaningless segments that aren’t translated anyhow. This makes it quite challenging when feeding the results for further automatic processing.
Setting up a Google Cloud account
When creating a new Google Cloud account, there’s an automatic credit of $300 to spend for three months. So there’s plenty of room for much needed experimenting. Too see the status of the evaluation period, go to Billing > Cost Breakdown and wait a minute or so for the “Free trial status” strip to appear at the top of the page. There’s no problem with “activating full account” immediately. The free trial credits remain, but it also means that real billing occurs when the credits are consumed and/or the trial period is over.
First create a new Google cloud account and enable the Google Translate API.
I went for Basic v2 translation (and not Advanced, v3). Their pricing is the same, but v3 is not allowed with an API key, and I really wasn’t into setting up a service account and struggle with OAuth2. The main advantage with v3 is the possibility to train the machine to adapt to a specific language pattern, but as mentioned in that separate post, I’m hiding away anything but common English language patterns.
As for authentication, I went for API keys. I don’t need any personalized info, so that’s the simple way to go. To obtain the keys, go to main menu (hamburger icon) > APIs and services > Credentials and pick Create Credentials, and choose to create API keys. Copy the string and use it in the key=API_KEY parameters in POST requests. It’s possible to restrict the usage of this key in various ways (HTTP referrer, IP address etc.) but it wasn’t relevant in my case, because the script runs only on my computer.
The web interface for setting up cloud services is horribly slow, which is slightly ironic and a bit odd for a company like Google.
The translation script
I wrote a simple script for taking a piece of text in English and translating it into the language of choice:
use warnings;
use strict;
use LWP::UserAgent;
use JSON qw[ from_json ];
our $WASTEMONEY = 0;
my $MAXLEN = 500000;
my $chars_per_dollar = 50000;
our $APIkey = 'your API key here';
my ($outfile, $origfile, $lang) = @ARGV;
die("Usage: $0 outfile origfile langcode\n")
unless (defined $origfile);
my $input = readfile($origfile);
askuser() unless ($WASTEMONEY);
my $len = length $input;
die("Cowardly refusing to translate $len characters\n")
if ($len > $MAXLEN);
writefile($outfile, translate($input, $lang));
sub writefile {
my ($fname, $data) = @_;
open(my $out, ">", $fname)
or die "Can't open \"$fname\" for write: $!\n";
binmode($out, ":utf8");
print $out $data;
close $out;
}
sub readfile {
my ($fname) = @_;
local $/;
open(my $in, "<", $fname)
or die "Can't open $fname for read: $!\n";
my $input = <$in>;
close $in;
return $input;
}
sub askuser {
my $len = length $input;
my $cost = sprintf('$%.02f', $len / $chars_per_dollar);
print "\n\n*** Approval to access Google Translate ***\n";
print "$len bytes to $lang, $cost\n";
print "Source file: $origfile\n";
print "Proceed? [y/N] ";
my $ans = <STDIN>;
die("Aborted due to lack of consent to proceed\n")
unless ($ans =~ /^y/i);
}
sub translate {
my ($text, $lang) = @_;
my $ua = LWP::UserAgent->new;
my $url = 'https://translation.googleapis.com/language/translate/v2';
my $res = $ua->post($url,
[
source => 'en',
target => $lang,
format => 'html',
key => $APIkey,
q => $text,
]);
die("Failed to access server: ". $res->status_line . "\n")
unless ($res->is_success);
my $data = $res->content;
my $json = from_json($data, { utf8 => 1 } );
my $translated;
eval {
my $d = $json->{data};
die("Missing \"data\" entry\n") unless (defined $d);
my $tr = $d->{translations};
die("Missing \"translations\" entry\n")
unless ((defined $tr) && (ref $tr eq 'ARRAY') &&
(ref $tr->[0] eq 'HASH'));
$translated = $tr->[0]->{translatedText};
die("No translated text\n")
unless (defined $translated);
};
die("Malformed response from server: $@\n") if ($@);
$translated =~ s/(<\/(?:p|h\d+)>)[ \t\n\r]*/"$1\n"/ge;
return $translated;
}
The substitution at the end of the translate() function adds a newline after each closing tag for a paragraph or header (e.g. </p>, <h1> etc.) so that the HTML is more readable with a text editor. Otherwise it’s all in one single line.
Protecting your money
By obtaining an API key, you effectively give your computer permission to spend money. Which is fine as long as it works as intended, but a plain bug in a script that leads to an infinite loop or recursion, or maybe just feeding the system with a huge file by mistake, can end up with consequences that are well beyond the CPU fan spinning a bit.
So there are two protection mechanisms in the script itself:
- The script prompts for permission, stating how much it will cost (based upon $20 / million chars).
- It limits a single translation to 500k chars (to avoid a huge file from being processed accidentally).
Another safety mechanism is to set up budgets and budget alerts. Go to Main menu (hamburger) > Billing > Budgets & Alerts. Be sure to check “Email alerts to billing admins and users”. If I got it right, budgets don’t protect against spending, but only sends notifications. So I selected a sum, and enabled only the 100% threshold. It seems to make sense to check all the Discounts and Promotion options in the Credits part, which makes sure that the alert is given for the money to be spent by deducing all promotion credits.
On top of that, it’s a good idea to set quota limits: Go to Main menu (hamburger) > IAM & Admin > Quotas. Set the filter to Translation to get rid of a lot of lines.
It’s also the place to get an accurate figure for the current consumption.
Enable the quota for “v2 and v3 general model characters per day”, which is the only character limit that isn’t per minute, and set it to something sensible, for example 2 million characters if you’re a modest user like myself. That’s $40, which is fairly acceptable damage if the computer goes crazy, and high enough not to hit the roof normally.
Also do something with “v3 batch translation characters using general models per day” and same with AutoML custom models. I don’t use these, so I set both to zero. Just to be safe.
There’s “Edit Quotas” to the top right. Which didn’t work, probably because I did this during the trial period, so quotas are meaningless, and apparently disabled anyhow (or more precisely, enabled to fixed limits).
So the way to do it was somewhat tricky (as it’s probably pointless): To enable a quota, right-click the “Cloud Translation API” to the left of the quota item, and open it in a new tab. Set up the quota figure there. But this description on how to do it might not be accurate for a real-life use. Actually, the system ignored my attempts to impose limits. They appeared on the page for editing them, but not on the main page.
Supporting CJK in LaTeX
I’m wrapping up this post with notes on how to feed LaTeX (pdflatex, more precisely) with Chinese, Japanese and Korean, with UTF-8 encoding, and get a hopefully reasonable result.
So first grab a few packages:
# apt install texlive-lang-european
# apt install texlive-lang-chinese
# apt install texlive-lang-korean
# apt install texlive-cjk-all
Actually, texlive-lang-european isn’t related, but as its name implies, it’s useful for European languages.
I first attempted with
\usepackage[UTF8]{ctex}
but pdflatex failed miserably with an error saying that the fontset ‘fandol’ is unavailable in current mode, whatever that means. After trying a few options back and forth, I eventually went for the rather hacky solution of using CJKutf8. The problem is that CJK chars are allowed only within
\begin{CJK}{UTF8}{gbsn}
[ ... ]
\end{CJK}
but I want it on the whole document, and I need the language setting to be made in a file that is included by the main LaTeX file (a different included file for each language). So I went for this simple hack:
\AtBeginDocument{\begin{CJK}{UTF8}{gbsn}}
\AtEndDocument{\end{CJK}}
As for the font, it appears like gbsn or gkai fonts should be used with Simplified Chinese, and bsmi or bkai for with Traditional Chinese. Since I translated into Simplified Chinese, some characters just vanished from the output document when trying bsmi and bkai. The back-translation to English of a document made with bsmi was significantly worse, so these dropped characters had a clear impact in intelligibility of the Chinese text.
I got this LaTeX warning saying
LaTeX Font Warning: Some font shapes were not available, defaults substituted.
no matter which of these fonts I chose, so it doesn’t mean much.
So the choice is between gbsn or gkai, but which one? To decide, I copy-pasted Chinese text from updated Chinese websites, and compared the outcome of LaTeX, based upon the TeX file shown below. It was quite clear that gbsn is closer to the fonts in use in these sites, even though I suspect it’s a bit of a Times New Roman: The fonts used on the web have less serifs than gbsn. So gbsn it is, even though it would have been nicer with a font with less serifs.
For Japanese, there’s “min”, “maru” and “goth” fonts. “Min” is a serif font, giving it a traditional look (calligraphy style) and judging from Japanese websites, it appears to be used primarily for logos and formal text (the welcoming words of a university’s president, for example).
“Maru” and “goth” are based upon simple lines, similar to plain text in Japanese websites. The latter is a bit of a bold version of “maru”, but it’s what seems to be popular. So I went with “goth”, which has a clean and simple appearance, similar to the vast majority of Japanese websites, even though the bold of “goth” can get a bit messy with densely drawn characters. It’s just that “maru” looks a bit thin compared to what is commonly preferred.
Korean has two fonts in theory, “mj” and “gt”. “mj” is a serif font with an old fashioned look, and “gt” is once again the plain, gothic version. I first failed to use the “gt” font even though it was clearly installed (there were a lot of files in the same directories as where the “mj” files were installed, only with “gt”). Nevertheless, trying the “gt” font instead of “mj” failed with
LaTeX Font Warning: Font shape `C70/gt/m/it' undefined
(Font) using `C70/song/m/n' instead on input line 8.
! Undefined control sequence.
try@size@range ...extract@rangefontinfo font@info
<-*>@nil <@nnil
But as it turns out, it should be referred to as “nanumgt”, e.g.
\begin{CJK}{UTF8}{nanumgt}
나는 멋진 글꼴을 원한다
\end{CJK}
It’s worth mentioning XeLaTeX, which allows using an arbitrary True Type font withing LaTeX, so the font selection is less limited.
See this page on fonts in Japanese and Korean.
For these tests, I used the following LaTeX file for use with e.g.
$ pdflatex test.tex
\documentclass{hitec}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{CJKutf8}
\newcommand{\thetext}
{
它说什么并不重要，重要的是它是如何写的。
}
\AtBeginDocument{}
\AtEndDocument{}
\title{This document}
\begin{document}
gbsn:
\begin{CJK}{UTF8}{gbsn}
\thetext
\end{CJK}
gkai:
\begin{CJK}{UTF8}{gkai}
\thetext
\end{CJK}
bsmi:
\begin{CJK}{UTF8}{bsmi}
\thetext
\end{CJK}
bkai:
\begin{CJK}{UTF8}{bkai}
\thetext
\end{CJK}
\end{document}
Introduction
This post summarizes my insights as I worked my way through translating some technical documents, written in LaTeX, into Chinese, Japanese and Korean. The immediate approach was to feed Google Translate with the pdf documents, but not only are the results ugly, but then there are a lot of technical terms in the documents which are better not translated. Even worse, there are code examples with explanation in the text, file names, references to variable names and other elements that become meaningless if translated.
One of the advantages of having the document written in LaTeX to begin with, is that the LaTeX text formatting commands effectively flag the parts that aren’t just plain language in the text, so it’s relatively easy to spot them and protect them. But that alone was a long way from the finish line, as elaborated in this unexpectedly long post.
A different post discusses the technical aspects of talking with Google Cloud’s API as well as creating documents in these languages with LaTeX.
I also did something similar with translating web pages. For example, the translation of this post to Chinese, Japanese and Korean.
This post was written in the summer of 2022, and odds are that things will change dramatically over the course of time.
Is translation by human better?
The short answer: Yes, as of 2023, human translation is much better. It’s mainly because there is no way to give the translating tool hints about the context. For example, the word “driver” could be a car driver or a term related to a computer. All translation tools just pick one meaning. Some tools allow choosing a specific dictionary, and there are ways to shape the behavior of the translator. But the results are far from satisfactory.
However but both options have their disadvantages: Working with a human necessarily requires trusting that a specific person will perform the job thoroughly, and well, that’s anything but taken for granted. It’s extremely difficult to verify that the work was done well, in particular when the document is technical, as it’s not possible to give it to just someone and ask if it’s well written. An automatic reverse translation will miss some poor translations (in particular poor translations of technical terms) and at the same time make false alarms.
But the worst problem with human translation is that every future change in text requires contacting the people who made the translation, and ask them to make the adjustments. They may not be so willing to do that. So unless you employ these people full-time, it may be difficult to translate small edits.
Another problem with humans is that significant errors in the meaning of the text might occur. It’s easy to reverse or otherwise obscure the meaning of a sentence because of a simple human error. “Be sure not to turn on the power supply” can easily turn into “Be sure to turn on the power supply”. Automatic reverse translation can reveal this, but it’s easy to miss an error like this, when the person that verifies the text already knows what it should say.
Automatic translation should be less likely to make a mistake of this sort, but the truth is that Google Translate, with all its Neural Network magic, turns out to be more human than desired in this matter: It’s not completely unusual that the meaning of the text changes, sometimes to the complete opposite.
It also has a variety of passive-aggressive behaviors, in particular ignoring whole sentences or part of them, mostly when the text becomes a bit rambling.
I had a case where the automatic translation ignored a “non-” prefix on a noun, and by doing so reversed the meaning of the sentence. I’ve also had a case where “must not” was translated into the equivalent of “doesn’t have to”.
The clear disadvantage of an automatic translation is poor expression and grammar. If the technique explained below is adopted, it’s however possible to end up with a fairly good result, even if the language is a bit off at times.
But this disadvantage can be mitigated by letting someone who knows the target language well proof-read the result. This person doesn’t need to know English well, but only be sensitive to the target language, so it’s easier to find someone for that job. And in particular when translating to Asian languages, it’s easy to tell the persons involved to ignore technical terms, as they are easily distinguishable, written in Latin script.
The results of this proof-reading session are only slight changes in word choice or ordering, and they can be verified against automatic translation as well as another person. In fact, in most cases, the best way is to improve the wording in the original language, until the person checking the text confirms it sounds OK.
Whether it’s worth the effort and cost to make this language cleanup is an open question. It’s a matter of how much the target audience appreciates the fact that the documentation is available in their language vs. how much the language defects come across badly.
Another issue with automatic translation is that words with more than one meaning can be mistranslated, in particular when the intended meaning is the less common one for a specific word (examples for that below). A back-translation doesn’t necessarily reveal a problem of this sort.
So with the possibility of having someone read through the translated text, the only remaining problem is when the meaning is changed unnoticed during the translation. Frankly speaking, I don’t know which option, human or machine, is better regarding this problem. The only real solution anyhow is to back-translate the text and read it through. Good luck with that.
General insights on automatic translation
Google Translate is based upon a neural network machine learning algorithm, which means that it’s chaotic by nature (in the scientific sense). That gives it a bit of a human touch, which surely makes the translations better, but also makes it quite unpredictable. In particular, it’s capable of making any possible mistake, no matter how pointless and unexpected. It’s impossible to be 100% sure that it won’t do this or that, and it’s not even a bug when a phrase in the original text just disappears, or when a meaningless string of characters gets translated to something else, also completely meaningless. Those small glitches are part of the game, and it makes automated processing of the translated text quite challenging.
Having said that, the general rule is that if Google Translate does weird things, it’s because it was fed with something it found hard to digest. So even if the weirdness doesn’t appear to be related to language, the best way to rectify this is to change the original text into a simpler, more common way to express the same idea. Unfortunately, this calls for dull, play-it-safe English. However with by far less silly typos and grammar mistakes.
If I was to speculate how Google Translate’s algorithm works, I would say something like this: Attempt to find recognizable words in the sentence, fix spelling mistakes (“did-you-mean” style) and try to match the words that are recognized with a known phrase from the huge training corpus. Pick the known translation into the desired language of the word pattern that fits best. Fill in the words that were unknown in the original language in the translated text in their natural positions — these are treated as names (of persons, places etc.).
Punctuation like full period and commas, as well as enclosure in parentheses, makes the translator treat each part separately, more or less.
The destination language matters a lot regarding the interpretation of the meaning of the text. It doesn’t seem like the question is “what does the original text mean” but “which language pattern was most common in the training data for translating into language X”.
The main takeaways for this speculation, regarding how to write for translation are:
- Use common expressions and language patterns. In particular, use the most commonly used words to express a certain meaning.
- Be super-careful with trivial spelling mistakes, as they break the statistics for the desired language pattern.
- If the translation is successful to one language, in the sense that the original meaning was “understood”, it doesn’t necessarily means it will be as successful to another one. Same goes with failure. It seems to depend on what the translations between the two languages are usually used for. In other words, judging by the results, it seems like translations into Hebrew are used more for everyday text, but translation into east Asian languages is more for technical documents. Hence the selection of meaning tends to be more technical with the latter.
- As there is no analysis of the semantics of the original sentence, anything can happen, including a translation that says the opposite of the original.
Interestingly enough, I’m under the impression that the translation with Google Lens is much better than the cloud based translation service. In particular, the cloud translation is more likely to produce nonsense translations because of small disturbances in the middle of text, where Google Lens’ translation seems to have extra wisdom to overcome such.
Translating to a foreign language
How do you know a translation is OK, when you don’t know the language it’s translated into? The short answer is that one can’t really know. It helps a lot knowing another language, even if it’s different from the target language, because it allows spotting misinterpretations of certain words, in particular technical ones. But often a word is poorly translated into one language, but fine with another.
There’s the possibility to translate it back to English, but that doesn’t always spot problems. Technical words like “clock”, “bus”, “sink”, “assertion” are translated to ridiculous words in Hebrew, for example, but the translation back looks OK in English. In particular a work like “sink” translates into the word in Hebrew that means kitchen sink, and then goes back to the correct work in English, of course.
But then comes the question: Why translate these words at all?
Quality of translation
Among the three target languages, the translation to (simplified) Chinese is the best by far. Probably because the natural flow of Chinese is the closest to western languages. The runner-up is Korean, and the worst is Japanese.
The worst problem with both Korean and Japanese is that parts of the original text can just disappear. This happens often when the semantic structure gets too complicated, or if there’s no normal way to say something in Japanese. For example, the sentence “you’re absolutely welcome to mess up completely, the tools won’t stand in your way” lost the entire first part in Japanese. So it just says “no tools get in the way”. If only the first part is translated separately, it turns into “completely ruined is welcome” (it had to give me something back when that sentence stood alone).
So short and plainly informative sentences are best translated into Japanese and Korean. Chinese seems to work with anything.
As for words like “it”, Chinese tolerates that best too. The two other languages are more likely to need repeating the word that “it” refers to, and hence possibly pick the wrong word to repeat.
Testing by translating to Hebrew
Since I happen to speak Hebrew fluently, I checked the translation to Hebrew of all documents, not for the purpose of publishing this translation, but because I soon found out that Google Translate struggles with Hebrew. So the idea was that if it’s OK with Hebrew, it’s probably OK any language.
For this, I tried two cases where the translation got wrong, as indicated by the result in Hebrew.
The first sentence that failed was “Close the window after a successful generation”. The problem was that the word “generation” was interpreted as the relationship between age groups, and not from the word “generate” as intended. This, in itself, is easily fixed by changing it into “Close the window after a successful generation of the file“. It was a matter of fitting the entire sentence into a different pattern of words.
Surprisingly enough, the translation into Chinese, Japanese and Korean was correct even without the fix. This can be verified by looking at the translation back to English, and isolate the word or couple of words of interest.
The next problematic phrase was “The non-X81X are ignored by almost all X82X computers”. In the translation to Hebrew, the “non-” part was ignored, so the sentence’ meaning was reversed. Once again, the translation into the three other languages was correct (the X81X thing is explained below).
So if I once had the speculation that the machine translates the words into an intermediate format that somehow contains the meaning, and takes it into the target language from there, it’s definitely not the case. Whether there’s a misunderstanding or not in the translation depends on the target language.
I’m optimistic and hope that Hebrew is in particular prone to errors, so if I clean up the translation to Hebrew, it will hopefully work with other languages. However odds are that each language has its own pitfalls. Even though it really seems like the translation to Hebrew from any language is bad in particular. Including changing the meaning of the text. Also, I’ve found that plain idioms like “it doesn’t hurt” are often translated horribly to Hebrew but get perfectly OK in CJK languages. But then, I don’t know about misses in CJK languages that were OK in Hebrew…? And yet, after checking numerous expressions (“bite back”, “copy-paste” and a lot of this sort) it really seems like Hebrew is really bad off.
This way or another, the only sure benefit of checking the translation to Hebrew is that it does, after all, remove some ambiguities, whether that is necessary or not. Actually, I found tons of plain typos by looking at this translation, so that alone justifies this phase. It’s difficult to proofread text exactly as it was written, but reading it again in another language feels as if someone else wrote it.
I also had the opportunity to have a translation into Japanese by a helpful person, and it was quite clear that the problems were in the places where the Hebrew translation also limped.
Hands-on insights
After quite some back and forth, I learned that the best way to work with Google Translate with text is to feed it with paragraphs of text in HTML, enclosed in <p> (or <hN>) tags. Plain formatting tags is fine (<b>, <i> and even <span> etc.) but it’s important not to insert anything that breaks the continuity of the sentences: No <br> or <img> tags in the middle, or anything else that isn’t expected in the middle of a sentence. It makes Google Translate translate the part before and after the break as separate sentences, and that’s a disaster.
Newlines are ignored in the cloud interface with HTML, as they should be. This is contrary to the web interface for Google Translate, which is extremely sensitive to newlines, so copy-pasting a chunk of text from a pdf document can result in a horrible translation, because there are newlines between each row in the original text, which makes the translator treat each row a separate phrase.
But the real difficulty is the fact that the translated text is technical. Google Translate is trained with mainly non-technical text (I guess), so its interpretation of technical terms that happen to also have a non-technical meaning is naturally inclined towards the non-technical meaning. Words like “driver”, “compile”, “assert” and “board” are not only likely to be translated incorrectly, but also stir a mess in that imaginary brain that holds all those neurons, resulting in a completely unintelligible translation.
The problematic words are those that have a possible non-technical meaning. The word “boot” could mean a piece of footwear, to boot a computer could be mistaken for “to give the computer the boot”, but to reboot a computer could only mean one thing. So it’s not all that much about the word being technical, like the fact that it could be remotely confusing.
Other ambiguities occur with words like “target”. Using it in any form, i.e. “to target” or “targeting” as well as “targeted” as in “depending on which software version is targeted” leads to a completely wonky translation, at least into Hebrew.
Surprisingly enough, it copes quite well with sentences that contain untranslatable items. I guess it treats anything it can’t handle as a name. Since it’s supposed to be able to translate “Joseph prefers Paris over Berlin”, it works fine with “X prefers Y over Z” as well. So the trick is to remove all technical terms from the the text, and replace them with something that Google Translate will treat as a name, something it can’t translate. And then return those words into the translated text.
This means that all technical terms remain in English in the translated text, which is perfectly fine, because a technical reader is expected to know these terms. It’s the blah-blah part that needs translation, and with the technical words out of the way, Google Translate does a good job on that.
The problem that remains is how to feed the translator with these untranslatable X, Y and Z placeholders, when there can be thousands of these, and they must all be left intact in the translation (well, except for Russian and Greek, see below). The section below on placeholders tells the full story, but the spoiler is that I used X0X, X1X, X2X, and so on. It’s not watertight, but it works best. I tried quite a few options.
The main thing to keep in mind is that it’s all about word patterns: If Google Translate recognizes the structure of the sentence, based upon words that are commonly used together for a certain meaning, it translates that part correctly, and then puts the placeholders in the right places, treating them as names.
I should mention that Google Translate offers a “notranslate” style, which can be used to enclose e.g. <span> segments of text that shouldn’t be translated. I didn’t attempt using it, in particular as people out there in the web have complained that it disrupts the translation exactly like that. Another problem is that chunks that shouldn’t be translated often have a different formatting (e.g. Courier font for variable names), and Google Translate tends to behave in an unpredictable manner, making it difficult to rely on its output for feeding LaTeX with directly.
Also worth mentioning is that Google offers an advanced version of the translation API, with the ability to train the learning machine and possibly feed it with specific word translations, but that would require knowing the correct term in the target language. How do you say “compile” in Chinese and Japanese? But it could have been helpful for solving the problem with verbs, that have a technical meaning (“compile”, “boot”, “implement”, “overflow”, you name it).
How I actually did it
The idea was to extract translatable text from the LaTeX source, and feed Google Translate’s cloud API with it in HTML mode. Then take the translated text and implant it back into the LaTeX doc.
The overall goal is to feed Google Translate with hollow phrases, albeit with a solid and common semantic structure, of the form of “X with Y is more important that Z”. This makes it easy for the translator to detect the structure of the phrase, and translate it to a meaningful sentence in the foreign language. That gives good odds for meaningful sentence when the placeholders are replaced with the actual technical words in the translated phrase.
In more detail:
- Fetch paragraphs of text and enclose them in <p> or <h1>, <h2> or <h3> tags. Each of these tags have a unique “id” attribute, so when the translation returns, it’s possible to track which text segments should be written back to which part in the LaTeX file. This is why HTML mode came handy. I haven’t had a single case of these attributes being messed up (yet?).
- Turn some LaTeX formatting into plain HTML tags, e.g. <b>, <i> etc. Then do the opposite when implanting the text back. The advantage is that this doesn’t break Google Translate’s view of the text as a contiguous sentence. Once again, HTML mode is required for this stunt.
- Anything that shouldn’t be translated — technical terms, references to variables, file names, references to paragraphs, labels etc. — is replaced with a unique identifier (“placeholder”) that Google Translate doesn’t attempt to translate. The major caveat with this method is that it works only with nouns. This requires rewording, in particular turning verbs into nouns (e.g. “perform a compilation” instead of “compile”). More on this below.
Note that some parts of the LaTeX document are completely out of this game, as they aren’t even given to the translator to look at. For example, verbatim environment chunks, and even the newlines between the text paragraphs. They remain the same because they aren’t overwritten when the translated text is transformed back and implanted in the relevant segment.
Work flow
I wrote a Perl script for the back-and-forth manipulations between LaTeX and HTML, but I won’t get into that too much, because it’s complicated and really specific to the documents at hand for translation. Among others, this script loaded a list of words that are always replaced with placeholders, and I also added a LaTeX command, \notranslate{}, which just leaves the content as is when interpreted by LaTeX, but to the script it means that the entire chunk should be replaced with a placeholder as well.
Writing scripts and all that is nice, but there’s still some manual preparation required. So this was the procedure I adopted:
- Run the script that creates the HTML file that is sent to Google Translate. View that file with a web browser, and look for words that are technical and can be mistranslated. When such are found, either add the word or phrase to the list of words to automatically replace with placeholders, or fix it specifically with \notranslate{} LaTeX statements.
- In fact, I also wrote a script that puts \notranslate{} on certain words and patterns (e.g. sets of upper case characters) so I ran this script, and then verified each such occurrence. This is faster than finding them manually, and is useful for words that may have a non-technical meaning, or otherwise require human attention to get 100% right. For example, the word “image” should be translated when it refers to a picture in the text, but not when it’s an image of a disk.
- Go through the text manually, and apply the guidelines listed below (the do’s and don’ts).
- Translate the text into Hebrew, and read through the result. If something ends up unclear, fix it. The further the language is from English, the better. The one sure benefit of this check is that small typos are spotted (e.g. “in” instead of “is”) because the translation gets weird. The fact that the order of words changes in the translation also helps spotting ambiguities, that are often solved with works like “which is” or punctuation.
- Translate into the target language. Make the necessary fixes. Don’t bother to find out why certain placeholders are missing in the translation. Rather, look at the original text and try to figure out why it was difficult to translate, and fix that instead. Sometimes a missing placeholder is due to a whole sentence being dropped off, in particular with Korean. It’s as if the algorithm said “I have no idea how to reorganize this sentence into something that makes sense in Korean, so I’ll just skip it”.
- Maybe attempt to translate the document back as a pdf file (with Google Translate’s web interface) or use Google Lens’ translation feature for a more sporadic check. I’m not sure if this is worth the time.
The order of translation is Korean first, then Japanese and finally Chinese. This is because the translation to Korean is the most troublesome, however often fixing the problems consists of changes that are likely to benefit the other translations.
All in all, it appears like using placeholders instead of technical terms actually improved the translation regardless of these specific words. It seems like these words confused the translation machinery, which made it create obscure phrasing. With the technical words out of the way, inserted as opaque symbols, it seems like Google Translate managed much better to handle the rest, which now consisted of commonly spoken language.
So my ultimate approach was to put placeholders instead of virtually all technical terms which are nouns. That’s quite a lot of them, and the translated documents ended up full with terms in English. I’m not sure what Chinese are going to think about this, but if they have the same problem as in Hebrew — weird “official words” for technical terms — it’s going to be just fine.
The do’s and don’ts
Based upon quite some trial and error, these are the guidelines I ended up with for producing text with placeholders that translates well.
- The text should consist of hollow sentences like “If the X113X process fails, the X641X’s output may supply hints on what went wrong. The X102X application on the computer should be configured for X114X, no X115X and no X116X ( X640X )”. However sentences like “The X219X for X220X on X221X or X222X is part of the majority of X223X and distributions, as explained below” should be fixed, inserting some meaningful words between those four placeholders with just one word between each. In this example, it’s not clear whether the last “or” refers to instead of X221X alone or all of the three. If the translation requires word reordering, this will obscure the meaning of the sentence.
- Use punctuation (in particular commas) and parentheses to chop up long sentences into segments. This prevents ambiguity. In particular, text inside parentheses is translated into parentheses, so this is a good tool for breaking up long and complicated sentences.
- Try to keep phrases short and concise (and somewhat boring), partly because sentences are short in the target languages. If the sentence is long, try to mitigate the damage with punctuation.
- Use plain and explicit English. Don’t leave out “which”, “that” and all those words that explicitly define the role of each word. Even a simple expression like “for the curious” can go wrong, but works perfectly well when changed into “for those who are curious”. Yuck, but translates well.
- Avoid words that refer back to something earlier in the sentence, unless it’s very obvious. In particular, the word “it” is often replaced with the word it’s supposed to refer to during the translation, and sometimes the wrong word is chosen in the translation. When this happens, the translation explicitly changes the meaning. Because the translation into CJK languages often involves splitting a long sentence into shorter ones, without a possibility to use a word like “it”, implicit references of this sort easily translate into nonsense. To make things worse, the back-translation may bring back the “it”, so there’s no way to spot the mistaken translation. There are cases where these duplications are safe, for example expressions like “one thing to another” (which is often translated into “one thing to another thing”).
- Prefer “the red book and the blue book” over “the red and blue books”. The order of the words may be changed during the translation, and in that case, it’s possible that only the “blue books” is moved to the new position in the sentence, so the result is rubbish. These overly explicit sentence are less awkward to read than they are awkward to write, but they are nevertheless slightly awkward as the same word is repeated over and over again.
- Avoid idioms. Even the simplest ones, like “out of the box” may and may not translate into something that makes sense. Because of the statistical nature of the translations, idioms might get translated with the right spirit into a certain language, and fail completely with another. So dull language it is.
- Avoid verbs in passive form, in particular if it comes with a “by”. Passive form is useful for not naming the doer, but if it’s named anyhow, use the active form. A lot of times, the passive form, and the tangled sentences that it usually creates, were the reason for problems in translation.
- Use possessive form for clarification. For example, if the word “register” is replaced with a placeholder, “register modification” should change to “modification of registers” or likewise “registers’ modification”. Using the ‘s suffix works great, so use it as long as it doesn’t create an ambiguity on who the owner is.
- In fact, there’s no problem at all with segments like “X400X’s X401X”, as possessive form. This translates well, surprisingly enough.
- Don’t replace partial expressions with placeholders. For example, in the expression “the user-space application”, don’t replace just “user-space”, but rather “user-space application”. Word ordering might be different in another language, which can at worst lead to a complete disassociation between the placeholder and its related word in English, with a completely unclear result.
- Avoid replacement of parts of expressions with placeholders. For example, in “VGA port”, if only “VGA” is replaced, it’s not sure if this will translate fine. “VGA-port” increases the changes. If it’s a common language pattern, e.g. “VGA-based”, there’s a good chance for proper translation. Same goes with “the X500X file”, because it’s a common language pattern.
- Don’t use “non-” as a prefix. It’s often missed, reversing the meaning.
- Look out for ambiguous words. For example, the word “signals” could be the verb (to signal) but also the plural of the noun. Avoid less common uses of words, such as “writes” to say several write operations, and use “write operations” instead.
- Be extra careful with trivial spelling mistakes and typos, in particular mixing “is” with “it” and such. These are overlooked when reading the text in English, but they kill the translation, sometimes by changing the meaning significantly, and sometimes by just confusing the translation algorithm into producing something weird.
- Bonus: Check all duplication of placeholders, and verify that the correct one is duplicated. Because these duplications are usually the result of a word that refers back to something (“which”, “that”, “it” etc.), it’s a good idea to verify that the reference goes to the correct placeholder. In theory, this should be done with all uses of back referencing, but that means proofreading the entire text. So with placeholders it’s less work (and less gain). Having run through a checkup of my own translations, I’d say about 10% of these duplications garble the meaning, by explicitly duplicating the wrong word.
Caching translation results?
Since the document is chopped into paragraphs, each within a <p> enclosure, does it matter if each is sent separately or if all are sent in one API transaction as a concatenated string? Does it matter if the translator sees the entire text?
Because if each <p> enclosure is treated separately, it’s possible to cache the pieces of text that have already been translated.
Caching is more than just a money saver. It allows making manual changes in Google Translate’s output (in particular if it messed up the placeholders) and then not having to repeat this every time the entire document is translated.
Even more important, avoiding the repeated translation of parts that have already been translated means avoiding the possible mishaps that may suddenly occur (like suddenly dropping a sentence). Think about making a small change, and then the translation fails on something completely different. But it worked last time!
This is also important if there’s feedback from readers that corrects a poor translation at a specific place. So caching is very helpful for the incremental kind of work that is necessary to maintain the document in the long run.
So I tried this with translating from English to Hebrew, and a bit with Chinese as well (by looking at the translation back to English). As it turns out, there are occasional differences between the translation of an isolated paragraph and that made with a context. But it doesn’t seem like an intelligent use of the context. Comparing the results, the isolated translation was sometimes better, sometimes worse, with a very slight difference in most cases. So it looks more like the algorithm randomly picked another wording, for no apparent reason. It was usually exchanging equally valid synonyms, or choosing to translate the name “Linux” to Hebrew or not.
Another observation I made is that the use of context is poor. For example, the word “call” is translated to the the word in Hebrew that means a phone call, but “function call” is translated correctly. So what if there’s a sentence saying something about a “function call”, and a sentence afterwards uses the word “caller”? In the <p> enclosure, that is. Well, the translation of “caller” still relates to a phone call. The neural network clearly didn’t learn anything from the first sentence.
So it makes perfect sense to cache translations at a paragraph level. If the original document changes, request a translation only on the enclosure that actually changed.
Finding the right kind of placeholder
This is a long explanation on why ended up with the XnX placeholders. I would skip this part if I were you.
As mentioned above, the main problem with translating a technical document is that some technical terms are translated into an unhelpful, sometimes ridiculous way, and that it confuses the translation algorithm. As the reader of the document is most likely familiar with the English term, it’s safer to leave these words as is. The problem is how to insert these terms in a way that ensures they don’t get translated, and at the same time retain their position in the context.
As it turned out, the main problem with inserting an untranslated chunk into the text is that it may disrupt the translation, in particular as Google Translate tends to treat the part before and after the chunk as separate sentences, which results in a poor translation that misses the point of the sentence.
I began with adding markers in a plain text (like <%103%>, [^29^] and ^^26^^), however Google Translate inserted a space in the middle of some of these (so it turned out to be e.g. “< %103%>”) and also threw in some markups where they shouldn’t be. A complete disaster, in short. This could have worked with non-HTML translation, but well, it didn’t work.
Another attempt was to use translation of HTML, with <b id=”n23″>P</b> markers as placeholders. The id allowed to identify which placeholder to insert, and the “P” to give the translator something to consider as a word. This failed as well, in many ways: The fact that the “P” part sometimes got translated into “PP” (why on earth) didn’t matter much, because it’s not really important. The real problem was that at times there were other words inserted into the <b> enclosure as well (for no apparent reason). Even worse, sometimes a completely different word, somewhere else in the sentence, got into a <b> enclosure with the same id. So processing this would have been complicated.
Another thing I tried was to use <var>n</var> enclosures, where n is the number of the placeholder. That failed, partly because some of these disappeared for no clear reason, and others were manipulated (for example, characters from previously outside the enclosure went into it).
To ensure that the placeholder is fully opaque, I tried <img id=n23>. The clear advantage was that Google Translate didn’t duplicate these not modify them, but they broke the sentence into fragments. Google Translate assumed that no sentence will have an image in the middle of it.
So if not images, what about emoticons? Or even better, I made an attempt to use the Unicode range U+13000 to U+1342e (Egyptian Hieroglyphs) as placeholders instead of <img> markups. The idea was that Google Translate would have to pass them through as is, and that they would be considered to be names. In order to make this work, there had to be a whitespace on both sides of the Hieroglyph, but even with that, Google Translate would mess up and occasionally add completely unrelated characters instead.
In the end, I went for inserting words like X0X, X1X, X2X, and so forth. These remain intact through translation, however they are occasionally duplicated, in particular with sentences like “that is possible with X, which is the best option” which can turn into “that is possible with X, and X is the best option”. The word “it” is also translated sometimes into the placeholder instead. But that’s actually a correct translation, and it’s easy to process. Even though this worked almost flawlessly, there were occasional surprises, including rare cases where Google Translate changed the number between the Xs without myself being able to figure out why on earth, and why that specific change. So there’s always a certain amount of manual cleanup after the translation.
These duplications are common with east Asian languages, and usually occur when a long sentence is chopped into several shorter ones. In these languages, it’s more common to repeat the word than to use “it”, “which” and such.
When translating to Russian and Greek, the “X” character was occasionally replaced with the Russian capital letter “Ha” (Unicode U+0425) or the Greek capital letter “Chi” (Unicode U+03A7). Both look exactly like an “X”, so the replacement is understandable. Once this issue is known, it’s quite easy to handle, so it’s not a big deal.
As for the quality of the translation, this worked well, and Google Translate combined these nicely into the translation, even when changing the word ordering was necessary. This works however only when the placeholder is used as a noun. So it doesn’t solve the problem with verbs like “assert”, “raise”. In some cases, a word like “overflow”, used as a verb, can be replaced with something like “cause an overflow”, so it can be translated properly.
Another thing with these XnX placeholders is that there must be a whitespace in either side of it, or Google Translate gets confused. To ensure that the placeholder is restored properly, the strategy was to include any surrounding whitespaces in the string that was stored to replace the placeholder later on, and then add a whitespace in either side of the XnX string. When reverting the process, all whitespaces around the XnX string were removed before restoring the original string. This results in a perfectly consistent back-and-forth, even if the translator adds or removes whitespaces (which happens a lot).
As a side note, Google charges for all characters, even those not translated. Hence it’s a good idea to keep the placeholders short markups. Not a big deal, but still.
Sanity checks on placeholders
The natural expectation is that any placeholder in the text for translation will result in a single placeholder in the translation. I’ve already mentioned above that some placeholders turned into two in the translated text, and it was actually correct. But what if the placeholder disappears?
The answer is that it’s always an error, and it has to be fixed manually. In fact, it’s often an indication that something worse happened, which would have been left unspotted had it not been for the missing placeholder. Sometimes the number between the Xs is changed arbitrarily, but it happens in conjunction with other placeholders in the vicinity being messed up.
Sometimes the absent placeholder was the result of a part of a sentence that was completely eliminated. The small piece of information it contained was simply absent in the translation. This can happen for several reasons, but the most recurring one seems to be when it’s not clear what “which” or “that” refers to, earlier in the same sentence. One can get away with that in translations to European languages, but because the sentence is built differently in east Asian languages, the translator is forced to make a pick. So instead of doing that, it just eliminates the part it can’t decide upon. A neural network algorithm showing a bit of human behavior, I would say.
It also seems that a colon sign (‘:’) tends to eliminate what comes immediately after it, fully or partly. Changing it to a full stop often returned chunks of texts from the dead in Korean and Japanese. Or splitting the text, so that part after the colon is in a separate enclosure (note to self: possibly with a \skipthis{}).
Same thing with a sentence starting with “likewise”.
Another somewhat weird phenomenon with Korean and Japanese is that a whole sentence was sometimes dropped. The really weird thing was that when the same sentence was put in a separate <p> enclosure, it was translated properly. So it was like Google Translate said “nah, this is too much rubbish, I’ll drop the last sentence”.
So in this sense, the placeholders help spotting other problems with the translation. I got an error of this sort for each few thousand translated words, which practically means a bit of fixing for each document. What’s really worrying is how many sentences without any placeholders have vanished unnoticed?
Placeholders that contain a word in plural
One problem that is inevitable with placeholders is that the information on the word’s plural vs. singular form is hidden away from the translator. So if the work that is hidden is “compilers”, the surrounding text in the translation might refer to it in singular, and that makes the sentence sound a bit off.
In some cases, the translator can deduce it from the surrounding words (e.g. if “is” or “are” is used in reference to it), but sometimes there are no hints. Luckily, the plural-singular thing isn’t very present in Chinese, Japanese and Korean, so the effect of this ambiguity is expected to be small. Try, for example to translate and back-translate “He gave me the books” with these languages, and you get “he gave me a book” — the indication for plural is lost. But there’s also a backside to this: The fact that the original word in English appears in its plural form will probably feel uneasy to an East Asian reader. I’m not sure about this, but it appears like they would use the English word in singular form anyhow, even if it refers to several pieces of whatever it is. So any use of plural will probably feel wrong to them.
Surprisingly, this can be fixed by using a placeholder like X205Xs (with the “s” in the end). This appears to be translated correctly into plural, and even the possessive form (e.g. X205Xs’) seems to work well into Hebrew.
But this hack creates a new problem: The translation might add suffixes and other grammatical elements to mark the plural form of the hidden word. If this happens, there will create a double plural. In German, for example, there are many ways to go from singular to plural, so this extra “s” just remains, when it comes after an XnX placeholder. If it isn’t removed, the result is “compilerss” (with a double “s” at the end). In Norwegian, it may add “-er” for plural (with the dash).
OK, so remove anything alphanumeric that comes after a placeholder, so that if the “s” remains, it’s gone? That may not work well either. For example, the possessive form in Swedish is expressed with a “:s” suffix and “:n” in Finnish (at least on a placeholder), so removing suffixes blindly takes its toll as well.
So even though appealing, there “s” method won’t work as a clean way to hint that the word is plural, in particular because the placeholder might get conjugated into plural in the translation. And there’s no catch-all solution for getting rid of this possible conjugation.
Given that the problem with plural is a relatively minor nuisance, that happens only when the context doesn’t say that it’s plural, it’s not worth the risk of adding garbage characters, or mistakenly removing relevant conjugation characters.
On the wishlist: The possibility to tell the translator that a blob is a noun in plural. Actually, wouldn’t it be nice to be able to do that with verbs as well, saying which tense and person?
Placeholders and Korean particles
In English, we have this thing that we say “a book” and “an orange”. The choice of the indefinite article, “a” or “an”, depends on whether the word that comes after it starts with a vowel or consonant sound.
In Korean, there are particles that are added after a noun to mark if it’s the subject, the topic or the object in the sentence. The particle is chosen according to whether the preceding word ends with a consonant or a vowel, respectively:
- Topic particles: 은 or 는 (eun or neun)
- Subject particles: 이 or 가 (i or ga)
- Object particles: 을 or 를 (eul or leul)
Not surprisingly, the particles that come after a vocal begin with a consonant, so there’s always a consonant in the game. Same principle as English’ indefinite article.
And here’s the crux: When a placeholder is used instead of a noun, Google Translate gets XnX instead of the real word, so the particle is chosen according to the “word” at hand.
So “I read the book” is translated by Google to 난 책을 읽는다 (book is 책, chaeg, ends with a consonant, hence the choice of the object particle 을, eul). But if “book” is replaced with “X10X”, we get 나는 X10X를 읽었다. “X” sounds like “eksae” in Korean, so it ends with a vowel, hence the 를 particle was used. (The word that means “I” changed from 난 to 나는, but the former is just a contraction of the latter, so it’s like “I’m” vs. “I am”)
This can be fixed automatically by looking for these particles: They are always immediately after a placeholder, and there’s a whitespace after them. The tricky part is to identify whether the replaced word ends with a consonant or a vowel, the way it’s pronounced in Korea (which may be different from the English pronunciation?).
The possessive particle, 의, as well as several other particles are indifferent to this matter.
It doesn’t seem like there’s a similar problem with Japanese nor Chinese, but I reached that conclusion based upon not finding anything related with a Google search. I will be really surprised if there was anything like this in Chinese because its script is generally unrelated to pronunciation. But with Japanese, I’m not that sure.
Maybe use a word in the target language?
I haven’t experimented a lot on this option, but maybe it will work: If a text is translated into Hebrew, and there is a Hebrew word in the middle of the text, it’s used correctly in the translation. So for example, “I ran back to בית quickly” is translated to “רצתי בחזרה לבית במהירות”. This isn’t perfect (הביתה would have been better) but it shows that a word in Hebrew is conjugated slightly and correctly.
So this opens for the possibility to replace technical terms with their relevant word in the target language. It seems like the grammar in CJK languages is exceptionally forgiving regarding nouns: There is generally no plural form, and it also seems like other conjugations are made with separate words (e.g possessive form).
Even more interesting, it works with verbs as well. “I רץ back to בית quickly” translated into “אני חוזר מהר לבית” which means “I quickly return home”. The word for “run” (רץ) was magically replaced with “return”, which is an interesting interpretation.
So maybe this can work. Not sure how much it improves, though.
It’s 2022, Perl isn’t as popular as it used to be, and for a moment I questioned its relevance. Until I had a task requiring a lot of pattern matching, which reminded me why Perl is that loyal companion that always has an on-spot solution to whatever I need.
These are a few notes I took as I discovered the more advanced, and well-needed, features of Perl regexps.
- If a regex is passed as a value generated by qr//, the modifiers in this qr// have a significance. So e.g. if the match should be case-insensitive, add it after the qr//.
- Quantifiers can be used on regex groups, whether they capture or not. For example, \d+(?:\.\d+)+ means one or more digits followed by one or more patterns of a dot and one or more digits. Think BNF.
- Complex regular expressions can be created relatively easily by breaking them down into smaller pieces and assigning each a variable with qr//. The complex expression becomes fairly readable this way. Almost needless to say, quantifiers can be applied on each of these subexpressions.
- It’s possible to give capture elements names, e.g. $t =~ /^(?<pre>.*?)(?<found>[ \t\n]*${regex}[ \t\n]*)(?<post>.*)$/s. The capture results then appear in e.g. $+{pre}, $+{found} and $+{post}. This is useful in particular if the regex in the middle may have capture elements of its own, so the usual counting method doesn’t work.
- Captured elements can be used in the regex itself, e.g. /([\'\"])(.*?)\1/ so \1 stands for either a single or double quote, whichever was found.
- Even better, there’s e.g \g{-1} instead of numeric grouping, which in this case means that last group captured. Once again, useful in a regex that can be used in more complicated contexts.
- When there are nested unnamed capture parentheses, the outer parenthesis gets the first capture number.
- If there are several capture parentheses with a ‘|’ between them, all of them produce a capture position, but those that weren’t in use for matching get undef.
- (?:…) grouping can be followed by a quantifier, so this makes perfect sense ((?:[^\\\{\}]|\\\\|\\\{|\\\})*) for any number of characters that aren’t a backslash or a curly bracket, or any of these followed by an escape.
- Quantifiers can be super-greedy in the sense that they don’t allow backtracking. So e.g. /a++b/ is exactly like /a+b/, but with the former the computer won’t attempt to consume less a’s (if such are found) in order to try to find a “b”. This is just an optimization for speed. All of these extra-greedy quantifiers are made with an extra plus sign.
- There’s lookbehind and lookahead assertions, which are really great. In particular, the negative assertions. E.g. /(?<![ \t\n\r])(d+)/ captures a number that isn’t after a whitespace, and /(\d+)(?![ \t\n\r])/ captures a number that isn’t followed by a whitespace. Note that the parentheses around these assertions are for grouping, but not capturing, so in these examples only the number was captured.
- Lookaheads and lookbehinds also work inside grouping parentheses (whether capturing or not), as grouping is treated as an independent regex.
About this messy post
As I rooted my Google Pixel 6 Pro, there were a few things to get in place on my Linux Mint 19 machine. These are random notes I took as I went along.
Install ADB and fastboot
# apt install android-tools-adb android-tools-fastboot
So that was easy.
Next, opening the phone for ADB access, the standard Android way: Go to the phone settings, into About Phone. Tap seven times on Build Number. The phone prompts for the PIN number, and then the “You are now a developer” message appears.
Now, under System, there’s the Developer options. Enable USB debugging. And then at shell prompt:
$ adb devices
List of devices attached
23011xxxxxxxx no permissions (user in plugdev group; are your udev rules wrong?); see [http://developer.android.com/tools/device.html]
Arghh. This was because of a lacking udev rule for when the phone isn’t in File Transfer / Android Auto mode. So I enabled file transfer (even though the real solution is to fix the udev file, as described below). And that’s when a popup appears asking if the computer should be allowed USB debugging. So yes, and pick “Always allow from this computer”.
Now we’re talking:
$ adb devices
23011xxxxxxxx device
Yey. Even more yey: Using adb shell, I got
$ id
uid=2000(shell) gid=2000(shell) groups=2000(shell),1004(input),1007(log),1011(adb),1015(sdcard_rw),1028(sdcard_r),1078(ext_data_rw),1079(ext_obb_rw),3001(net_bt_admin),3002(net_bt),3003(inet),3006(net_bw_stats),3009(readproc),3011(uhid) context=u:r:shell:s0
The privileges in this mode is much better than as an SSH client (see below). For example, one can list the files in /system/bin and /dev with adb shell and not through ssh (unless the ssh session gets root). Same goes with using “top”: It shows all processes, not just the same users’, as with ssh (once again, if the ssh session gets root, sky’s the limit).
There are nice executables in /vendor/bin too (even ifconfig and nc)
Getting the udev rule right
To resolve the “permission denied” thing with adb without file system access and fastboot, a udev rule needs to be added.
With MTP, file transfer on and off, I checked which udev rules got in action for the device with
$ udevadm test -a add $(udevadm info -q path -n /dev/bus/usb/002/004)
The 002/004 in the end are the bus and device numbers as found in lsusb.
It turns out that the relevant rule was in /lib/udev/rules.d/69-libmtp.rules:
# Google Inc Nexus/Pixel (MTP+ADB)
ATTR{idVendor}=="18d1", ATTR{idProduct}=="4ee2", SYMLINK+="libmtp-%k", MODE="660", GROUP="audio", ENV{ID_MTP_DEVICE}="1", ENV{ID_MEDIA_PLAYER}="1"
Note that it sets the group to “audio” and not “plugdev” The ENV{} assignments prevent the call to mtp-probe in this same udev file.
The truth is that I don’t completely understand why that works at all.
Anyhow, I ended up adding the following as /etc/udev/rules.d/20-google-pixel.rules:
# Google Inc Pixel 6 Pro, support for no-file transfer ADB mode
ATTR{idVendor}=="18d1", ATTR{idProduct}=="4ee7", MODE="660", GROUP="plugdev"
# Google Inc Pixel 6 Pro, support for USB tethering + ADB mode
ATTR{idVendor}=="18d1", ATTR{idProduct}=="4eec", MODE="660", GROUP="plugdev"
# Google Inc Pixel 6 Pro, support for MIDI + ADB mode
ATTR{idVendor}=="18d1", ATTR{idProduct}=="4ee9", MODE="660", GROUP="plugdev"
# Google Inc Pixel 6 Pro, support for fastboot mode
ATTR{idVendor}=="18d1", ATTR{idProduct}=="4ee0", MODE="660", GROUP="plugdev"
and don’t forget
$ sudo udevadm control --reload
I assigned the group “plugdev” or else fastboot required root (on the host computer) to detect the device. Also, I didn’t add any of the ENV{ID_MTP_DEVICE}=”1″ commands, because doing so prevents running mtp-probe. And oddly enough, if mtp-probe doesn’t run, I get a popup saying “Unable to mount mtp device”.
As a side note, mtp-probe runs on virtually every USB device. This is quite ugly, actually.
Backing up some of the phone’s data
You can’t just use tar to backup the root directory. Not sure why, but nothing happened when I tried. Maybe an SeLinux thing? So it boils down to something like this:
$ time adb shell 'su -c "tar -cz /storage/emulated/0/"' | sudo tar -xvz -C /mnt/tmp/googlepixel/
This backs up the visible user data. For hidden application data, repeat this with data/:
$ time adb shell 'su -c "tar -cz /data/"' | sudo tar -xvz -C /mnt/tmp/googlepixel/
And here’s the crux: There are several duplicate mounts in the filesystem. It seems like a lot under /data/user/0 is a repeated backup. And even more so, /data/media/0 is apparently a duplicate of /storage/emulated/0/ (or the other way around? I’m confused), so maybe there’s no point backing up /storage/emulated/0 if /data is backed up.
For a round-up of backing up things that are probably completely useless:
$ time adb shell 'su -c "tar -cz /mnt/vendor /apex /metadata"' | sudo tar -xvz -C /mnt/tmp/googlepixel/
Have I missed something that should be backed up? I don’t know.
Grabbing system info
Be sure to check out this reference and this cheat sheet for adb commands.
There’s a utility, dumpsys, that allows getting system information easily from the phone.
So, to tell which application is responsible for a window that just showed up:
$ adb shell dumpsys window windows > windows.txt
Or get some memory info (which application takes how much memory?):
$ adb shell dumpsys meminfo > mem.txt
What caused apps to terminate? This gives a log of last exits for each package.
$ adb shell dumpsys activity exit-info > exit.txt
Or everything at once (takes about 30 seconds to run, and emits a lot of error messages):
$ adb shell dumpsys > all.txt
Note that the file is output on the host, not on the phone with these commands.
To get the system log, use logcat:
$ adb shell logcat -d > all-log.txt
The -d flag tells logcat to terminate when it reaches the end of the log. Otherwise it continues running in “tail -f” style.
To get a list of all running processes:
$ adb shell ps -A > ps-all.txt
Look at this page for some additional utilities, in particular the Package Manager (pm) and Activity Manager (am). For a more in-depth understanding of the machinery, look for information on Intents and Activities.
Also try
$ adb shell device_config list
Bonus: device_config also allows to set the listed parameters, and these settings survive reboot (generally speaking).
List all installed packages:
$ adb shell cmd package list packages > packages.txt
Remove all user data for a package (at adb prompt, com.tencent.mm is WeChat)
pm clear com.tencent.mm
Where apps keep their data
For example, Whatsapp:
- /data/data/com.whatsapp/
- /mnt/installer/0/emulated/0/Android/data/com.whatsapp
The sensitive stuff is in the former. The latter may be accessible to anyone.
System startup
Services that are started during boot are listed in /etc/init/. There’s update_engine.rc and update_verifier.rc services there by the way.
Linux kernel
To download the Linux kernel, go
git clone https://android.googlesource.com/kernel/gs
This is a Generic Kernel Image kernel, meaning that it includes the parts that are common to all devices. Device-specific drivers are applied as kernel modules instead.
The Linux kernel that runs on my machine is commit 740f7fbe5f3917e0a5fa0582b98543af4a15a7ba (with no special tag), which identifies itself as v5.10.43. I deduced the commit based upon the “g740f7fbe5f39″ part in the kernel version that appears in the phone’s about part (5.10.43-android12-9-00005-g740f7fbe5f39).
Note however that some drivers don’t come from this kernel tree, but are rather loaded as modules from other sources. google_charge.c, for example.
SSH session with simpleSSHD
This is a no-cost app, which is essentially the Dropbear server.
Note: It’s also possible to connect with “adb shell” through a plain USB cable, and there’s “adb push” and “adb pull” to transfer files. So the advantage of SSH is limited.
Having SimpleSSHD running and started on the phone, I connected with the address provided on the screen
$ ssh -p 2222 user@10.11.12.13
the password appears on the phone’s screen. To use automatic login, go
cat > ~/authorized_keys
and copy-paste the content of ~/.ssh/id_rsa.pub there. Note that the file on the phone is not under a .ssh/ directory, which is probably why the ssh-copy-id utility doesn’t cut. Note however that once this file is found, a password login is not attempted if the host’s public key doesn’t match, so now try to log in from a computer that doesn’t have one of the listed keys.
In principle, the available executables are in /system/bin. The path contains more directories, but this is the only effective one.
Who am I?
$ id
uid=10225(u0_a225) gid=10225(u0_a225) groups=10225(u0_a225),3003(inet),9997(everybody),20225(u0_a225_cache),50225(all_a225) context=u:r:untrusted_app_29:s0:c225,c256,c512,c768
Backing up directly from phone (command run on receiving host):
$ ssh -p 2222 user@10.11.12.13 tar -cvz /storage/emulated/0/
The execution path…?
$ echo $PATH
/sbin:/system/sbin:/system/bin:/system/xbin
Unfortunately, these directories aren’t readable as a regular user, so it’s impossible to do a plain “ls” on them. Just in case I was looking for excuses to root the phone.
“more” also works fine, but there’s no “less”.
“top” works nicely, but shows only the same user’s processes. There’s also “ps”, but it seems to do the same. But hey, I rooted the phone. So
$ su
# id
uid=0(root) gid=0(root) groups=0(root) context=u:r:magisk:s0
And that’s the reason I rooted the phone, actually.
