Android is not my profession
I just want to say this first: I do completely different things for living. I’m not an Android expert, the last time I wrote code in Java was 20 years ago, and what’s written below is really not professional information nor advice. This is the stuff I found out while trying to get my phone under control.
When the phone doesn’t take “no” for an answer
This is an increasing phenomenon: Software treating us like kids, not asking what to do, not suggesting, but telling us what it’s about to do. Or just does it.
So let’s assume for a second that we’re grown-ups who can make our own decisions, and one of those decisions was not to update our phone’s software. Let’s leave the question if this is clever or not. It’s about who has control over the piece of electronics that you consider yours.
For the record, I would have been perfectly fine with having security patches applied frequently. The problem with updates is that there are “improvements” along with the security fixes, so there’s always something that suddenly doesn’t work after them. Because of a bug that will be fixed on the next update, of course.
The bad news is that to really stop the updates, you need a rooted phone, and if you haven’t done that yet, odds are that the price for doing that (i.e. wiping the phone completely) is much worse than an update itself.
Plus you need adb installed and know how to work with it (I discuss adb briefly in this post). Maybe that will change in the future, if there will be a simple app performing the necessary operations. Or maybe this will be integrated into the app that assists with rooting?
In case you want to get right to the point, jump to “Disabling activities” below. That’s where it says what to do.
All said below relates to Android 12, build SQ1D.220105.007 on a Google Pixel 6 Pro (and it quite appears like it’s going to stay that way).
That nagging popup window
After rooting the phone, there are two immediate actions that are supposed to turn off automatic system updates. Not that it helped much in my case, but here they are:
- Under Developer Options (which are enabled for unlocking anyhow). Turn off “Automatic system updates”.
- Go to Settings > Notifications > App Notifications > Google Play Settings and turn off System Update (to silence the “System update paused” notification)
And after some time, the famous “Install update to keep device secure” popup appears. And again, and again:
Which is a nuisance, and there’s always the risk of mistakenly tap “Install now”. But then it escalated to “Install now to control when your device updates”:
This is a simple ultimatum, made by the machine I’m supposed to own: Either you do the update yourself, or I do it anyhow. And if that turns out to blow up your phone bill, your problem. How cute.
Who’s behind the popup window?
The first step is to figure out what package initiates the request for an update.
But even before that, it’s necessary to understand how Intents and Activities work together to put things on the screen.
This page explains Activities and their relations with Intents, and this page shows a simple code example on this matter.
Activities are (primarily?) foreground tasks that represent a piece of GUI interaction, that are always visible on the screen, and they get paused as soon as the user chooses something else on the GUI.
Then there’s a thing called Intent in Android, which is an abstraction for performing a certain operation. This allows any software component to ask for a certain task to be completed, and that translates into an Activity. So the idea is that any software components requests an operation as an Intent, and some other software component (or itself) listens for these requests, and carries out the matching Activity in response.
For example, WhatsApp (like many other apps) has an Intent published for sharing files (ACTION_SEND), so when any application wants to share something, it opens a menu of candidates for sharing (which is all Intents published for that purpose), and when the user selects WhatsApp to share with, the application calls the Activity that is registered by WhatsApp for that Intent. Which file to share is given as an “extra” to the Intent (which simply means that the call has an argument). Note that what actually happens is that WhatApp takes over the screen completely, which is exactly the idea behind Activities.
Now some hands-on. When the popup appears, go
$ adb shell dumpsys window windows > dump.txt
That produces a lot of output, but there was this segment:
Window #10 Window{28d0746 u0 com.google.android.gms/com.google.android.gms.update.phone.PopupDialog}:
mDisplayId=0 rootTaskId=26 mSession=Session{55a992b 9996:u0a10146} mClient=android.os.BinderProxy@95ec721
mOwnerUid=10146 showForAllUsers=false package=com.google.android.gms appop=NONE
mAttrs={(0,0)(wrapxwrap) sim={adjust=pan forwardNavigation} ty=BASE_APPLICATION fmt=TRANSLUCENT wanim=0x10302f4 surfaceInsets=Rect(112, 112 - 112, 112)
fl=DIM_BEHIND SPLIT_TOUCH HARDWARE_ACCELERATED
pfl=USE_BLAST INSET_PARENT_FRAME_BY_IME
bhv=DEFAULT
fitTypes=STATUS_BARS NAVIGATION_BARS CAPTION_BAR}
Requested w=1120 h=1589 mLayoutSeq=4826
mBaseLayer=21000 mSubLayer=0 mToken=ActivityRecord{577992f u0 com.google.android.gms/.update.phone.PopupDialog t26}
mActivityRecord=ActivityRecord{577992f u0 com.google.android.gms/.update.phone.PopupDialog t26}
mAppDied=false drawnStateEvaluated=true mightAffectAllDrawn=true
mViewVisibility=0x0 mHaveFrame=true mObscured=false
mGivenContentInsets=[0,0][0,0] mGivenVisibleInsets=[0,0][0,0]
mFullConfiguration={1.0 425mcc1mnc [en_US,iw_IL,ar_PS] ldltr sw411dp w411dp h834dp 560dpi nrml long hdr widecg port night finger -keyb/v/h -nav/h winConfig={ mBounds=Rect(0, 0 - 1440, 3120) mAppBounds=Rect(0, 130 - 1440, 3064) mMaxBounds=Rect(0, 0 - 1440, 3120) mWindowingMode=fullscreen mDisplayWindowingMode=fullscreen mActivityType=standard mAlwaysOnTop=undefined mRotation=ROTATION_0} as.2 s.1 fontWeightAdjustment=0}
mLastReportedConfiguration={1.0 425mcc1mnc [en_US,iw_IL,ar_PS] ldltr sw411dp w411dp h834dp 560dpi nrml long hdr widecg port night finger -keyb/v/h -nav/h winConfig={ mBounds=Rect(0, 0 - 1440, 3120) mAppBounds=Rect(0, 130 - 1440, 3064) mMaxBounds=Rect(0, 0 - 1440, 3120) mWindowingMode=fullscreen mDisplayWindowingMode=fullscreen mActivityType=standard mAlwaysOnTop=undefined mRotation=ROTATION_0} as.2 s.1 fontWeightAdjustment=0}
mHasSurface=true isReadyForDisplay()=true mWindowRemovalAllowed=false
Frames: containing=[0,145][1440,3064] parent=[0,145][1440,3064] display=[0,145][1440,3064]
mFrame=[160,810][1280,2399] last=[160,810][1280,2399]
surface=[112,112][112,112]
WindowStateAnimator{9e9cbdf com.google.android.gms/com.google.android.gms.update.phone.PopupDialog}:
mSurface=Surface(name=com.google.android.gms/com.google.android.gms.update.phone.PopupDialog)/@0x223f02c
Surface: shown=true layer=0 alpha=1.0 rect=(0.0,0.0) transform=(1.0, 0.0, 0.0, 1.0)
mDrawState=HAS_DRAWN mLastHidden=false
mEnterAnimationPending=false mSystemDecorRect=[0,0][0,0]
mForceSeamlesslyRotate=false seamlesslyRotate: pending=null finishedFrameNumber=0
mDrawLock=WakeLock{a82daf5 held=false, refCount=5}
isOnScreen=true
isVisible=true
Also, in the output of
$ adb shell dumpsys > all.txt
there was a much more to-the-point section saying (pretty much at the beginning of this huge file):
Display 4619827677550801152 HWC layers:
---------------------------------------------------------------------------------------------------------------------------------------------------------------
Layer name
Z | Window Type | Comp Type | Transform | Disp Frame (LTRB) | Source Crop (LTRB) | Frame Rate (Explicit) (Seamlessness) [Focused]
---------------------------------------------------------------------------------------------------------------------------------------------------------------
Wallpaper BBQ wrapper#0
rel 0 | 2013 | DEVICE | 0 | 0 0 1440 3120 | 65.0 142.0 1375.0 2978.0 | [ ]
---------------------------------------------------------------------------------------------------------------------------------------------------------------
com.google.android.apps.nexuslaunche[...]exuslauncher.NexusLauncherActivity#0
rel 0 | 1 | DEVICE | 0 | 0 0 1440 3120 | 0.0 0.0 1440.0 3120.0 | [ ]
---------------------------------------------------------------------------------------------------------------------------------------------------------------
Dim Layer for - WindowedMagnification:0:31#0
rel -1 | 0 | DEVICE | 0 | 0 0 1440 3120 | 0.0 0.0 0.0 0.0 | [ ]
---------------------------------------------------------------------------------------------------------------------------------------------------------------
com.google.android.gms/com.google.android.gms.update.phone.PopupDialog#0
rel 0 | 1 | DEVICE | 0 | 48 698 1392 2511 | 0.0 0.0 1344.0 1813.0 | [*]
---------------------------------------------------------------------------------------------------------------------------------------------------------------
StatusBar#0
rel 0 | 2000 | DEVICE | 0 | 0 0 1440 145 | 0.0 0.0 1440.0 145.0 | [ ]
---------------------------------------------------------------------------------------------------------------------------------------------------------------
NavigationBar0#0
rel 0 | 2019 | DEVICE | 0 | 0 2952 1440 3120 | 0.0 0.0 1440.0 168.0 | [ ]
---------------------------------------------------------------------------------------------------------------------------------------------------------------
ScreenDecorOverlay#0
rel 0 | 2024 | DEVICE | 0 | 0 0 1440 176 | 0.0 0.0 1440.0 176.0 | [ ]
---------------------------------------------------------------------------------------------------------------------------------------------------------------
ScreenDecorOverlayBottom#0
rel 0 | 2024 | DEVICE | 0 | 0 2944 1440 3120 | 0.0 0.0 1440.0 176.0 | [ ]
---------------------------------------------------------------------------------------------------------------------------------------------------------------
This is much better, because the window in focus is clearly marked. No need to guess.
Another place to look at is
$ adb shell dumpsys activity recents > recent.txt
Where it said:
* Recent #0: Task{51b5cb8 #26 type=standard A=10146:com.google.android.gms U=0 visible=true mode=fullscreen translucent=true sz=1}
userId=0 effectiveUid=u0a146 mCallingUid=u0a146 mUserSetupComplete=true mCallingPackage=com.google.android.gms mCallingFeatureId=com.google.android.gms.ota_base
affinity=10146:com.google.android.gms
intent={act=android.settings.SYSTEM_UPDATE_COMPLETE flg=0x10848000 pkg=com.google.android.gms cmp=com.google.android.gms/.update.phone.PopupDialog}
mActivityComponent=com.google.android.gms/.update.phone.PopupDialog
rootWasReset=false mNeverRelinquishIdentity=true mReuseTask=false mLockTaskAuth=LOCK_TASK_AUTH_PINNABLE
Activities=[ActivityRecord{577992f u0 com.google.android.gms/.update.phone.PopupDialog t26}]
askedCompatMode=false inRecents=true isAvailable=true
taskId=26 rootTaskId=26
hasChildPipActivity=false
mHasBeenVisible=true
mResizeMode=RESIZE_MODE_RESIZEABLE_VIA_SDK_VERSION mSupportsPictureInPicture=false isResizeable=true
lastActiveTime=232790582 (inactive for 23s)
This is interesting, as it says which Intent and Activity stand behind the popup, just by asking what the last Activity requests were. Even more important, if the popup was dismissed or disappeared for any other reason, it can still be found here.
So no doubt, it’s com.google.android.gms that stands behind this popup. That’s Google Mobile Service, and it’s a package that is responsible for a whole lot. So disabling it is out of the question (and uninstalling impossible).
Under the section “ACTIVITY MANAGER PENDING INTENTS (dumpsys activity intents)” there was
#8: PendingIntentRecord{50a35f1 com.google.android.gms/com.google.android.gms.ota_base broadcastIntent}
uid=10146 packageName=com.google.android.gms featureId=com.google.android.gms.ota_base type=broadcastIntent flags=0x2000000
requestIntent=act=com.google.android.chimera.IntentOperation.TARGETED_INTENT dat=chimeraio:/com.google.android.gms.chimera.GmsIntentOperationService/com.google.android.gms.update.INSTALL_UPDATE pkg=com.google.android.gms (has extras)
sent=true canceled=false
which I suppose indicates that com.google.android.gms has requested to run its own .update.INSTALL_UPDATE at a later stage. In other words, this is the indication of the recurring request to run the INSTALL_UPDATE intent.
Disabling activities
The trick to disable the popup, as well as the update itself, is to disable certain Android Activities. This is inspired by this post in XDA forums.
First, find all activities in the system:
$ adb shell dumpsys package > dumpsys-package.txt
Then look for those specific to the package:
$ grep com.google.android.gms dumpsys-package.txt | less
Actually, one can narrow it down even more to those having the “.update.” substring:
$ grep com.google.android.gms dumpsys-package.txt | grep -i \\.update\\. | less
And eventually, disable what appears to be relevant activities (adb shell commands as root):
# pm disable com.google.android.gms/.update.SystemUpdateActivity
# pm disable com.google.android.gms/.update.SystemUpdateService
# pm disable com.google.android.gms/.update.SystemUpdateGcmTaskService
And possibly also the popup itself:
# pm disable com.google.android.gms/.update.phone.PopupDialog
# pm disable com.google.android.gms/.update.OtaSuggestionActivity
I’m not sure if all of these are necessary. The list might change across different versions of Android.
For each command, pm responds with a confirmation, e.g.
# pm disable com.google.android.gms/.update.SystemUpdateActivity
Component {com.google.android.gms/com.google.android.gms.update.SystemUpdateActivity} new state: disabled
And then reboot (not sure it’s necessary, but often “disable” doesn’t mean “stop”).
Are the changes in effect? Make a dump of the package:
$ adb shell pm dump com.google.android.gms > gms-dump.txt
and search for e.g. SystemUpdateActivity in the file. These features should appear under “disabledComponents:”.
However running “adb shell dumpsys activity intents”, it’s evident that the com.google.android.gms.update.INSTALL_UPDATE intent is still active. So this Intent will still be requested in the future. See below what happens with that.
So it’s quite clear that the popup can be disabled, but it’s less obvious what happens if the system wants to update itself when the relevant activity is disabled. Will it or will it not prevent the update? The proof is in the pudding.
So here’s the pudding
To begin with, the phone didn’t bug me again on updating, and neither has it done anything in that direction. Regardless (or not), there’s no problem updating all apps on the phone, and neither does it provoke any unwanted stuff. I’ve seen some speculations on the web, that System Update was somehow related to Google Play, and given my experience, I don’t think this is the case.
So disabling the Activities did the trick. It’s also possible to see exactly what happened by looking at the output of
$ adb shell logcat -d > all-log.txt
where it says this somewhere around the time it should have started messing around:
08-22 18:39:19.063 1461 1972 I ActivityTaskManager: START u0 {act=android.settings.SYSTEM_UPDATE_COMPLETE flg=0x10048000 pkg=com.google.android.gms (has extras)} from uid 10146
--------- beginning of crash
08-22 18:39:19.069 2838 7279 E AndroidRuntime: FATAL EXCEPTION: [com.google.android.gms.chimera.container.intentoperation.GmsIntentOperationChimeraService-Executor] idle
08-22 18:39:19.069 2838 7279 E AndroidRuntime: Process: com.google.android.gms, PID: 2838
08-22 18:39:19.069 2838 7279 E AndroidRuntime: android.content.ActivityNotFoundException: No Activity found to handle Intent { act=android.settings.SYSTEM_UPDATE_COMPLETE flg=0x10048000 pkg=com.google.android.gms (has extras) }
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at android.app.Instrumentation.checkStartActivityResult(Instrumentation.java:2087)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at android.app.Instrumentation.execStartActivity(Instrumentation.java:1747)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at android.app.ContextImpl.startActivity(ContextImpl.java:1085)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at android.app.ContextImpl.startActivity(ContextImpl.java:1056)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at android.content.ContextWrapper.startActivity(ContextWrapper.java:411)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at com.google.android.gms.update.reminder.UpdateReminderDialogIntentOperation.a(:com.google.android.gms@222615044@22.26.15 (190400-461192076):67)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at com.google.android.gms.update.reminder.UpdateReminderDialogIntentOperation.onHandleIntent(:com.google.android.gms@222615044@22.26.15 (190400-461192076):10)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at com.google.android.chimera.IntentOperation.onHandleIntent(:com.google.android.gms@222615044@22.26.15 (190400-461192076):2)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at uzr.onHandleIntent(:com.google.android.gms@222615044@22.26.15 (190400-461192076):4)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at ffi.run(:com.google.android.gms@222615044@22.26.15 (190400-461192076):3)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at ffh.run(:com.google.android.gms@222615044@22.26.15 (190400-461192076):11)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at cfvf.run(:com.google.android.gms@222615044@22.26.15 (190400-461192076):2)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1137)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:637)
08-22 18:39:19.069 2838 7279 E AndroidRuntime: at java.lang.Thread.run(Thread.java:1012)
08-22 18:39:19.092 1461 4210 I DropBoxManagerService: add tag=system_app_crash isTagEnabled=true flags=0x2
08-22 18:39:19.094 1461 1552 W BroadcastQueue: Background execution not allowed: receiving Intent { act=android.intent.action.DROPBOX_ENTRY_ADDED flg=0x10 (has extras) } to com.google.android.gms/.stats.service.DropBoxEntryAddedReceiver
08-22 18:39:19.094 1461 1552 W BroadcastQueue: Background execution not allowed: receiving Intent { act=android.intent.action.DROPBOX_ENTRY_ADDED flg=0x10 (has extras) } to com.google.android.gms/.chimera.GmsIntentOperationService$PersistentTrustedReceiver
08-22 18:39:19.270 1461 4215 I ActivityManager: Process com.google.android.gms (pid 2838) has died: fg SVC
08-22 18:39:19.271 1461 4215 W ActivityManager: Scheduling restart of crashed service com.google.android.gms/.chimera.GmsIntentOperationService in 1000ms for start-requested
08-22 18:39:20.273 1461 1552 D CompatibilityChangeReporter: Compat change id reported: 135634846; UID 10146; state: DISABLED
08-22 18:39:20.274 1461 1552 D CompatibilityChangeReporter: Compat change id reported: 177438394; UID 10146; state: DISABLED
08-22 18:39:20.274 1461 1552 D CompatibilityChangeReporter: Compat change id reported: 135772972; UID 10146; state: DISABLED
08-22 18:39:20.274 1461 1552 D CompatibilityChangeReporter: Compat change id reported: 135754954; UID 10146; state: ENABLED
08-22 18:39:20.275 1461 1553 D CompatibilityChangeReporter: Compat change id reported: 143937733; UID 10146; state: ENABLED
08-22 18:39:20.296 1461 1553 I ActivityManager: Start proc 7305:com.google.android.gms/u0a146 for service {com.google.android.gms/com.google.android.gms.chimera.GmsIntentOperationService}
Clearly, disabling the Activities made them ineligible for handling the SYSTEM_UPDATE_COMPLETE Intent, so an ActivityNotFoundException was thrown. Surprisingly enough, this exception wasn’t caught, so the com.google.android.gms process simply died and was quickly restarted by the system.
I also found these later on:
08-24 07:20:00.085 16126 25006 I SystemUpdate: [Execution,InstallationIntentOperation] Received intent: Intent { act=com.google.android.gms.update.INSTALL_UPDATE cat=[targeted_intent_op_prefix:.update.execution.InstallationIntentOperation] pkg=com.google.android.gms cmp=com.google.android.gms/.chimera.GmsIntentOperationService }.
08-24 07:20:00.114 16126 25006 W GmsTaskScheduler: com.google.android.gms.update.SystemUpdateGcmTaskService is not available. This may cause the task to be lost.
08-24 07:20:00.118 16126 25006 W GmsTaskScheduler: com.google.android.gms.update.SystemUpdateGcmTaskService is not available. This may cause the task to be lost.
08-24 07:20:00.119 16126 25006 W GmsTaskScheduler: com.google.android.gms.update.SystemUpdateGcmTaskService is not available. This may cause the task to be lost.
[ ... ]
08-24 07:20:00.198 16126 25006 I SystemUpdate: [Control,InstallationControl] Installation progress updated to (0x413, -1.000).
08-24 07:20:00.273 16126 25006 I SystemUpdate: [Control,ChimeraGcmTaskService] Scheduling task: DeviceIdle.
08-24 07:20:00.273 16126 25006 W GmsTaskScheduler: com.google.android.gms.update.SystemUpdateGcmTaskService is not available. This may cause the task to be lost.
08-24 07:20:00.274 16126 25006 I SystemUpdate: [Execution,ExecutionManager] Action streaming-apply executed for 0.17 seconds.
08-24 07:20:00.283 16126 25006 I SystemUpdate: [Execution,ExecutionManager] Action fixed-delay-execution executed for 0.01 seconds.
Bottom line: Disabling the activity causes the GMS service to die and restart every time it thinks about updating the system and/or nagging about it. So it won’t happen. Mission accomplished.
Actually, I’m not sure it’s possible to update the phone now, even if I wanted to.
It wasn’t obvious that this would work. Disabling the activity could have canceled just the GUI part of the update process. But it did.
It would have been more elegant to add another package, that supplies an Activity for the SYSTEM_UPDATE_COMPLETE Intent, that runs instead of the original, disabled one. This would have avoided this recurring crash. I don’t know if this is possible though.
Or even better, to disable the recurring call to this Intent. Ideas are welcome.
Or did it really work?
A couple of months later, and I noted something that looked like a huge download. Using “top” on “adb shell” revealed that the update_engine consumed some 60% CPU. Using logcat as above revealed several entries like these:
10-29 20:43:38.677 18745 24156 I SystemUpdate: [Control,InstallationControl] Installation progress updated to (0x111, 0.704).
10-29 20:43:38.717 18745 24156 I SystemUpdate: [Control,InstallationControl] Update engine status updated to 0x003.
10-29 20:43:38.832 1106 1106 I update_engine: [INFO:delta_performer.cc(115)] Completed 1691/2292 operations (73%), 1429588024/2027780609 bytes downloaded (70%), overall progress 71%
So there is definitely some download activity going on. The question is what will happen when it reaches 100%. And even more important, if there’s a way to prevent this from happening.
And maybe I’m just a bit too uptight. Using (as root)
# find /mnt/installer/0/emulated/ -cmin -1 2>/dev/null
to find newly update files, it appears like the activity is in /mnt/installer/0/emulated/0/Android/data/com.google.android.googlequicksearchbox/files/download_cache, and that the downloaded files are like soda-en-US-v7025-3.zip, which appears to be Speech On Device Access. And I’m fine with that.
Or did it really work II?
I owe this part to Namelesswonder’s very useful comment below. Indeed, the logs are at /data/misc/update_engine_log/, and there are indeed some files there from October, which is when there was some update activity. The progress of the download, in percents and bytes, is written in these logs.
The first log has an interesting row:
[1003/115142.712102] [INFO:main.cc(55)] A/B Update Engine starting
[1003/115142.720019] [INFO:boot_control_android.cc(68)] Loaded boot control hidl hal.
[1003/115142.738330] [INFO:update_attempter_android.cc(1070)] Scheduling CleanupPreviousUpdateAction.
[1003/115142.747361] [INFO:action_processor.cc(51)] ActionProcessor: starting CleanupPreviousUpdateAction
Any idea how to schedule CleanupPreviousUpdateAction manually? Sounds like fun to me.
The logs also name the URL of the downloaded package, https://ota.googlezip.net/packages/ota-api/package/d895ce906129e5138db6141ec735740cd1cd1b07.zip which is about 1.9 GB, so it’s definitely a full-blown update.
What is even more interesting from the logs is this line, which appears every time before starting or resuming an OTA download:
[1029/201649.018352] [INFO:delta_performer.cc(1009)] Verifying using certificates: /system/etc/security/otacerts.zip
It’s a ZIP file, which contains a single, bare-bone self-signed certificate in PEM format, which contains no fancy information.
It seems quite obvious that this certificate is used to authenticate the update file. What happens if this file is absent? No authentication, no update. Probably no download. So the immediate idea is to rename this file to something else. But it’s not that easy:
# mv otacerts.zip renamed-otacerts.zip
mv: bad 'otacerts.zip': Read-only file system
That’s true. /system/etc/security/ is on the root file system, and that happens to be a read-only one. But it’s a good candidate for a Magisk override, I guess.
Another thing that was mentioned in the said comment, is that the data goes to /data/ota_package/. That’s interesting, because in my system this directory is nearly empty, but the files’ modification timestamp is slightly before I neutralized the update activities, as mentioned above.
So it appears like the downloaded data goes directly into a raw partition, rather than a file.
There’s also /data/misc/update_engine/prefs/, which contains the current status of the download and other metadata. For example, update-state-next-data-offset apparently says how much has been downloaded already. What happens if this directory is nonexistent? Is it recreated, or is this too much for the updater to take?
As an initial approach, I renamed the “prefs” directory to “renamed-prefs”. A few days later, a new “prefs” directory was created, with just boot-id, previous-slot and previous-version files. Their timestamp matched a new log entry in update_engine_log/, which consisted of a CleanupPreviousUpdateAction. So apparently, the previous download was aborted and restarted.
So this time I renamed update_engine into something else, and added update_engine as a plain file. Before doing this, I tried to make the directory unreadable, but as root it’s accessible anyhow (in Android, not a normal Linux system). So the idea is to make it a non-directory, and maybe that will cause an irrecoverable failure.
Update 8.12.22: Nope, that didn’t help. A new log entry appeared in update_engine_log/, not a single word about the failure to create update_engine/prefs/, but a whole lot of rows indicating the download’s progress. /data/misc/update_engine remained as a plain file, so nothing could be stored in the correct place for the prefs/ subdirectory. I failed to find files with the names of those that are usually in prefs/ anywhere in the filesystem, so I wonder how the download will resume after it’s interrupted.
These are a few unrelated topics, which didn’t turn out helpful. So they’re left here, just in case.
Downloading the source
I had the idea of grabbing the sources for the GMS package, and see what went on there.
The list of Git repos is here (or the GitHub mirror here). I went for the this one:
git clone https://android.googlesource.com/platform/frameworks/base
or the GitHub mirror:
git clone https://github.com/aosp-mirror/platform_frameworks_base.git
however this doesn’t seem to include the relevant parts.
Permissions
Another approach I had in mind was to turn off the permission to make a system update. I don’t know if there actually is such.
Android’s permission system allows granting and revoking permissions as required. Just typing “pm” (package manager) on adb shell returns help information
To get a list of all permissions in the system:
$ adb shell pm list permissions -g > list.txt
But that covers everything, so the trick is to search for the appearances of the specific package of interest. Something like
$ grep com.google.android.gms list.txt | sort | less
and look at that list. I didn’t anything that appears to related to system update.
Introduction
Due to an incident that is beyond the scope of this blog, I wanted to put a 24/7 camera that watched a certain something, just in case that incident repeated itself.
Having a laptop that I barely use, and a cheap e-bay web camera, I thought I set up something and let ffmpeg do the job.
I’m not sure if a Raspberry Pi would be up for this job, even when connected to an external hard disk through USB. It depends much on how well ffmpeg performs on that platform. Haven’t tried. The laptop’s clear advantage is when there’s a brief power outage.
Overall verdict: It’s as good as the stability of the USB connection with the camera.
Note to self: I keep this in the misc/utils git repo, under surveillance-cam/.
Warming up
Show the webcam’s image on screen, the ffmpeg way:
$ ffplay -f video4linux2 /dev/video0
Let ffmpeg list the formats:
$ ffplay -f video4linux2 -list_formats all /dev/video0
Or with a dedicated tool:
# apt install v4l-utils
and then
$ v4l2-ctl --list-formats-ext -d /dev/video0
Possibly also use “lsusb -v” on the device: It lists the format information, not necessarily in a user-friendly way, but that’s the actual source of information.
Get all parameters that can be tweaked:
$ v4l2-ctl --all
See an example output for this command at the bottom of this post.
If control over the exposure time is available, it will be listed as “exposure_absolute” (none of the webcams I tried had this). The exposure time is given in units of 100µs (see e.g. the definition of V4L2_CID_EXPOSURE_ABSOLUTE).
Get a specific parameter, e.g. brightness
$ v4l2-ctl --get-ctrl=brightness
brightness: 137
Set the control (can be done while the camera is capturing video)
$ v4l2-ctl --set-ctrl=brightness=255
Continuous capturing
This is a simple bash script that creates .mp4 files from the captured video:
#!/bin/bash
OUTDIR=/extra/videos SRC=/dev/v4l/by-id/usb-Generic*
DURATION=3600
while [ 1 ]; do
TIME=`date +%F-%H%M%S`
if ! ffmpeg -f video4linux2 -i $SRC -t $DURATION -r 10 $OUTDIR/video-$TIME.mp4 < /dev/null ; then
echo 2-2 | sudo tee /sys/bus/usb/drivers/usb/unbind
echo 2-2 | sudo tee /sys/bus/usb/drivers/usb/bind
sleep 5;
fi
done
Comments on the script:
- To make this a real surveillance application, there must be another script that deletes old files, so that the disk isn’t full. My script on this matter is so hacky, that I left it out here.
- The real problem I encountered was occasional USB errors. They happened every now and then, without any specific pattern. Sometimes the camera disconnected briefly and reconnected right away, sometimes it failed to come back for a few minutes. Once in a week or so, it didn’t come back at all, and only a lot of USB errors appeared in the kernel log, so a reboot was required. This is most likely some kind of combination of cheap hardware, a long and not so good USB cable and maybe hardware + kernel driver issues. I don’t know. This wasn’t important enough to solve in a bulletproof way.
- Because of these USB errors, those two “echo 2-2″ commands attempt to reset the USB port if ffmpeg fails, and then sleep 5 seconds. The “2-2″ is the physical position of the USB port to which the USB camera was connected. Ugly hardcoding, yes. I know for sure that these commands were called occasionally, but whether this helped, I’m not sure.
- Also because of these disconnections, the length of the videos wasn’t always 60 minutes as requested. But this doesn’t matter all that much, as long as the time between the clips is short. Which it usually was (less than 5 seconds, the result of a brief disconnection).
- Note that the device file for the camera is found using a /dev/v4l/by-id/ path rather than /dev/video0, not just to avoid mixing between the external and built-in webcam: There were sporadic USB disconnections after which the external webcam ended up as /dev/video2. And then back to /dev/video1 after the next disconnection. The by-id path remained constant in the sense that it could be found with the * wildcard.
- Frame rate is always a dilemma, as it ends up influencing the file’s size, and hence how long back videos are stored. At 5 fps, an hour long .mp4 took about 800 MB for daytime footage, and much less than so during night. At 10 fps, it got up to 1.1 GB, so by all means, 10 fps is better.
- Run the recording on a text console, rather than inside a terminal window inside X-Windows (i.e. use Ctrl-Alt-F1 and Ctrl-Alt-F7 to go back to X). This is because the graphical desktop crashed at some point — see below on why. So if this happens again, the recording will keep going.
- For the purpose of running ffmpeg without a console (i.e. run in the background with an “&” and then log out), note that the ffmpeg command has a “< /dev/null”. Otherwise ffmpeg expects to be interactive, meaning it does nothing if it runs in the background. There’s supposed to be a -nostdin flag for this, and ffmpeg recognized it on my machine, but expected a console nevertheless. So I went for the old method.
How a wobbling USB camera crashes X-Windows
First, the spoiler: I solved this problem by putting a physical weight on the USB cable, close to the plug. This held the connector steady in place, and the vast majority of the problems were gone.
I also have a separate post about how I tried to make Linux ignore the offending bogus keyboard from being. Needless to say, that failed (because either you ban the entire USB device or you don’t ban at all).
This is the smoking gun in /var/log/Xorg.0.log: Lots of
[1194182.076] (II) config/udev: Adding input device USB2.0 PC CAMERA: USB2.0 PC CAM (/dev/input/event421)
[1194182.076] (**) USB2.0 PC CAMERA: USB2.0 PC CAM: Applying InputClass "evdev keyboard catchall"
[1194182.076] (II) Using input driver 'evdev' for 'USB2.0 PC CAMERA: USB2.0 PC CAM'
[1194182.076] (**) USB2.0 PC CAMERA: USB2.0 PC CAM: always reports core events
[1194182.076] (**) evdev: USB2.0 PC CAMERA: USB2.0 PC CAM: Device: "/dev/input/event421"
[1194182.076] (--) evdev: USB2.0 PC CAMERA: USB2.0 PC CAM: Vendor 0x1908 Product 0x2311
[1194182.076] (--) evdev: USB2.0 PC CAMERA: USB2.0 PC CAM: Found keys
[1194182.076] (II) evdev: USB2.0 PC CAMERA: USB2.0 PC CAM: Configuring as keyboard
[1194182.076] (EE) Too many input devices. Ignoring USB2.0 PC CAMERA: USB2.0 PC CAM
[1194182.076] (II) UnloadModule: "evdev"
and at some point the sad end:
[1194192.408] (II) config/udev: Adding input device USB2.0 PC CAMERA: USB2.0 PC CAM (/dev/input/event423)
[1194192.408] (**) USB2.0 PC CAMERA: USB2.0 PC CAM: Applying InputClass "evdev keyboard catchall"
[1194192.408] (II) Using input driver 'evdev' for 'USB2.0 PC CAMERA: USB2.0 PC CAM'
[1194192.408] (**) USB2.0 PC CAMERA: USB2.0 PC CAM: always reports core events
[1194192.408] (**) evdev: USB2.0 PC CAMERA: USB2.0 PC CAM: Device: "/dev/input/event423"
[1194192.445] (EE)
[1194192.445] (EE) Backtrace:
[1194192.445] (EE) 0: /usr/bin/X (xorg_backtrace+0x48) [0x564128416d28]
[1194192.445] (EE) 1: /usr/bin/X (0x56412826e000+0x1aca19) [0x56412841aa19]
[1194192.445] (EE) 2: /lib/x86_64-linux-gnu/libpthread.so.0 (0x7f6e4d8b4000+0x10340) [0x7f6e4d8c4340]
[1194192.445] (EE) 3: /usr/lib/xorg/modules/input/evdev_drv.so (0x7f6e45c4c000+0x39f5) [0x7f6e45c4f9f5]
[1194192.445] (EE) 4: /usr/lib/xorg/modules/input/evdev_drv.so (0x7f6e45c4c000+0x68df) [0x7f6e45c528df]
[1194192.445] (EE) 5: /usr/bin/X (0x56412826e000+0xa1721) [0x56412830f721]
[1194192.446] (EE) 6: /usr/bin/X (0x56412826e000+0xb731b) [0x56412832531b]
[1194192.446] (EE) 7: /usr/bin/X (0x56412826e000+0xb7658) [0x564128325658]
[1194192.446] (EE) 8: /usr/bin/X (WakeupHandler+0x6d) [0x5641282c839d]
[1194192.446] (EE) 9: /usr/bin/X (WaitForSomething+0x1bf) [0x5641284142df]
[1194192.446] (EE) 10: /usr/bin/X (0x56412826e000+0x55771) [0x5641282c3771]
[1194192.446] (EE) 11: /usr/bin/X (0x56412826e000+0x598aa) [0x5641282c78aa]
[1194192.446] (EE) 12: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0xf5) [0x7f6e4c2f3ec5]
[1194192.446] (EE) 13: /usr/bin/X (0x56412826e000+0x44dde) [0x5641282b2dde]
[1194192.446] (EE)
[1194192.446] (EE) Segmentation fault at address 0x10200000adb
[1194192.446] (EE)
Fatal server error:
[1194192.446] (EE) Caught signal 11 (Segmentation fault). Server aborting
[1194192.446] (EE)
The thing is that webcam presents itself as a keyboard, among others. I guess the chipset has inputs for control buttons (which the specific webcam doesn’t have), so as the USB device goes on and off, X windows registers the nonexistent keyboard on and off, and eventually some bug causes it to crash (note that number of the event device is 423, so there were quite a few on and offs). It might very well be that the camera camera connected, started some kind of connection event handler, which didn’t finish its job before it disconnected. Somewhere in the code, the handler fetched information that didn’t exist, it got a bad pointer instead (NULL?) and used it. Boom. Just a wild guess, but this is the typical scenario.
The crash can be avoided by making X windows ignore this “keyboard”. I did this by adding a new file named /usr/share/X11/xorg.conf.d/10-nocamera.conf as follows:
# Ignore bogus button on webcam
Section "InputClass"
Identifier "Blacklist USB webcam button as keyboard"
MatchUSBID "1908:2311"
Option "Ignore" "on"
EndSection
This way, X windows didn’t fiddle with the bogus buttons, and hence didn’t care if they suddenly went away.
Anyhow, it’s a really old OS (Ubuntu 14.04.1) so this bug might have been solved long ago.
Accumulation of /dev/input/event files
Another problem with this wobbling is that /dev/input/ becomes crowded with a lot of eventN files:
$ ls /dev/input/event*
/dev/input/event0 /dev/input/event267 /dev/input/event295
/dev/input/event1 /dev/input/event268 /dev/input/event296
/dev/input/event10 /dev/input/event269 /dev/input/event297
/dev/input/event11 /dev/input/event27 /dev/input/event298
/dev/input/event12 /dev/input/event270 /dev/input/event299
/dev/input/event13 /dev/input/event271 /dev/input/event3
/dev/input/event14 /dev/input/event272 /dev/input/event30
/dev/input/event15 /dev/input/event273 /dev/input/event300
/dev/input/event16 /dev/input/event274 /dev/input/event301
/dev/input/event17 /dev/input/event275 /dev/input/event302
/dev/input/event18 /dev/input/event276 /dev/input/event303
/dev/input/event19 /dev/input/event277 /dev/input/event304
/dev/input/event2 /dev/input/event278 /dev/input/event305
/dev/input/event20 /dev/input/event279 /dev/input/event306
/dev/input/event21 /dev/input/event28 /dev/input/event307
/dev/input/event22 /dev/input/event280 /dev/input/event308
/dev/input/event23 /dev/input/event281 /dev/input/event309
/dev/input/event24 /dev/input/event282 /dev/input/event31
/dev/input/event25 /dev/input/event283 /dev/input/event310
/dev/input/event256 /dev/input/event284 /dev/input/event311
/dev/input/event257 /dev/input/event285 /dev/input/event312
/dev/input/event258 /dev/input/event286 /dev/input/event313
/dev/input/event259 /dev/input/event287 /dev/input/event314
/dev/input/event26 /dev/input/event288 /dev/input/event315
/dev/input/event260 /dev/input/event289 /dev/input/event316
/dev/input/event261 /dev/input/event29 /dev/input/event4
/dev/input/event262 /dev/input/event290 /dev/input/event5
/dev/input/event263 /dev/input/event291 /dev/input/event6
/dev/input/event264 /dev/input/event292 /dev/input/event7
/dev/input/event265 /dev/input/event293 /dev/input/event8
/dev/input/event266 /dev/input/event294 /dev/input/event9
Cute, huh? And this is even before there was a problem. So what does X windows make of this?
$ xinput list
⎡ Virtual core pointer id=2 [master pointer (3)]
⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)]
⎜ ↳ ELAN Touchscreen id=9 [slave pointer (2)]
⎜ ↳ SynPS/2 Synaptics TouchPad id=13 [slave pointer (2)]
⎣ Virtual core keyboard id=3 [master keyboard (2)]
↳ Virtual core XTEST keyboard id=5 [slave keyboard (3)]
↳ Power Button id=6 [slave keyboard (3)]
↳ Video Bus id=7 [slave keyboard (3)]
↳ Power Button id=8 [slave keyboard (3)]
↳ Lenovo EasyCamera: Lenovo EasyC id=10 [slave keyboard (3)]
↳ Ideapad extra buttons id=11 [slave keyboard (3)]
↳ AT Translated Set 2 keyboard id=12 [slave keyboard (3)]
↳ USB 2.0 PC Cam id=14 [slave keyboard (3)]
↳ USB 2.0 PC Cam id=15 [slave keyboard (3)]
↳ USB 2.0 PC Cam id=16 [slave keyboard (3)]
↳ USB 2.0 PC Cam id=17 [slave keyboard (3)]
↳ USB 2.0 PC Cam id=18 [slave keyboard (3)]
↳ USB 2.0 PC Cam id=19 [slave keyboard (3)]
Now, let me assure you that there were not six webcams connected when I did this. Actually, not a single one.
Anyhow, I didn’t dig further into this. The real problem is that all of these /dev/input/event files have the same major. Which means that when there are really a lot of them, the system runs out of minors. So if the normal kernel log for plugging in the webcam was this,
usb 2-2: new high-speed USB device number 22 using xhci_hcd
usb 2-2: New USB device found, idVendor=1908, idProduct=2311
usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 2-2: Product: USB2.0 PC CAMERA
usb 2-2: Manufacturer: Generic
uvcvideo: Found UVC 1.00 device USB2.0 PC CAMERA (1908:2311)
uvcvideo 2-2:1.0: Entity type for entity Processing 2 was not initialized!
uvcvideo 2-2:1.0: Entity type for entity Camera 1 was not initialized!
input: USB2.0 PC CAMERA: USB2.0 PC CAM as /devices/pci0000:00/0000:00:14.0/usb2/2-2/2-2:1.0/input/input274
after all minors ran out, I got this:
usb 2-2: new high-speed USB device number 24 using xhci_hcd
usb 2-2: New USB device found, idVendor=1908, idProduct=2311
usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 2-2: Product: USB2.0 PC CAMERA
usb 2-2: Manufacturer: Generic
uvcvideo: Found UVC 1.00 device USB2.0 PC CAMERA (1908:2311)
uvcvideo 2-2:1.0: Entity type for entity Processing 2 was not initialized!
uvcvideo 2-2:1.0: Entity type for entity Camera 1 was not initialized!
media: could not get a free minor
And then immediately after:
systemd-udevd[4487]: Failed to apply ACL on /dev/video2: No such file or directory
systemd-udevd[4487]: Failed to apply ACL on /dev/video2: No such file or directory
Why these eventN files aren’t removed is unclear. The kernel is pretty old, v4.14, so maybe this has been fixed since.
Sample output of v412-all
This is small & junky webcam. Clearly no control over exposure time.
$ v4l2-ctl --all -d /dev/v4l/by-id/usb-Generic_USB2.0_PC_CAMERA-video-index0
Driver Info (not using libv4l2):
Driver name : uvcvideo
Card type : USB2.0 PC CAMERA: USB2.0 PC CAM
Bus info : usb-0000:00:14.0-2
Driver version: 4.14.0
Capabilities : 0x84200001
Video Capture
Streaming
Device Capabilities
Device Caps : 0x04200001
Video Capture
Streaming
Priority: 2
Video input : 0 (Camera 1: ok)
Format Video Capture:
Width/Height : 640/480
Pixel Format : 'YUYV'
Field : None
Bytes per Line: 1280
Size Image : 614400
Colorspace : Unknown (00000000)
Custom Info : feedcafe
Crop Capability Video Capture:
Bounds : Left 0, Top 0, Width 640, Height 480
Default : Left 0, Top 0, Width 640, Height 480
Pixel Aspect: 1/1
Selection: crop_default, Left 0, Top 0, Width 640, Height 480
Selection: crop_bounds, Left 0, Top 0, Width 640, Height 480
Streaming Parameters Video Capture:
Capabilities : timeperframe
Frames per second: 30.000 (30/1)
Read buffers : 0
brightness (int) : min=0 max=255 step=1 default=128 value=128
contrast (int) : min=0 max=255 step=1 default=130 value=130
saturation (int) : min=0 max=255 step=1 default=64 value=64
hue (int) : min=-127 max=127 step=1 default=0 value=0
gamma (int) : min=1 max=8 step=1 default=4 value=4
power_line_frequency (menu) : min=0 max=2 default=1 value=1
sharpness (int) : min=0 max=15 step=1 default=13 value=13
backlight_compensation (int) : min=1 max=5 step=1 default=1 value=1
Introduction
I bought a Google Pixel 6 Pro (P6P henceforth) a few months ago, and as always, I write down what I do as I do it. The result is below. I hope this will never be useful for myself, because if it does, it means I had the phone completely reset, and not I’m starting from scratch.
I’m not a great fan of mobile phones in general, and even less playing around with upgrading and setting them up. I know that to a lot of people, that is a bit of a technology kick, but being an Electrical Engineer, I don’t need another toy to play with.
So I had my previous smartphone for six year, and it ran Android 5.5 to its last day. Jumping to Android 12 is quite a leap, and it’s quite possible that things that I write about below are related to this jump, rather than the Pixel itself.
Generally speaking, the user interface has taken a sharp turn towards sweet talking, and I can’t say I like it very much. It’s clearly aimed for plain people (to put it gently). Everything is automatic and “helpful”, but it’s difficult to figure out how to get control over the machine. In short, annoying.
Another thing is that the software is designed to have Wifi available most of the time. Or at least a very good Internet connection over the cellular network. Everything is backed up to the cloud, run on a cloud server and whatnot. Without Internet, the phone is crippled.
Google does however live up to my expectation from them in the sense that there is a way to mute all these annoyances. It just requires knocking them down one by one.
Notes while setting up
The most important thing: If you want to root your phone, do it immediately as you get it, before staring to set it up. As I’ve explained in a separate post on rooting, you will have to reset the phone to root the phone. So all data and setup will be lost.
- Do the initial startup properly and with Wifi present. Alternatively, go to Settings > System > Reset options > Erase all data (factory reset) to start over again. This doesn’t necessarily erase downloaded SIMs.
- There’s a “Pixel Tips” app, which is useful for getting information on what the phone can do. Something to keep me busy in relatively short periods of otherwise wasted time.
- WhatsApp’s backup on Google Cloud is perfectly fine for restoring all conversations. However it happens only on installation, so I had to reinstall the app to get the restoration kicked off. Worked like a charm, though.
- SIM: LG G4 uses a Micro SIM, and P6P is with a Nano SIM or eSIM. The latter is non-removable on the phone, and the idea is to download the information into it. Unfortunately, Partner (my cellular provider) doesn’t support eSIM on phones, so I used a regular SIM. There’s a tool to pick out the SIM tray from the phone in the phone’s kit, but any needle would probably do likewise.
- It’s possible to connect keyboard, mouse and even a USB stick.
- As for the USB stick, it requested to format the driver, and I agreed. Maybe because it had two partitions? Anyhow, it formatted it into plain VFAT, but then I couldn’t access the files nor copy files into it. To be investigated.
- Tap on the home screen (that is, on the wallpaper), pick “Home Settings” and disable “Swipe to access Google app” or else a silly news feed appears from the home screen.
- At an early stage, enter Google Photos with no network connectivity and tap the icon at the top right. Set the app to work without an account. Otherwise, all junk that has been uploaded into the Photos account appears in the app.
- I opted out Chat Features. SMS is old, and I don’t want any hassle with that.
- Also opted out: Settings > Internet > Wi-fi > Network preferences > Notify for public networks (I don’t want the phone to look for them at all).
- As for the ringtone, nothing like the LG’s good old Little Spheres.
- I initially disallowed apps to scan for Wifi, even when Wifi is turned off: Settings > Location > Location services > Wi-Fi scanning. However it turns out that some apps (Facebook Messenger, for example) fail to start in this situation. So I turned it on again.
- Turn off Assistant’s voice response: Settings > Apps > Default Apps > Digital assistant app > settings icon to top right (flywheel) > Spoken results. Hands-free searches only.
- Apps to uninstall: Google One (no backups, thanks).
- First prize for unnecessary notification: Need time to focus? Open Focus mode to pause distracting apps. So I opened the Settings > Notifications > App settings and turn off notification for Digital Wellbeing. Digital wellbeing is, among others, not being bothered by silly notifications.
- Improving the fingerprint detection I: Settings > Display > Increase Touch Sensitivity. Doesn’t help.
- Improving the fingerprint detection II: Press firmly, in particular in the training session. That gives the sensor more area to work with. It also appears like using different fingers each time doesn’t help.
- Turn off autocorrect (correction while typing on keyboard). It’s better to have a lot of typos than a text that is perfectly spelled but meaning something else than intended.
- Turn off revocation and removal of temporary files of apps that aren’t used for 90 days.
Disable right edge swipe for “back”
The point is to disable the feature that a swipe from the right means “back”. Swiping from the left edge is enough for me, thanks.
The official suggestion is going to Settings > System > Gestures > System navigation > Gesture Settings, set Back Sensitivity for Right Edge to Low. But that merely reduces the sensitivity, not turning it off completely.
There’s the possibility to return to three-button navigation on that menu, but left edge sweep is actually good. It’s the other edge I wanted to get rid of.
So use adb shell to hack the setting, as suggested here:
$ adb shell
raven:/ $ settings put secure back_gesture_inset_scale_right 0
This worked. Amazing, huh? Power to command-line even on Android.
And rooting the phone is not required for this one. Developer mode and knowing a thing or two with computers, definitely yes.
Copying data from old phone
At some point in the setup process, the phone asked me to connect to my previous phone. So I took the little USB C to USB A adapter that came with the phone, and connected it to the same USB cable that I usually connect with a computer.
Some information was copied to the new phone: Photos and videos (if requested, this is slow), apps, contacts and my SMSes. But no application information: Instagram and Facebook didn’t know which user I was.. Not to mention WhatsApp conversations (but that can be obtained from the cloud backup).
But most important: The Google ID moves along to the new phone.
Prepare for a long session of app downloads, installations and upgrades after this copying.
Note that this copying can’t be repeated later on, unless the phone is reset per request.
“Room for improvement”
- There’s apparently a need to reboot the phone before using Waze, or else the voice instructions are missing or partial. I suppose that will fix itself somehow over time with a Waze upgrade.
- In screensaver mode, the phone usually displays the time with thin and dim numbers. However when a notification is canceled, it may display gibberish or even the wrong time (that is, the displayed time is stuck).
- The fingerprint detection is indeed not good. It works most of the time. I speculate that there’s a suspicion level factor involved, so if the phone thinks the overall situation is somewhat fishy, it becomes extra picky with matching my fingerprint.
Charging
An anecdotal set of measurements of currents, with the Ampere app: Starts at ~4540 mA, drops to ~2150 mA at 50%. But then it may wander towards ~3100 mA at 66%. Eventually it drops to ~1600 mA at 89%. At 98% it’s down at 500 mA.
When connected to a USB 2.0 port, it charges at ~600 mA, but only when the phone is in screensaver mode (probably because there’s not enough current when it’s on).
When connected to a QC fast charger, it ran at 1400 mA (at high charging percentages).
As a lot of people (on the web) have pointed out, the phone is indeed picky with charging cables, and I’m not sure yet what makes it accept a cable to charge rapidly with.
As for USB A to C charging cables, it’s a bit of a lottery. A random cable that I got directly from China (Ebay) connects the phone as USB 3.0, and charges the phone at fairly rapid speed even with QC (it was used in the experiments above).
But then I have two USB 2.0 cables: One is a Wesdar T38 which didn’t charge neither from the computer nor a charger, and a Miracase USB Type C cable, which charges the phone from both. The peculiar thing is that the phone was detected by the computer as a High Speed device (USB 2.0), but there was no charging initiated with the former. And this cable failed to charge with a various of sources I tried, while the latter was successful with all. Go figure.
Introduction
I had some really annoying bots on one of my websites. Of the sort that make a million requests (like really, a million) per month, identifying themselves as a browser.
So IP blocking it is. I went for a minimalistic DIY approach. There are plenty of tools out there, but my experience with things like this is that in the end, it’s me and the scripts. So I might as well write them myself.
The IP set feature
Iptables has an IP set module, which allows feeding it with a set of random IP addresses. Internally, it creates a hash with these addresses, so it’s an efficient way to keep track of multiple addresses.
IP sets has been in the kernel since ages, but it has to be opted in the kernel with CONFIG_IP_SET. Which it most likely is.
The ipset utility may need to be installed, with something like
# apt install ipset
There seems to be a protocol mismatch issue with the kernel, which apparently is a non-issue. But every time something goes wrong with ipset, there’s a warning message about this mismatch, which is misleading. So it looks something like this.
# ipset [ ... something stupid or malformed ... ]
ipset v6.23: Kernel support protocol versions 6-7 while userspace supports protocol versions 6-6
[ ... some error message related to the stupidity ... ]
So the important thing is to be aware of is that odds are that the problem isn’t the version mismatch, but between chair and keyboard.
Hello, world
A quick session
# ipset create testset hash:ip
# ipset add testset 1.2.3.4
# iptables -I INPUT -m set --match-set testset src -j DROP
# ipset del testset 1.2.3.4
Attempting to add an IP address that is already in the list causes a warning, and the address isn’t added. So no need to check if the address is already there. Besides, there the -exist option, which is really great.
List the members of the IP set:
# ipset -L
Timeout
An entry can have a timeout feature, which works exactly as one would expect: The rule vanishes after the timeout expires. The timeout entry in ipset -L counts down.
For this to work, the set must be created with a default timeout attribute. Zero means that timeout is disabled (which I chose as a default in this example).
# ipset create testset hash:ip timeout 0
# ipset add testset 1.2.3.4 timeout 10
The ‘-exist’ flag causes ipset to re-add an existing entry, which also resets its timeout. So this is the way to keep the list fresh.
Don’t put the DROP rule first
It’s tempting to put the DROP rule with –match-set first, because hey, let’s give those intruders the boot right away. But doing that, there might be TCP connections lingering, because the last FIN packet is caught by the firewall as the new rule is added. Given that adding an IP address is the result of a flood of requests, this is a realistic scenario.
The solution is simple: There’s most likely a “state RELATED,ESTABLISHED” rule somewhere in the list. So push it to the top. The rationale is simple: If a connection has begun, don’t chop it in the middle in any case. It’s the first packet that we want killed.
Persistence
The rule in iptables must refer to an existing set. So if the rule that relies on the set is part of the persistent firewall rules, it must be created before the script that brings up iptables runs.
This is easily done by adding a rule file like this as /usr/share/netfilter-persistent/plugins.d/10-ipset
#!/bin/sh
IPSET=/sbin/ipset
SET=mysiteset
case "$1" in
start|restart|reload|force-reload)
$IPSET destroy
$IPSET create $SET hash:ip timeout 0
;;
save)
echo "ipset-persistent: The save option does nothing"
;;
stop|flush)
$IPSET flush $SET
;;
*)
echo "Usage: $0 {start|restart|reload|force-reload|save|flush}" >&2
exit 1
;;
esac
exit 0
The idea is that the index 10 in the file’s name is smaller than the rule that sets up iptables, so it runs first.
This script is a dirty hack, but hey, it works. There’s a small project on this, for those who like to do it properly.
The operating system in question is systemd-based, but this old school style is still in effect.
Maybe block by country?
Since all offending requests came from the same country (cough, cough, China, from more than 4000 different IP addresses) I’m considering to block them in one go. A list of 4000+ IP addresses that I busted in August 2022 with aggressive bots (all from China) can be downloaded as a simple compressed text file.
So the idea is going something like
ipset create foo hash:net
ipset add foo 192.168.0.0/24
ipset add foo 10.1.0.0/16
ipset add foo 192.168.0/24
and download the per-country IP ranges from IP deny. That’s a simple and crude tool for denial by geolocation. The only thing that puts me down a bit is that it’s > 7000 rules, so I wonder if that doesn’t put a load on the server. But what really counts is the number of sizes of submasks, because each submask size has its own hash. So if the list covers all possible sizes, from a full /32 down to say, 16/, there are 17 hashes to look up for each packet arriving.
On the other hand, since the rule should be after the “state RELATED,ESTABLISHED” rule, it only covers SYN packets. And if this whole thing is put as late as possible in the list of rules, it boils down to handling only packets that are intended for the web server’s ports, or those that are going to be dropped anyhow. So compared with the CPU cycles of handling the http request, even 17 hashes isn’t all that much.
The biggest caveat is however if other websites are colocated on the server. It’s one thing to block offending IPs, but blocking a whole country from all sites, that’s a bit too much.
Note to self: In the end, I wrote a little Perl-XS module that says if the IP belongs to a group. Look for byip.pm.
The blacklisting script
The Perl script that performs the blacklisting is crude and inaccurate, but simple. This is the part to tweak and play with, and in particular adapt to each specific website. It’s all about detecting abnormal access.
Truth to be told, I replaced this script with a more sophisticated mechanism pretty much right away on my own system. But what’s really interesting is the calls to ipset.
This script reads through Apache’s access log file, and analyzes each minute in time (as in 60 seconds). In other words, all accesses that have the same timestamp, with the seconds part ignored. Note that the regex part that captures $time in the script ignores the last part of :\d\d.
If the same IP address appears more than 50 times, that address is blacklisted, with a timeout of 86400 seconds (24 hours). Log file that correspond to page requisites and such (images, style files etc.) are skipped for this purpose. Otherwise, it’s easy to reach 50 accesses within a minute with legit web browsing.
There are several imperfections about this script, among others:
- Since it reads through the entire log file each time, it keeps relisting each IP address until the access file is rotated away, and a new one is started. This causes an update of the timeout, so effectively the blacklisting takes place for up to 48 hours.
- Looking in segments of accesses that happen to have the same minute in the timestamp is quite inaccurate regarding which IPs are caught and which aren’t.
The script goes as follows:
use warnings;
use strict;
my $logfile = '/var/log/mysite.com/access.log';
my $limit = 50;
my $timeout = 86400;
open(my $in, "<", $logfile)
or die "Can't open $logfile for read: $!\n";
my $current = '';
my $l;
my %h;
my %blacklist;
while (defined ($l = <$in>)) {
my ($ip, $time, $req) = ($l =~ /^([^ ]+).*?\[(.+?):\d\d[ ].*?\"\w+[ ]+([^\"]+)/);
unless (defined $ip) {
next;
}
next
if ($req =~ /^\/(?:media\/|robots\.txt)/);
unless ($time eq $current) {
foreach my $k (sort keys %h) {
$blacklist{$k} = 1
if ($h{$k} >= $limit);
}
%h = ();
$current = $time;
}
$h{$ip}++;
}
close $in;
foreach my $k (sort keys %blacklist) {
system('/sbin/ipset', 'add', '-exist', 'mysiteset', $k, 'timeout', $timeout);
}
It has to be run as root, of course. Most likely as a cronjob.
Introduction
These post contains a few technical notes of using Google Translate for translating LaTeX documents into Chinese, Japanese and Korean. The insights on the language-related issues are written down in a separate post.
Text vs. HTML
Google’s cloud translator can be fed with either plain text or HTML, and it returns the same format. Plain text format is out of the question for anything but translating short sentences, as it becomes impossible to maintain the text’s formatting. So I went for the HTML interface.
The thing with HTML is that whitespaces can take different forms and shapes, and they are redundant in many situations. For example, a newline is often equivalent to a plain space, and neither make any difference between two paragraphs that are enclosed by <p> tags.
Google Translate takes this notion to the extreme, and typically removes all newlines from the original text. OK, that’s understandable. But it also adds and removes whitespaces where it had no business doing anything, in particular around meaningless segments that aren’t translated anyhow. This makes it quite challenging when feeding the results for further automatic processing.
Setting up a Google Cloud account
When creating a new Google Cloud account, there’s an automatic credit of $300 to spend for three months. So there’s plenty of room for much needed experimenting. Too see the status of the evaluation period, go to Billing > Cost Breakdown and wait a minute or so for the “Free trial status” strip to appear at the top of the page. There’s no problem with “activating full account” immediately. The free trial credits remain, but it also means that real billing occurs when the credits are consumed and/or the trial period is over.
First create a new Google cloud account and enable the Google Translate API.
I went for Basic v2 translation (and not Advanced, v3). Their pricing is the same, but v3 is not allowed with an API key, and I really wasn’t into setting up a service account and struggle with OAuth2. The main advantage with v3 is the possibility to train the machine to adapt to a specific language pattern, but as mentioned in that separate post, I’m hiding away anything but common English language patterns.
As for authentication, I went for API keys. I don’t need any personalized info, so that’s the simple way to go. To obtain the keys, go to main menu (hamburger icon) > APIs and services > Credentials and pick Create Credentials, and choose to create API keys. Copy the string and use it in the key=API_KEY parameters in POST requests. It’s possible to restrict the usage of this key in various ways (HTTP referrer, IP address etc.) but it wasn’t relevant in my case, because the script runs only on my computer.
The web interface for setting up cloud services is horribly slow, which is slightly ironic and a bit odd for a company like Google.
The translation script
I wrote a simple script for taking a piece of text in English and translating it into the language of choice:
use warnings;
use strict;
use LWP::UserAgent;
use JSON qw[ from_json ];
our $WASTEMONEY = 0;
my $MAXLEN = 500000;
my $chars_per_dollar = 50000;
our $APIkey = 'your API key here';
my ($outfile, $origfile, $lang) = @ARGV;
die("Usage: $0 outfile origfile langcode\n")
unless (defined $origfile);
my $input = readfile($origfile);
askuser() unless ($WASTEMONEY);
my $len = length $input;
die("Cowardly refusing to translate $len characters\n")
if ($len > $MAXLEN);
writefile($outfile, translate($input, $lang));
sub writefile {
my ($fname, $data) = @_;
open(my $out, ">", $fname)
or die "Can't open \"$fname\" for write: $!\n";
binmode($out, ":utf8");
print $out $data;
close $out;
}
sub readfile {
my ($fname) = @_;
local $/;
open(my $in, "<", $fname)
or die "Can't open $fname for read: $!\n";
my $input = <$in>;
close $in;
return $input;
}
sub askuser {
my $len = length $input;
my $cost = sprintf('$%.02f', $len / $chars_per_dollar);
print "\n\n*** Approval to access Google Translate ***\n";
print "$len bytes to $lang, $cost\n";
print "Source file: $origfile\n";
print "Proceed? [y/N] ";
my $ans = <STDIN>;
die("Aborted due to lack of consent to proceed\n")
unless ($ans =~ /^y/i);
}
sub translate {
my ($text, $lang) = @_;
my $ua = LWP::UserAgent->new;
my $url = 'https://translation.googleapis.com/language/translate/v2';
my $res = $ua->post($url,
[
source => 'en',
target => $lang,
format => 'html',
key => $APIkey,
q => $text,
]);
die("Failed to access server: ". $res->status_line . "\n")
unless ($res->is_success);
my $data = $res->content;
my $json = from_json($data, { utf8 => 1 } );
my $translated;
eval {
my $d = $json->{data};
die("Missing \"data\" entry\n") unless (defined $d);
my $tr = $d->{translations};
die("Missing \"translations\" entry\n")
unless ((defined $tr) && (ref $tr eq 'ARRAY') &&
(ref $tr->[0] eq 'HASH'));
$translated = $tr->[0]->{translatedText};
die("No translated text\n")
unless (defined $translated);
};
die("Malformed response from server: $@\n") if ($@);
$translated =~ s/(<\/(?:p|h\d+)>)[ \t\n\r]*/"$1\n"/ge;
return $translated;
}
The substitution at the end of the translate() function adds a newline after each closing tag for a paragraph or header (e.g. </p>, <h1> etc.) so that the HTML is more readable with a text editor. Otherwise it’s all in one single line.
Protecting your money
By obtaining an API key, you effectively give your computer permission to spend money. Which is fine as long as it works as intended, but a plain bug in a script that leads to an infinite loop or recursion, or maybe just feeding the system with a huge file by mistake, can end up with consequences that are well beyond the CPU fan spinning a bit.
So there are two protection mechanisms in the script itself:
- The script prompts for permission, stating how much it will cost (based upon $20 / million chars).
- It limits a single translation to 500k chars (to avoid a huge file from being processed accidentally).
Another safety mechanism is to set up budgets and budget alerts. Go to Main menu (hamburger) > Billing > Budgets & Alerts. Be sure to check “Email alerts to billing admins and users”. If I got it right, budgets don’t protect against spending, but only sends notifications. So I selected a sum, and enabled only the 100% threshold. It seems to make sense to check all the Discounts and Promotion options in the Credits part, which makes sure that the alert is given for the money to be spent by deducing all promotion credits.
On top of that, it’s a good idea to set quota limits: Go to Main menu (hamburger) > IAM & Admin > Quotas. Set the filter to Translation to get rid of a lot of lines.
It’s also the place to get an accurate figure for the current consumption.
Enable the quota for “v2 and v3 general model characters per day”, which is the only character limit that isn’t per minute, and set it to something sensible, for example 2 million characters if you’re a modest user like myself. That’s $40, which is fairly acceptable damage if the computer goes crazy, and high enough not to hit the roof normally.
Also do something with “v3 batch translation characters using general models per day” and same with AutoML custom models. I don’t use these, so I set both to zero. Just to be safe.
There’s “Edit Quotas” to the top right. Which didn’t work, probably because I did this during the trial period, so quotas are meaningless, and apparently disabled anyhow (or more precisely, enabled to fixed limits).
So the way to do it was somewhat tricky (as it’s probably pointless): To enable a quota, right-click the “Cloud Translation API” to the left of the quota item, and open it in a new tab. Set up the quota figure there. But this description on how to do it might not be accurate for a real-life use. Actually, the system ignored my attempts to impose limits. They appeared on the page for editing them, but not on the main page.
Supporting CJK in LaTeX
I’m wrapping up this post with notes on how to feed LaTeX (pdflatex, more precisely) with Chinese, Japanese and Korean, with UTF-8 encoding, and get a hopefully reasonable result.
So first grab a few packages:
# apt install texlive-lang-european
# apt install texlive-lang-chinese
# apt install texlive-lang-korean
# apt install texlive-cjk-all
Actually, texlive-lang-european isn’t related, but as its name implies, it’s useful for European languages.
I first attempted with
\usepackage[UTF8]{ctex}
but pdflatex failed miserably with an error saying that the fontset ‘fandol’ is unavailable in current mode, whatever that means. After trying a few options back and forth, I eventually went for the rather hacky solution of using CJKutf8. The problem is that CJK chars are allowed only within
\begin{CJK}{UTF8}{gbsn}
[ ... ]
\end{CJK}
but I want it on the whole document, and I need the language setting to be made in a file that is included by the main LaTeX file (a different included file for each language). So I went for this simple hack:
\AtBeginDocument{\begin{CJK}{UTF8}{gbsn}}
\AtEndDocument{\end{CJK}}
As for the font, it appears like gbsn or gkai fonts should be used with Simplified Chinese, and bsmi or bkai for with Traditional Chinese. Since I translated into Simplified Chinese, some characters just vanished from the output document when trying bsmi and bkai. The back-translation to English of a document made with bsmi was significantly worse, so these dropped characters had a clear impact in intelligibility of the Chinese text.
I got this LaTeX warning saying
LaTeX Font Warning: Some font shapes were not available, defaults substituted.
no matter which of these fonts I chose, so it doesn’t mean much.
So the choice is between gbsn or gkai, but which one? To decide, I copy-pasted Chinese text from updated Chinese websites, and compared the outcome of LaTeX, based upon the TeX file shown below. It was quite clear that gbsn is closer to the fonts in use in these sites, even though I suspect it’s a bit of a Times New Roman: The fonts used on the web have less serifs than gbsn. So gbsn it is, even though it would have been nicer with a font with less serifs.
For Japanese, there’s “min”, “maru” and “goth” fonts. “Min” is a serif font, giving it a traditional look (calligraphy style) and judging from Japanese websites, it appears to be used primarily for logos and formal text (the welcoming words of a university’s president, for example).
“Maru” and “goth” are based upon simple lines, similar to plain text in Japanese websites. The latter is a bit of a bold version of “maru”, but it’s what seems to be popular. So I went with “goth”, which has a clean and simple appearance, similar to the vast majority of Japanese websites, even though the bold of “goth” can get a bit messy with densely drawn characters. It’s just that “maru” looks a bit thin compared to what is commonly preferred.
Korean has two fonts in theory, “mj” and “gt”. “mj” is a serif font with an old fashioned look, and “gt” is once again the plain, gothic version. I first failed to use the “gt” font even though it was clearly installed (there were a lot of files in the same directories as where the “mj” files were installed, only with “gt”). Nevertheless, trying the “gt” font instead of “mj” failed with
LaTeX Font Warning: Font shape `C70/gt/m/it' undefined
(Font) using `C70/song/m/n' instead on input line 8.
! Undefined control sequence.
try@size@range ...extract@rangefontinfo font@info
<-*>@nil <@nnil
But as it turns out, it should be referred to as “nanumgt”, e.g.
\begin{CJK}{UTF8}{nanumgt}
나는 멋진 글꼴을 원한다
\end{CJK}
It’s worth mentioning XeLaTeX, which allows using an arbitrary True Type font withing LaTeX, so the font selection is less limited.
See this page on fonts in Japanese and Korean.
For these tests, I used the following LaTeX file for use with e.g.
$ pdflatex test.tex
\documentclass{hitec}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{CJKutf8}
\newcommand{\thetext}
{
它说什么并不重要，重要的是它是如何写的。
}
\AtBeginDocument{}
\AtEndDocument{}
\title{This document}
\begin{document}
gbsn:
\begin{CJK}{UTF8}{gbsn}
\thetext
\end{CJK}
gkai:
\begin{CJK}{UTF8}{gkai}
\thetext
\end{CJK}
bsmi:
\begin{CJK}{UTF8}{bsmi}
\thetext
\end{CJK}
bkai:
\begin{CJK}{UTF8}{bkai}
\thetext
\end{CJK}
\end{document}
Introduction
This post summarizes my insights as I worked my way through translating some technical documents, written in LaTeX, into Chinese, Japanese and Korean. The immediate approach was to feed Google Translate with the pdf documents, but not only are the results ugly, but then there are a lot of technical terms in the documents which are better not translated. Even worse, there are code examples with explanation in the text, file names, references to variable names and other elements that become meaningless if translated.
One of the advantages of having the document written in LaTeX to begin with, is that the LaTeX text formatting commands effectively flag the parts that aren’t just plain language in the text, so it’s relatively easy to spot them and protect them. But that alone was a long way from the finish line, as elaborated in this unexpectedly long post.
A different post discusses the technical aspects of talking with Google Cloud’s API as well as creating documents in these languages with LaTeX.
I also did something similar with translating web pages. For example, the translation of this post to Chinese, Japanese and Korean.
This post was written in the summer of 2022, and odds are that things will change dramatically over the course of time.
Is translation by human better?
The short answer: Yes, as of 2023, human translation is much better. It’s mainly because there is no way to give the translating tool hints about the context. For example, the word “driver” could be a car driver or a term related to a computer. All translation tools just pick one meaning. Some tools allow choosing a specific dictionary, and there are ways to shape the behavior of the translator. But the results are far from satisfactory.
However but both options have their disadvantages: Working with a human necessarily requires trusting that a specific person will perform the job thoroughly, and well, that’s anything but taken for granted. It’s extremely difficult to verify that the work was done well, in particular when the document is technical, as it’s not possible to give it to just someone and ask if it’s well written. An automatic reverse translation will miss some poor translations (in particular poor translations of technical terms) and at the same time make false alarms.
But the worst problem with human translation is that every future change in text requires contacting the people who made the translation, and ask them to make the adjustments. They may not be so willing to do that. So unless you employ these people full-time, it may be difficult to translate small edits.
Another problem with humans is that significant errors in the meaning of the text might occur. It’s easy to reverse or otherwise obscure the meaning of a sentence because of a simple human error. “Be sure not to turn on the power supply” can easily turn into “Be sure to turn on the power supply”. Automatic reverse translation can reveal this, but it’s easy to miss an error like this, when the person that verifies the text already knows what it should say.
Automatic translation should be less likely to make a mistake of this sort, but the truth is that Google Translate, with all its Neural Network magic, turns out to be more human than desired in this matter: It’s not completely unusual that the meaning of the text changes, sometimes to the complete opposite.
It also has a variety of passive-aggressive behaviors, in particular ignoring whole sentences or part of them, mostly when the text becomes a bit rambling.
I had a case where the automatic translation ignored a “non-” prefix on a noun, and by doing so reversed the meaning of the sentence. I’ve also had a case where “must not” was translated into the equivalent of “doesn’t have to”.
The clear disadvantage of an automatic translation is poor expression and grammar. If the technique explained below is adopted, it’s however possible to end up with a fairly good result, even if the language is a bit off at times.
But this disadvantage can be mitigated by letting someone who knows the target language well proof-read the result. This person doesn’t need to know English well, but only be sensitive to the target language, so it’s easier to find someone for that job. And in particular when translating to Asian languages, it’s easy to tell the persons involved to ignore technical terms, as they are easily distinguishable, written in Latin script.
The results of this proof-reading session are only slight changes in word choice or ordering, and they can be verified against automatic translation as well as another person. In fact, in most cases, the best way is to improve the wording in the original language, until the person checking the text confirms it sounds OK.
Whether it’s worth the effort and cost to make this language cleanup is an open question. It’s a matter of how much the target audience appreciates the fact that the documentation is available in their language vs. how much the language defects come across badly.
Another issue with automatic translation is that words with more than one meaning can be mistranslated, in particular when the intended meaning is the less common one for a specific word (examples for that below). A back-translation doesn’t necessarily reveal a problem of this sort.
So with the possibility of having someone read through the translated text, the only remaining problem is when the meaning is changed unnoticed during the translation. Frankly speaking, I don’t know which option, human or machine, is better regarding this problem. The only real solution anyhow is to back-translate the text and read it through. Good luck with that.
General insights on automatic translation
Google Translate is based upon a neural network machine learning algorithm, which means that it’s chaotic by nature (in the scientific sense). That gives it a bit of a human touch, which surely makes the translations better, but also makes it quite unpredictable. In particular, it’s capable of making any possible mistake, no matter how pointless and unexpected. It’s impossible to be 100% sure that it won’t do this or that, and it’s not even a bug when a phrase in the original text just disappears, or when a meaningless string of characters gets translated to something else, also completely meaningless. Those small glitches are part of the game, and it makes automated processing of the translated text quite challenging.
Having said that, the general rule is that if Google Translate does weird things, it’s because it was fed with something it found hard to digest. So even if the weirdness doesn’t appear to be related to language, the best way to rectify this is to change the original text into a simpler, more common way to express the same idea. Unfortunately, this calls for dull, play-it-safe English. However with by far less silly typos and grammar mistakes.
If I was to speculate how Google Translate’s algorithm works, I would say something like this: Attempt to find recognizable words in the sentence, fix spelling mistakes (“did-you-mean” style) and try to match the words that are recognized with a known phrase from the huge training corpus. Pick the known translation into the desired language of the word pattern that fits best. Fill in the words that were unknown in the original language in the translated text in their natural positions — these are treated as names (of persons, places etc.).
Punctuation like full period and commas, as well as enclosure in parentheses, makes the translator treat each part separately, more or less.
The destination language matters a lot regarding the interpretation of the meaning of the text. It doesn’t seem like the question is “what does the original text mean” but “which language pattern was most common in the training data for translating into language X”.
The main takeaways for this speculation, regarding how to write for translation are:
- Use common expressions and language patterns. In particular, use the most commonly used words to express a certain meaning.
- Be super-careful with trivial spelling mistakes, as they break the statistics for the desired language pattern.
- If the translation is successful to one language, in the sense that the original meaning was “understood”, it doesn’t necessarily means it will be as successful to another one. Same goes with failure. It seems to depend on what the translations between the two languages are usually used for. In other words, judging by the results, it seems like translations into Hebrew are used more for everyday text, but translation into east Asian languages is more for technical documents. Hence the selection of meaning tends to be more technical with the latter.
- As there is no analysis of the semantics of the original sentence, anything can happen, including a translation that says the opposite of the original.
Interestingly enough, I’m under the impression that the translation with Google Lens is much better than the cloud based translation service. In particular, the cloud translation is more likely to produce nonsense translations because of small disturbances in the middle of text, where Google Lens’ translation seems to have extra wisdom to overcome such.
Translating to a foreign language
How do you know a translation is OK, when you don’t know the language it’s translated into? The short answer is that one can’t really know. It helps a lot knowing another language, even if it’s different from the target language, because it allows spotting misinterpretations of certain words, in particular technical ones. But often a word is poorly translated into one language, but fine with another.
There’s the possibility to translate it back to English, but that doesn’t always spot problems. Technical words like “clock”, “bus”, “sink”, “assertion” are translated to ridiculous words in Hebrew, for example, but the translation back looks OK in English. In particular a work like “sink” translates into the word in Hebrew that means kitchen sink, and then goes back to the correct work in English, of course.
But then comes the question: Why translate these words at all?
Quality of translation
Among the three target languages, the translation to (simplified) Chinese is the best by far. Probably because the natural flow of Chinese is the closest to western languages. The runner-up is Korean, and the worst is Japanese.
The worst problem with both Korean and Japanese is that parts of the original text can just disappear. This happens often when the semantic structure gets too complicated, or if there’s no normal way to say something in Japanese. For example, the sentence “you’re absolutely welcome to mess up completely, the tools won’t stand in your way” lost the entire first part in Japanese. So it just says “no tools get in the way”. If only the first part is translated separately, it turns into “completely ruined is welcome” (it had to give me something back when that sentence stood alone).
So short and plainly informative sentences are best translated into Japanese and Korean. Chinese seems to work with anything.
As for words like “it”, Chinese tolerates that best too. The two other languages are more likely to need repeating the word that “it” refers to, and hence possibly pick the wrong word to repeat.
Testing by translating to Hebrew
Since I happen to speak Hebrew fluently, I checked the translation to Hebrew of all documents, not for the purpose of publishing this translation, but because I soon found out that Google Translate struggles with Hebrew. So the idea was that if it’s OK with Hebrew, it’s probably OK any language.
For this, I tried two cases where the translation got wrong, as indicated by the result in Hebrew.
The first sentence that failed was “Close the window after a successful generation”. The problem was that the word “generation” was interpreted as the relationship between age groups, and not from the word “generate” as intended. This, in itself, is easily fixed by changing it into “Close the window after a successful generation of the file“. It was a matter of fitting the entire sentence into a different pattern of words.
Surprisingly enough, the translation into Chinese, Japanese and Korean was correct even without the fix. This can be verified by looking at the translation back to English, and isolate the word or couple of words of interest.
The next problematic phrase was “The non-X81X are ignored by almost all X82X computers”. In the translation to Hebrew, the “non-” part was ignored, so the sentence’ meaning was reversed. Once again, the translation into the three other languages was correct (the X81X thing is explained below).
So if I once had the speculation that the machine translates the words into an intermediate format that somehow contains the meaning, and takes it into the target language from there, it’s definitely not the case. Whether there’s a misunderstanding or not in the translation depends on the target language.
I’m optimistic and hope that Hebrew is in particular prone to errors, so if I clean up the translation to Hebrew, it will hopefully work with other languages. However odds are that each language has its own pitfalls. Even though it really seems like the translation to Hebrew from any language is bad in particular. Including changing the meaning of the text. Also, I’ve found that plain idioms like “it doesn’t hurt” are often translated horribly to Hebrew but get perfectly OK in CJK languages. But then, I don’t know about misses in CJK languages that were OK in Hebrew…? And yet, after checking numerous expressions (“bite back”, “copy-paste” and a lot of this sort) it really seems like Hebrew is really bad off.
This way or another, the only sure benefit of checking the translation to Hebrew is that it does, after all, remove some ambiguities, whether that is necessary or not. Actually, I found tons of plain typos by looking at this translation, so that alone justifies this phase. It’s difficult to proofread text exactly as it was written, but reading it again in another language feels as if someone else wrote it.
I also had the opportunity to have a translation into Japanese by a helpful person, and it was quite clear that the problems were in the places where the Hebrew translation also limped.
Hands-on insights
After quite some back and forth, I learned that the best way to work with Google Translate with text is to feed it with paragraphs of text in HTML, enclosed in <p> (or <hN>) tags. Plain formatting tags is fine (<b>, <i> and even <span> etc.) but it’s important not to insert anything that breaks the continuity of the sentences: No <br> or <img> tags in the middle, or anything else that isn’t expected in the middle of a sentence. It makes Google Translate translate the part before and after the break as separate sentences, and that’s a disaster.
Newlines are ignored in the cloud interface with HTML, as they should be. This is contrary to the web interface for Google Translate, which is extremely sensitive to newlines, so copy-pasting a chunk of text from a pdf document can result in a horrible translation, because there are newlines between each row in the original text, which makes the translator treat each row a separate phrase.
But the real difficulty is the fact that the translated text is technical. Google Translate is trained with mainly non-technical text (I guess), so its interpretation of technical terms that happen to also have a non-technical meaning is naturally inclined towards the non-technical meaning. Words like “driver”, “compile”, “assert” and “board” are not only likely to be translated incorrectly, but also stir a mess in that imaginary brain that holds all those neurons, resulting in a completely unintelligible translation.
The problematic words are those that have a possible non-technical meaning. The word “boot” could mean a piece of footwear, to boot a computer could be mistaken for “to give the computer the boot”, but to reboot a computer could only mean one thing. So it’s not all that much about the word being technical, like the fact that it could be remotely confusing.
Other ambiguities occur with words like “target”. Using it in any form, i.e. “to target” or “targeting” as well as “targeted” as in “depending on which software version is targeted” leads to a completely wonky translation, at least into Hebrew.
Surprisingly enough, it copes quite well with sentences that contain untranslatable items. I guess it treats anything it can’t handle as a name. Since it’s supposed to be able to translate “Joseph prefers Paris over Berlin”, it works fine with “X prefers Y over Z” as well. So the trick is to remove all technical terms from the the text, and replace them with something that Google Translate will treat as a name, something it can’t translate. And then return those words into the translated text.
This means that all technical terms remain in English in the translated text, which is perfectly fine, because a technical reader is expected to know these terms. It’s the blah-blah part that needs translation, and with the technical words out of the way, Google Translate does a good job on that.
The problem that remains is how to feed the translator with these untranslatable X, Y and Z placeholders, when there can be thousands of these, and they must all be left intact in the translation (well, except for Russian and Greek, see below). The section below on placeholders tells the full story, but the spoiler is that I used X0X, X1X, X2X, and so on. It’s not watertight, but it works best. I tried quite a few options.
The main thing to keep in mind is that it’s all about word patterns: If Google Translate recognizes the structure of the sentence, based upon words that are commonly used together for a certain meaning, it translates that part correctly, and then puts the placeholders in the right places, treating them as names.
I should mention that Google Translate offers a “notranslate” style, which can be used to enclose e.g. <span> segments of text that shouldn’t be translated. I didn’t attempt using it, in particular as people out there in the web have complained that it disrupts the translation exactly like that. Another problem is that chunks that shouldn’t be translated often have a different formatting (e.g. Courier font for variable names), and Google Translate tends to behave in an unpredictable manner, making it difficult to rely on its output for feeding LaTeX with directly.
Also worth mentioning is that Google offers an advanced version of the translation API, with the ability to train the learning machine and possibly feed it with specific word translations, but that would require knowing the correct term in the target language. How do you say “compile” in Chinese and Japanese? But it could have been helpful for solving the problem with verbs, that have a technical meaning (“compile”, “boot”, “implement”, “overflow”, you name it).
How I actually did it
The idea was to extract translatable text from the LaTeX source, and feed Google Translate’s cloud API with it in HTML mode. Then take the translated text and implant it back into the LaTeX doc.
The overall goal is to feed Google Translate with hollow phrases, albeit with a solid and common semantic structure, of the form of “X with Y is more important that Z”. This makes it easy for the translator to detect the structure of the phrase, and translate it to a meaningful sentence in the foreign language. That gives good odds for meaningful sentence when the placeholders are replaced with the actual technical words in the translated phrase.
In more detail:
- Fetch paragraphs of text and enclose them in <p> or <h1>, <h2> or <h3> tags. Each of these tags have a unique “id” attribute, so when the translation returns, it’s possible to track which text segments should be written back to which part in the LaTeX file. This is why HTML mode came handy. I haven’t had a single case of these attributes being messed up (yet?).
- Turn some LaTeX formatting into plain HTML tags, e.g. <b>, <i> etc. Then do the opposite when implanting the text back. The advantage is that this doesn’t break Google Translate’s view of the text as a contiguous sentence. Once again, HTML mode is required for this stunt.
- Anything that shouldn’t be translated — technical terms, references to variables, file names, references to paragraphs, labels etc. — is replaced with a unique identifier (“placeholder”) that Google Translate doesn’t attempt to translate. The major caveat with this method is that it works only with nouns. This requires rewording, in particular turning verbs into nouns (e.g. “perform a compilation” instead of “compile”). More on this below.
Note that some parts of the LaTeX document are completely out of this game, as they aren’t even given to the translator to look at. For example, verbatim environment chunks, and even the newlines between the text paragraphs. They remain the same because they aren’t overwritten when the translated text is transformed back and implanted in the relevant segment.
Work flow
I wrote a Perl script for the back-and-forth manipulations between LaTeX and HTML, but I won’t get into that too much, because it’s complicated and really specific to the documents at hand for translation. Among others, this script loaded a list of words that are always replaced with placeholders, and I also added a LaTeX command, \notranslate{}, which just leaves the content as is when interpreted by LaTeX, but to the script it means that the entire chunk should be replaced with a placeholder as well.
Writing scripts and all that is nice, but there’s still some manual preparation required. So this was the procedure I adopted:
- Run the script that creates the HTML file that is sent to Google Translate. View that file with a web browser, and look for words that are technical and can be mistranslated. When such are found, either add the word or phrase to the list of words to automatically replace with placeholders, or fix it specifically with \notranslate{} LaTeX statements.
- In fact, I also wrote a script that puts \notranslate{} on certain words and patterns (e.g. sets of upper case characters) so I ran this script, and then verified each such occurrence. This is faster than finding them manually, and is useful for words that may have a non-technical meaning, or otherwise require human attention to get 100% right. For example, the word “image” should be translated when it refers to a picture in the text, but not when it’s an image of a disk.
- Go through the text manually, and apply the guidelines listed below (the do’s and don’ts).
- Translate the text into Hebrew, and read through the result. If something ends up unclear, fix it. The further the language is from English, the better. The one sure benefit of this check is that small typos are spotted (e.g. “in” instead of “is”) because the translation gets weird. The fact that the order of words changes in the translation also helps spotting ambiguities, that are often solved with works like “which is” or punctuation.
- Translate into the target language. Make the necessary fixes. Don’t bother to find out why certain placeholders are missing in the translation. Rather, look at the original text and try to figure out why it was difficult to translate, and fix that instead. Sometimes a missing placeholder is due to a whole sentence being dropped off, in particular with Korean. It’s as if the algorithm said “I have no idea how to reorganize this sentence into something that makes sense in Korean, so I’ll just skip it”.
- Maybe attempt to translate the document back as a pdf file (with Google Translate’s web interface) or use Google Lens’ translation feature for a more sporadic check. I’m not sure if this is worth the time.
The order of translation is Korean first, then Japanese and finally Chinese. This is because the translation to Korean is the most troublesome, however often fixing the problems consists of changes that are likely to benefit the other translations.
All in all, it appears like using placeholders instead of technical terms actually improved the translation regardless of these specific words. It seems like these words confused the translation machinery, which made it create obscure phrasing. With the technical words out of the way, inserted as opaque symbols, it seems like Google Translate managed much better to handle the rest, which now consisted of commonly spoken language.
So my ultimate approach was to put placeholders instead of virtually all technical terms which are nouns. That’s quite a lot of them, and the translated documents ended up full with terms in English. I’m not sure what Chinese are going to think about this, but if they have the same problem as in Hebrew — weird “official words” for technical terms — it’s going to be just fine.
The do’s and don’ts
Based upon quite some trial and error, these are the guidelines I ended up with for producing text with placeholders that translates well.
- The text should consist of hollow sentences like “If the X113X process fails, the X641X’s output may supply hints on what went wrong. The X102X application on the computer should be configured for X114X, no X115X and no X116X ( X640X )”. However sentences like “The X219X for X220X on X221X or X222X is part of the majority of X223X and distributions, as explained below” should be fixed, inserting some meaningful words between those four placeholders with just one word between each. In this example, it’s not clear whether the last “or” refers to instead of X221X alone or all of the three. If the translation requires word reordering, this will obscure the meaning of the sentence.
- Use punctuation (in particular commas) and parentheses to chop up long sentences into segments. This prevents ambiguity. In particular, text inside parentheses is translated into parentheses, so this is a good tool for breaking up long and complicated sentences.
- Try to keep phrases short and concise (and somewhat boring), partly because sentences are short in the target languages. If the sentence is long, try to mitigate the damage with punctuation.
- Use plain and explicit English. Don’t leave out “which”, “that” and all those words that explicitly define the role of each word. Even a simple expression like “for the curious” can go wrong, but works perfectly well when changed into “for those who are curious”. Yuck, but translates well.
- Avoid words that refer back to something earlier in the sentence, unless it’s very obvious. In particular, the word “it” is often replaced with the word it’s supposed to refer to during the translation, and sometimes the wrong word is chosen in the translation. When this happens, the translation explicitly changes the meaning. Because the translation into CJK languages often involves splitting a long sentence into shorter ones, without a possibility to use a word like “it”, implicit references of this sort easily translate into nonsense. To make things worse, the back-translation may bring back the “it”, so there’s no way to spot the mistaken translation. There are cases where these duplications are safe, for example expressions like “one thing to another” (which is often translated into “one thing to another thing”).
- Prefer “the red book and the blue book” over “the red and blue books”. The order of the words may be changed during the translation, and in that case, it’s possible that only the “blue books” is moved to the new position in the sentence, so the result is rubbish. These overly explicit sentence are less awkward to read than they are awkward to write, but they are nevertheless slightly awkward as the same word is repeated over and over again.
- Avoid idioms. Even the simplest ones, like “out of the box” may and may not translate into something that makes sense. Because of the statistical nature of the translations, idioms might get translated with the right spirit into a certain language, and fail completely with another. So dull language it is.
- Avoid verbs in passive form, in particular if it comes with a “by”. Passive form is useful for not naming the doer, but if it’s named anyhow, use the active form. A lot of times, the passive form, and the tangled sentences that it usually creates, were the reason for problems in translation.
- Use possessive form for clarification. For example, if the word “register” is replaced with a placeholder, “register modification” should change to “modification of registers” or likewise “registers’ modification”. Using the ‘s suffix works great, so use it as long as it doesn’t create an ambiguity on who the owner is.
- In fact, there’s no problem at all with segments like “X400X’s X401X”, as possessive form. This translates well, surprisingly enough.
- Don’t replace partial expressions with placeholders. For example, in the expression “the user-space application”, don’t replace just “user-space”, but rather “user-space application”. Word ordering might be different in another language, which can at worst lead to a complete disassociation between the placeholder and its related word in English, with a completely unclear result.
- Avoid replacement of parts of expressions with placeholders. For example, in “VGA port”, if only “VGA” is replaced, it’s not sure if this will translate fine. “VGA-port” increases the changes. If it’s a common language pattern, e.g. “VGA-based”, there’s a good chance for proper translation. Same goes with “the X500X file”, because it’s a common language pattern.
- Don’t use “non-” as a prefix. It’s often missed, reversing the meaning.
- Look out for ambiguous words. For example, the word “signals” could be the verb (to signal) but also the plural of the noun. Avoid less common uses of words, such as “writes” to say several write operations, and use “write operations” instead.
- Be extra careful with trivial spelling mistakes and typos, in particular mixing “is” with “it” and such. These are overlooked when reading the text in English, but they kill the translation, sometimes by changing the meaning significantly, and sometimes by just confusing the translation algorithm into producing something weird.
- Bonus: Check all duplication of placeholders, and verify that the correct one is duplicated. Because these duplications are usually the result of a word that refers back to something (“which”, “that”, “it” etc.), it’s a good idea to verify that the reference goes to the correct placeholder. In theory, this should be done with all uses of back referencing, but that means proofreading the entire text. So with placeholders it’s less work (and less gain). Having run through a checkup of my own translations, I’d say about 10% of these duplications garble the meaning, by explicitly duplicating the wrong word.
Caching translation results?
Since the document is chopped into paragraphs, each within a <p> enclosure, does it matter if each is sent separately or if all are sent in one API transaction as a concatenated string? Does it matter if the translator sees the entire text?
Because if each <p> enclosure is treated separately, it’s possible to cache the pieces of text that have already been translated.
Caching is more than just a money saver. It allows making manual changes in Google Translate’s output (in particular if it messed up the placeholders) and then not having to repeat this every time the entire document is translated.
Even more important, avoiding the repeated translation of parts that have already been translated means avoiding the possible mishaps that may suddenly occur (like suddenly dropping a sentence). Think about making a small change, and then the translation fails on something completely different. But it worked last time!
This is also important if there’s feedback from readers that corrects a poor translation at a specific place. So caching is very helpful for the incremental kind of work that is necessary to maintain the document in the long run.
So I tried this with translating from English to Hebrew, and a bit with Chinese as well (by looking at the translation back to English). As it turns out, there are occasional differences between the translation of an isolated paragraph and that made with a context. But it doesn’t seem like an intelligent use of the context. Comparing the results, the isolated translation was sometimes better, sometimes worse, with a very slight difference in most cases. So it looks more like the algorithm randomly picked another wording, for no apparent reason. It was usually exchanging equally valid synonyms, or choosing to translate the name “Linux” to Hebrew or not.
Another observation I made is that the use of context is poor. For example, the word “call” is translated to the the word in Hebrew that means a phone call, but “function call” is translated correctly. So what if there’s a sentence saying something about a “function call”, and a sentence afterwards uses the word “caller”? In the <p> enclosure, that is. Well, the translation of “caller” still relates to a phone call. The neural network clearly didn’t learn anything from the first sentence.
So it makes perfect sense to cache translations at a paragraph level. If the original document changes, request a translation only on the enclosure that actually changed.
Finding the right kind of placeholder
This is a long explanation on why ended up with the XnX placeholders. I would skip this part if I were you.
As mentioned above, the main problem with translating a technical document is that some technical terms are translated into an unhelpful, sometimes ridiculous way, and that it confuses the translation algorithm. As the reader of the document is most likely familiar with the English term, it’s safer to leave these words as is. The problem is how to insert these terms in a way that ensures they don’t get translated, and at the same time retain their position in the context.
As it turned out, the main problem with inserting an untranslated chunk into the text is that it may disrupt the translation, in particular as Google Translate tends to treat the part before and after the chunk as separate sentences, which results in a poor translation that misses the point of the sentence.
I began with adding markers in a plain text (like <%103%>, [^29^] and ^^26^^), however Google Translate inserted a space in the middle of some of these (so it turned out to be e.g. “< %103%>”) and also threw in some markups where they shouldn’t be. A complete disaster, in short. This could have worked with non-HTML translation, but well, it didn’t work.
Another attempt was to use translation of HTML, with <b id=”n23″>P</b> markers as placeholders. The id allowed to identify which placeholder to insert, and the “P” to give the translator something to consider as a word. This failed as well, in many ways: The fact that the “P” part sometimes got translated into “PP” (why on earth) didn’t matter much, because it’s not really important. The real problem was that at times there were other words inserted into the <b> enclosure as well (for no apparent reason). Even worse, sometimes a completely different word, somewhere else in the sentence, got into a <b> enclosure with the same id. So processing this would have been complicated.
Another thing I tried was to use <var>n</var> enclosures, where n is the number of the placeholder. That failed, partly because some of these disappeared for no clear reason, and others were manipulated (for example, characters from previously outside the enclosure went into it).
To ensure that the placeholder is fully opaque, I tried <img id=n23>. The clear advantage was that Google Translate didn’t duplicate these not modify them, but they broke the sentence into fragments. Google Translate assumed that no sentence will have an image in the middle of it.
So if not images, what about emoticons? Or even better, I made an attempt to use the Unicode range U+13000 to U+1342e (Egyptian Hieroglyphs) as placeholders instead of <img> markups. The idea was that Google Translate would have to pass them through as is, and that they would be considered to be names. In order to make this work, there had to be a whitespace on both sides of the Hieroglyph, but even with that, Google Translate would mess up and occasionally add completely unrelated characters instead.
In the end, I went for inserting words like X0X, X1X, X2X, and so forth. These remain intact through translation, however they are occasionally duplicated, in particular with sentences like “that is possible with X, which is the best option” which can turn into “that is possible with X, and X is the best option”. The word “it” is also translated sometimes into the placeholder instead. But that’s actually a correct translation, and it’s easy to process. Even though this worked almost flawlessly, there were occasional surprises, including rare cases where Google Translate changed the number between the Xs without myself being able to figure out why on earth, and why that specific change. So there’s always a certain amount of manual cleanup after the translation.
These duplications are common with east Asian languages, and usually occur when a long sentence is chopped into several shorter ones. In these languages, it’s more common to repeat the word than to use “it”, “which” and such.
When translating to Russian and Greek, the “X” character was occasionally replaced with the Russian capital letter “Ha” (Unicode U+0425) or the Greek capital letter “Chi” (Unicode U+03A7). Both look exactly like an “X”, so the replacement is understandable. Once this issue is known, it’s quite easy to handle, so it’s not a big deal.
As for the quality of the translation, this worked well, and Google Translate combined these nicely into the translation, even when changing the word ordering was necessary. This works however only when the placeholder is used as a noun. So it doesn’t solve the problem with verbs like “assert”, “raise”. In some cases, a word like “overflow”, used as a verb, can be replaced with something like “cause an overflow”, so it can be translated properly.
Another thing with these XnX placeholders is that there must be a whitespace in either side of it, or Google Translate gets confused. To ensure that the placeholder is restored properly, the strategy was to include any surrounding whitespaces in the string that was stored to replace the placeholder later on, and then add a whitespace in either side of the XnX string. When reverting the process, all whitespaces around the XnX string were removed before restoring the original string. This results in a perfectly consistent back-and-forth, even if the translator adds or removes whitespaces (which happens a lot).
As a side note, Google charges for all characters, even those not translated. Hence it’s a good idea to keep the placeholders short markups. Not a big deal, but still.
Sanity checks on placeholders
The natural expectation is that any placeholder in the text for translation will result in a single placeholder in the translation. I’ve already mentioned above that some placeholders turned into two in the translated text, and it was actually correct. But what if the placeholder disappears?
The answer is that it’s always an error, and it has to be fixed manually. In fact, it’s often an indication that something worse happened, which would have been left unspotted had it not been for the missing placeholder. Sometimes the number between the Xs is changed arbitrarily, but it happens in conjunction with other placeholders in the vicinity being messed up.
Sometimes the absent placeholder was the result of a part of a sentence that was completely eliminated. The small piece of information it contained was simply absent in the translation. This can happen for several reasons, but the most recurring one seems to be when it’s not clear what “which” or “that” refers to, earlier in the same sentence. One can get away with that in translations to European languages, but because the sentence is built differently in east Asian languages, the translator is forced to make a pick. So instead of doing that, it just eliminates the part it can’t decide upon. A neural network algorithm showing a bit of human behavior, I would say.
It also seems that a colon sign (‘:’) tends to eliminate what comes immediately after it, fully or partly. Changing it to a full stop often returned chunks of texts from the dead in Korean and Japanese. Or splitting the text, so that part after the colon is in a separate enclosure (note to self: possibly with a \skipthis{}).
Same thing with a sentence starting with “likewise”.
Another somewhat weird phenomenon with Korean and Japanese is that a whole sentence was sometimes dropped. The really weird thing was that when the same sentence was put in a separate <p> enclosure, it was translated properly. So it was like Google Translate said “nah, this is too much rubbish, I’ll drop the last sentence”.
So in this sense, the placeholders help spotting other problems with the translation. I got an error of this sort for each few thousand translated words, which practically means a bit of fixing for each document. What’s really worrying is how many sentences without any placeholders have vanished unnoticed?
Placeholders that contain a word in plural
One problem that is inevitable with placeholders is that the information on the word’s plural vs. singular form is hidden away from the translator. So if the work that is hidden is “compilers”, the surrounding text in the translation might refer to it in singular, and that makes the sentence sound a bit off.
In some cases, the translator can deduce it from the surrounding words (e.g. if “is” or “are” is used in reference to it), but sometimes there are no hints. Luckily, the plural-singular thing isn’t very present in Chinese, Japanese and Korean, so the effect of this ambiguity is expected to be small. Try, for example to translate and back-translate “He gave me the books” with these languages, and you get “he gave me a book” — the indication for plural is lost. But there’s also a backside to this: The fact that the original word in English appears in its plural form will probably feel uneasy to an East Asian reader. I’m not sure about this, but it appears like they would use the English word in singular form anyhow, even if it refers to several pieces of whatever it is. So any use of plural will probably feel wrong to them.
Surprisingly, this can be fixed by using a placeholder like X205Xs (with the “s” in the end). This appears to be translated correctly into plural, and even the possessive form (e.g. X205Xs’) seems to work well into Hebrew.
But this hack creates a new problem: The translation might add suffixes and other grammatical elements to mark the plural form of the hidden word. If this happens, there will create a double plural. In German, for example, there are many ways to go from singular to plural, so this extra “s” just remains, when it comes after an XnX placeholder. If it isn’t removed, the result is “compilerss” (with a double “s” at the end). In Norwegian, it may add “-er” for plural (with the dash).
OK, so remove anything alphanumeric that comes after a placeholder, so that if the “s” remains, it’s gone? That may not work well either. For example, the possessive form in Swedish is expressed with a “:s” suffix and “:n” in Finnish (at least on a placeholder), so removing suffixes blindly takes its toll as well.
So even though appealing, there “s” method won’t work as a clean way to hint that the word is plural, in particular because the placeholder might get conjugated into plural in the translation. And there’s no catch-all solution for getting rid of this possible conjugation.
Given that the problem with plural is a relatively minor nuisance, that happens only when the context doesn’t say that it’s plural, it’s not worth the risk of adding garbage characters, or mistakenly removing relevant conjugation characters.
On the wishlist: The possibility to tell the translator that a blob is a noun in plural. Actually, wouldn’t it be nice to be able to do that with verbs as well, saying which tense and person?
Placeholders and Korean particles
In English, we have this thing that we say “a book” and “an orange”. The choice of the indefinite article, “a” or “an”, depends on whether the word that comes after it starts with a vowel or consonant sound.
In Korean, there are particles that are added after a noun to mark if it’s the subject, the topic or the object in the sentence. The particle is chosen according to whether the preceding word ends with a consonant or a vowel, respectively:
- Topic particles: 은 or 는 (eun or neun)
- Subject particles: 이 or 가 (i or ga)
- Object particles: 을 or 를 (eul or leul)
Not surprisingly, the particles that come after a vocal begin with a consonant, so there’s always a consonant in the game. Same principle as English’ indefinite article.
And here’s the crux: When a placeholder is used instead of a noun, Google Translate gets XnX instead of the real word, so the particle is chosen according to the “word” at hand.
So “I read the book” is translated by Google to 난 책을 읽는다 (book is 책, chaeg, ends with a consonant, hence the choice of the object particle 을, eul). But if “book” is replaced with “X10X”, we get 나는 X10X를 읽었다. “X” sounds like “eksae” in Korean, so it ends with a vowel, hence the 를 particle was used. (The word that means “I” changed from 난 to 나는, but the former is just a contraction of the latter, so it’s like “I’m” vs. “I am”)
This can be fixed automatically by looking for these particles: They are always immediately after a placeholder, and there’s a whitespace after them. The tricky part is to identify whether the replaced word ends with a consonant or a vowel, the way it’s pronounced in Korea (which may be different from the English pronunciation?).
The possessive particle, 의, as well as several other particles are indifferent to this matter.
It doesn’t seem like there’s a similar problem with Japanese nor Chinese, but I reached that conclusion based upon not finding anything related with a Google search. I will be really surprised if there was anything like this in Chinese because its script is generally unrelated to pronunciation. But with Japanese, I’m not that sure.
Maybe use a word in the target language?
I haven’t experimented a lot on this option, but maybe it will work: If a text is translated into Hebrew, and there is a Hebrew word in the middle of the text, it’s used correctly in the translation. So for example, “I ran back to בית quickly” is translated to “רצתי בחזרה לבית במהירות”. This isn’t perfect (הביתה would have been better) but it shows that a word in Hebrew is conjugated slightly and correctly.
So this opens for the possibility to replace technical terms with their relevant word in the target language. It seems like the grammar in CJK languages is exceptionally forgiving regarding nouns: There is generally no plural form, and it also seems like other conjugations are made with separate words (e.g possessive form).
Even more interesting, it works with verbs as well. “I רץ back to בית quickly” translated into “אני חוזר מהר לבית” which means “I quickly return home”. The word for “run” (רץ) was magically replaced with “return”, which is an interesting interpretation.
So maybe this can work. Not sure how much it improves, though.
It’s 2022, Perl isn’t as popular as it used to be, and for a moment I questioned its relevance. Until I had a task requiring a lot of pattern matching, which reminded me why Perl is that loyal companion that always has an on-spot solution to whatever I need.
These are a few notes I took as I discovered the more advanced, and well-needed, features of Perl regexps.
- If a regex is passed as a value generated by qr//, the modifiers in this qr// have a significance. So e.g. if the match should be case-insensitive, add it after the qr//.
- Quantifiers can be used on regex groups, whether they capture or not. For example, \d+(?:\.\d+)+ means one or more digits followed by one or more patterns of a dot and one or more digits. Think BNF.
- Complex regular expressions can be created relatively easily by breaking them down into smaller pieces and assigning each a variable with qr//. The complex expression becomes fairly readable this way. Almost needless to say, quantifiers can be applied on each of these subexpressions.
- It’s possible to give capture elements names, e.g. $t =~ /^(?<pre>.*?)(?<found>[ \t\n]*${regex}[ \t\n]*)(?<post>.*)$/s. The capture results then appear in e.g. $+{pre}, $+{found} and $+{post}. This is useful in particular if the regex in the middle may have capture elements of its own, so the usual counting method doesn’t work.
- Captured elements can be used in the regex itself, e.g. /([\'\"])(.*?)\1/ so \1 stands for either a single or double quote, whichever was found.
- Even better, there’s e.g \g{-1} instead of numeric grouping, which in this case means that last group captured. Once again, useful in a regex that can be used in more complicated contexts.
- When there are nested unnamed capture parentheses, the outer parenthesis gets the first capture number.
- If there are several capture parentheses with a ‘|’ between them, all of them produce a capture position, but those that weren’t in use for matching get undef.
- (?:…) grouping can be followed by a quantifier, so this makes perfect sense ((?:[^\\\{\}]|\\\\|\\\{|\\\})*) for any number of characters that aren’t a backslash or a curly bracket, or any of these followed by an escape.
- Quantifiers can be super-greedy in the sense that they don’t allow backtracking. So e.g. /a++b/ is exactly like /a+b/, but with the former the computer won’t attempt to consume less a’s (if such are found) in order to try to find a “b”. This is just an optimization for speed. All of these extra-greedy quantifiers are made with an extra plus sign.
- There’s lookbehind and lookahead assertions, which are really great. In particular, the negative assertions. E.g. /(?<![ \t\n\r])(d+)/ captures a number that isn’t after a whitespace, and /(\d+)(?![ \t\n\r])/ captures a number that isn’t followed by a whitespace. Note that the parentheses around these assertions are for grouping, but not capturing, so in these examples only the number was captured.
- Lookaheads and lookbehinds also work inside grouping parentheses (whether capturing or not), as grouping is treated as an independent regex.
About this messy post
As I rooted my Google Pixel 6 Pro, there were a few things to get in place on my Linux Mint 19 machine. These are random notes I took as I went along.
Install ADB and fastboot
# apt install android-tools-adb android-tools-fastboot
So that was easy.
Next, opening the phone for ADB access, the standard Android way: Go to the phone settings, into About Phone. Tap seven times on Build Number. The phone prompts for the PIN number, and then the “You are now a developer” message appears.
Now, under System, there’s the Developer options. Enable USB debugging. And then at shell prompt:
$ adb devices
List of devices attached
23011xxxxxxxx no permissions (user in plugdev group; are your udev rules wrong?); see [http://developer.android.com/tools/device.html]
Arghh. This was because of a lacking udev rule for when the phone isn’t in File Transfer / Android Auto mode. So I enabled file transfer (even though the real solution is to fix the udev file, as described below). And that’s when a popup appears asking if the computer should be allowed USB debugging. So yes, and pick “Always allow from this computer”.
Now we’re talking:
$ adb devices
23011xxxxxxxx device
Yey. Even more yey: Using adb shell, I got
$ id
uid=2000(shell) gid=2000(shell) groups=2000(shell),1004(input),1007(log),1011(adb),1015(sdcard_rw),1028(sdcard_r),1078(ext_data_rw),1079(ext_obb_rw),3001(net_bt_admin),3002(net_bt),3003(inet),3006(net_bw_stats),3009(readproc),3011(uhid) context=u:r:shell:s0
The privileges in this mode is much better than as an SSH client (see below). For example, one can list the files in /system/bin and /dev with adb shell and not through ssh (unless the ssh session gets root). Same goes with using “top”: It shows all processes, not just the same users’, as with ssh (once again, if the ssh session gets root, sky’s the limit).
There are nice executables in /vendor/bin too (even ifconfig and nc)
Getting the udev rule right
To resolve the “permission denied” thing with adb without file system access and fastboot, a udev rule needs to be added.
With MTP, file transfer on and off, I checked which udev rules got in action for the device with
$ udevadm test -a add $(udevadm info -q path -n /dev/bus/usb/002/004)
The 002/004 in the end are the bus and device numbers as found in lsusb.
It turns out that the relevant rule was in /lib/udev/rules.d/69-libmtp.rules:
# Google Inc Nexus/Pixel (MTP+ADB)
ATTR{idVendor}=="18d1", ATTR{idProduct}=="4ee2", SYMLINK+="libmtp-%k", MODE="660", GROUP="audio", ENV{ID_MTP_DEVICE}="1", ENV{ID_MEDIA_PLAYER}="1"
Note that it sets the group to “audio” and not “plugdev” The ENV{} assignments prevent the call to mtp-probe in this same udev file.
The truth is that I don’t completely understand why that works at all.
Anyhow, I ended up adding the following as /etc/udev/rules.d/20-google-pixel.rules:
# Google Inc Pixel 6 Pro, support for no-file transfer ADB mode
ATTR{idVendor}=="18d1", ATTR{idProduct}=="4ee7", MODE="660", GROUP="plugdev"
# Google Inc Pixel 6 Pro, support for USB tethering + ADB mode
ATTR{idVendor}=="18d1", ATTR{idProduct}=="4eec", MODE="660", GROUP="plugdev"
# Google Inc Pixel 6 Pro, support for MIDI + ADB mode
ATTR{idVendor}=="18d1", ATTR{idProduct}=="4ee9", MODE="660", GROUP="plugdev"
# Google Inc Pixel 6 Pro, support for fastboot mode
ATTR{idVendor}=="18d1", ATTR{idProduct}=="4ee0", MODE="660", GROUP="plugdev"
and don’t forget
$ sudo udevadm control --reload
I assigned the group “plugdev” or else fastboot required root (on the host computer) to detect the device. Also, I didn’t add any of the ENV{ID_MTP_DEVICE}=”1″ commands, because doing so prevents running mtp-probe. And oddly enough, if mtp-probe doesn’t run, I get a popup saying “Unable to mount mtp device”.
As a side note, mtp-probe runs on virtually every USB device. This is quite ugly, actually.
Backing up some of the phone’s data
You can’t just use tar to backup the root directory. Not sure why, but nothing happened when I tried. Maybe an SeLinux thing? So it boils down to something like this:
$ time adb shell 'su -c "tar -cz /storage/emulated/0/"' | sudo tar -xvz -C /mnt/tmp/googlepixel/
This backs up the visible user data. For hidden application data, repeat this with data/:
$ time adb shell 'su -c "tar -cz /data/"' | sudo tar -xvz -C /mnt/tmp/googlepixel/
And here’s the crux: There are several duplicate mounts in the filesystem. It seems like a lot under /data/user/0 is a repeated backup. And even more so, /data/media/0 is apparently a duplicate of /storage/emulated/0/ (or the other way around? I’m confused), so maybe there’s no point backing up the latter if /data is backed up.
For a round-up of backing up things that are probably completely useless:
$ time adb shell 'su -c "tar -cz /mnt/vendor /apex /metadata"' | sudo tar -xvz -C /mnt/tmp/googlepixel/
Have I missed something that should be backed up? I don’t know.
Grabbing system info
Be sure to check out this reference and this cheat sheet for adb commands.
There’s a utility, dumpsys, that allows getting system information easily from the phone.
So, to tell which application is responsible for a window that just showed up:
$ adb shell dumpsys window windows > windows.txt
Or get some memory info (which application takes how much memory?):
$ adb shell dumpsys meminfo > mem.txt
What caused apps to terminate? This gives a log of last exits for each package.
$ adb shell dumpsys activity exit-info > exit.txt
Or everything at once (takes about 30 seconds to run, and emits a lot of error messages):
$ adb shell dumpsys > all.txt
Note that the file is output on the host, not on the phone with these commands.
To get the system log, use logcat:
$ adb shell logcat -d > all-log.txt
The -d flag tells logcat to terminate when it reaches the end of the log. Otherwise it continues running in “tail -f” style.
To get a list of all running processes:
$ adb shell ps -A > ps-all.txt
Look at this page for some additional utilities, in particular the Package Manager (pm) and Activity Manager (am). For a more in-depth understanding of the machinery, look for information on Intents and Activities.
Also try
$ adb shell device_config list
Bonus: device_config also allows to set the listed parameters, and these settings survive reboot (generally speaking).
List all installed packages:
$ adb shell cmd package list packages > packages.txt
Remove all user data for a package (at adb prompt, com.tencent.mm is WeChat)
pm clear com.tencent.mm
Where apps keep their data
For example, Whatsapp:
- /data/data/com.whatsapp/
- /mnt/installer/0/emulated/0/Android/data/com.whatsapp
The sensitive stuff is in the former. The latter may be accessible to anyone.
SSH session with simpleSSHD
This is a no-cost app, which is essentially the Dropbear server.
Note: It’s also possible to connect with “adb shell” through a plain USB cable, and there’s “adb push” and “adb pull” to transfer files. So the advantage of SSH is limited.
Having SimpleSSHD running and started on the phone, I connected with the address provided on the screen
$ ssh -p 2222 user@10.11.12.13
the password appears on the phone’s screen. To use automatic login, go
cat > ~/authorized_keys
and copy-paste the content of ~/.ssh/id_rsa.pub there. Note that the file on the phone is not under a .ssh/ directory, which is probably why the ssh-copy-id utility doesn’t cut. Note however that once this file is found, a password login is not attempted if the host’s public key doesn’t match, so now try to log in from a computer that doesn’t have one of the listed keys.
In principle, the available executables are in /system/bin. The path contains more directories, but this is the only effective one.
Who am I?
$ id
uid=10225(u0_a225) gid=10225(u0_a225) groups=10225(u0_a225),3003(inet),9997(everybody),20225(u0_a225_cache),50225(all_a225) context=u:r:untrusted_app_29:s0:c225,c256,c512,c768
Backing up directly from phone (command run on receiving host):
$ ssh -p 2222 user@10.11.12.13 tar -cvz /storage/emulated/0/
The execution path…?
$ echo $PATH
/sbin:/system/sbin:/system/bin:/system/xbin
Unfortunately, these directories aren’t readable as a regular user, so it’s impossible to do a plain “ls” on them. Just in case I was looking for excuses to root the phone.
“more” also works fine, but there’s no “less”.
“top” works nicely, but shows only the same user’s processes. There’s also “ps”, but it seems to do the same. But hey, I rooted the phone. So
$ su
# id
uid=0(root) gid=0(root) groups=0(root) context=u:r:magisk:s0
And that’s the reason I rooted the phone, actually.
Introduction
These are my notes to future self on rooting a Google Pixel 6 Pro. For reasons I’ll explain below, there’s a good chance I’ll find interest in this topic every now and then.
Being a Linux guy, who isn’t very fond of smartphones in general, I consider my Android phone a small Linux computer, and as such, I can’t stand the idea of not having full control over it. I own the thing, therefore I should be able to access any data on it.
There are also practical aspects to this, for example to prevent system updates and the ability to obtain the secret key that encrypts WhatsApp data, so the conversations can be properly saved for future access (that is, long after WhatsApp becomes history).
So all I want is “su” at shell prompt to turn me into root. So I can backup the entire thing with a tar command, for example. I couldn’t care less about tweaking the phone.
I should say that even though I definitely have experience with embedded Linux, my knowledge on Android is very limited, and consists of what I chose to know in order to maintain my own phone. Nothing professional, in short.
I’ve also written a similar post on my LG G4, which I eventually didn’t root.
Root your phone early
Or at least, unlock the bootloader early, because it resets the phone completely, including deleting all user data. There’s a bit of sense in doing this, because unlocking the bootloader can be a means to maliciously access the data on the phone. For us plain users, it means that the rooting should be done as part of the initial setup of the phone.
It also means that there’s no going back and forth. So if the phone is unrooted and subsequently locked for the purpose of making some sensitive app happy, there’s no going back to root. At least not without wiping the phone.
Overview of rooting
Android 12 is best rooted with Magisk. Previously, SuperSU was popular for this purpose, but as apps become more and more hostile to rooted phones due to security and DRM considerations, it’s also required to hide the fact that the phone has been rooted from these apps. In particular, apps involving payments and access to paid-for content (e.g. Netflix) won’t play ball if the phone appears to have been tampered with. SuperSU is not in the league for this kind of cat-and-mouse game.
Android is designed to never give root access to anything else but its own low-level machinery. suid doesn’t work, and Android also runs with SELinux in enforcing state, which I suppose also prevents privilege escalation.
The intervention is therefore done by Magisk when the Linux kernel boots. The trick is to boot up with an initial RAM disk image that is manipulated to bring up the system in a more permissive state.
Practically, the rooting procedure goes like this:
- Unlock the bootloader. What this means is that the bootloader that is responsible for loading the kernel and ramdisk image (a.k.a. fastboot) also agrees to write images to the boot partitions of the flash, something it normally refuses to do. Unlocking should be done early, because it resets the phone.
- Download (to computer) the boot images for the exact Android build that runs on the phone, and fetch the boot.img image file from it. It contains the Linux kernel and ramdisk for a normal boot (as opposed to e.g. Recovery boot).
- Sideload-install the Magisk app (i.e. install from an .apk file directly) on the phone.
- Feed the Magisk app with the boot.img file, so it creates a manipulated version of it (this runs on the phone itself).
- Boot the phone into fastboot mode and write the manipulated image to the phone’s boot partition. This will only work if the phone is “bootloader unlocked”, of course.
- Install and enable a Magisk module that camouflages the fact that the phone has been unlocked and rooted, to deceive apps into thinking it’s innocent.
- Disable system updates. See below.
A computer with the fastboot utility installed is required.
I’ve mainly followed the official installation guide. There’s also an XDA thread specific for P6P, but it’s quite confusing as it lists all possible rooting options, and to me it first appeared like I had to run through them all.
The beauty of this rooting process is that the only thing that is specific to the phone model is the boot.img, which is downloaded from the vendor’s site. In other words, those developing the rooting tools don’t release files for every single device. Magisk just modifies the existing boot image in a certain way.
For some background on Magisk by its author, look here.
Implications of rooting
Rooting a Pixel 6 Pro doesn’t void its warranty as far as I understand, but there are three main things to keep in mind:
- Because the boot image is replaced with a manipulated one, a system upgrade will mess up things, or at the very least revert the rooting. Android can’t therefore not be allowed to update automatically, as it usually does. Likewise, the regular “System update” feature (under Settings > System) should never be used. More on that below.
- As already mentioned, some apps don’t want to work on a manipulated phone, so there’s an extra layer of fooling those apps into thinking the phone is OK. It’s quite likely that these apps will improve their detection of manipulated phones, and that the countermeasures will keep improving too. So this is probably going to be an eternal Tom and Jerry.
- There will always be a warning that the phone is unlocked every time you boot. No way around this.
Note that there are two aspects of rooting that these sensitive apps don’t like: The fact that the phone is unlocked and hence more susceptible to manipulation, and the fact that the boot image is manipulated. Plus, they may also detect the countermeasures for making the phone look innocent.
There’s a framework called Google SafetyNet, which is the main mechanism that apps are expected to use to detect a manipulated phone. The good news is that it’s easy to check if the phone passes its tests (that is, if our camouflage works), but I will not be surprised if each app will have its own additional tests.
OK, enough with the talky talk. Time for some hands-on.
Unlocking
As one would expect, the phone is locked by default. This means that with the phone in fastboot mode (see below how to invoke it), it says so on the screen, and with the fastboot utility one gets:
$ fastboot flashing get_unlock_ability
...
(bootloader) get_unlock_ability: 0
OKAY [ 0.030s]
finished. total time: 0.030s
$ fastboot flashing unlock
...
FAILED (remote: flashing unlock is not allowed)
finished. total time: 0.032s
So boot the phone normally. If there is no Developer options item under System, the phone is not in developer mode. For this, go to About Phone. Tap seven times on Build Number. The phone prompts for the PIN number, and then the “You are now a developer” message appears.
Go to the phone’s settings, System > Developer options > OEM unlocking, and enable it (a PIN prompt and scary warning dialog box will appear).
Then back to fastboot mode. It’s time to issue the “flashing unlock: command which resets the phone completely. All user data and settings is lost.
$ fastboot flashing get_unlock_ability
...
(bootloader) get_unlock_ability: 1
OKAY [ 0.030s]
finished. total time: 0.030s
$ fastboot flashing unlock
...
OKAY [ 0.032s]
finished. total time: 0.032s
Even though the command finished immediately, the unlock occurs only after confirming it on the phone itself, with a scary warning message. And then it says “unlocked” next to the device state on the screen in fastboot mode.
The phone then starts exactly like it did the first time it was powered on. Completely blank.
Plus a warning on every boot that the bootloader is unlocked. Something to get used to, as there’s no remedy for this. Re-locking the phone is out of the question, because unlocking it back will wipe it again.
Sideload install of Magisk app
Download the latest version (v24.3 in my case) from here. Copy Magisk-v24.3.apk into some reachable directory on the phone (File Transfer, connected via USB, or any other method).
Then open Files on the phone, and navigate to the Download folder. Tap the .ask file, but alas, the phone refuses to “install unknown apps”. But hey, there’s a “Settings” option on the popup. So enable “Allow from this source”, and that’s it.
Patch the boot image
The original Android images can be downloaded from Google’s official page. So I picked the raven-xxx-factory-whatever.zip file that matched my build (find the exact build number in About Phone in the Settings).
Fetch the boot.img from the zip file inside the zip file, and copy it into the phone to somewhere reachable as well.
Now open the Magisk app, tap the Install button, and pick “Select and Patch a file” (there’s no other option, actually). Navigate to and select boot.img, and tap “let’s go”. The screen turns into a black text mode, some log messages appear, and then it says where it wrote the patched file to.
Copy the patched file to the computer, possibly with USB file transfer.
Note that the boot.img must come from a build that matches the one running on the phone exactly. The truth is that I would expect things to work fairly well even if there was a slight version mismatch, but there’s also a possibility for some unexplained glitches.
Therefore, if you want to upgrade Android, do so by downloading the factory build for the new version, and manipulate the boot.img file in the zip file (actually, it’s in the zip file inside the zip file). And then run the script that comes with that file (if you have the courage).
Invoking fastboot mode
Turn off the phone. Press and hold the volume down button and then the power button, and it’s up in no time.
The nice thing is that other modes can be invoked from the menu that shows up (Rescue and Recovery mode)
Write the image to the boot partition
It’s time to write the image to the phone. This is where there’s a possibility to brick the phone.
That said, boot.img is the kernel and ramdisk, not the initial bootloader (i.e. fastboot). In other words, if this goes wrong, the phone might not boot up, but since the bootloader is still intact, there’s always the possibility to power it up in fastboot mode and update any partition.
So the only partition I would be really reluctant to write to is the bootloader itself. Which isn’t the same as boot.
So “flash boot” sounds scary, but it’s not a dead end if it fails. There’s already a factory image zip file at hand for writing to all relevant partitions, and it has a flash-all.sh file that writes a complete set (including the bootloader, by the way, maybe delete that line from the script).
So let’s do the scary thing (the file name of the image may vary, of course):
$ fastboot flash boot ~/Desktop/magisk_patched-24300_28q74.img
target reported max download size of 261095424 bytes
sending 'boot_a' (65536 KB)...
OKAY [ 0.413s]
writing 'boot_a'...
OKAY [ 0.100s]
finished. total time: 0.513s
As it says, the whole thing took half a second. Reboot. The thing that stands out is that Magisk has a button for uninstallation.
And by the way, it would probably have been fine to write to the recovery partition instead. Possibly as a method to boot the phone with root capabilities occasionally. But that doesn’t calm those sensitive apps, because the phone remains unlocked. And locking it means no way back to unlock without a full wipe.
It works!
Opened a shell via adb, typed “su”. Magisk popped up a prompt on the screen to authorize root privileges, I approved, and that’s it.
Same goes for every other app that requests root privileges. Magisk brings up a popup for authorizing it. So unless there’s a vulnerability in Magisk, there’s no security hole about this.
Making SafetyNet happy
So now when the phone is rooted, it’s time to sweep the dirt under the carpet.
I downloaded SafetyNet Checker, and indeed, it failed on Basic Integrity and CTS Profile Match. The suggestions read “RESTORE_TO_FACTORY_ROM” and “LOCK_BOOTLOADER” so hey, I definitely got caught.
So there’s safetynet-fix. I downloaded v2.2.1 from here, copied safetynet-fix-v2.2.1.zip into the phone.
Then, within the Magisk app on the phone, I selected Modules at the bottom, and picked “Install from Storage”. And picked the same zip file. The screen went to log text on black screen, indicating everything was fine.
And from there, I clicked the Reboot button to the bottom right.
Ah, but that wasn’t enough. After the reboot, I went to Modules again, and it said “Modules suspended because Zygisk in not enabled”. And almost needless to say, the SafetyNet test failed.
So I clicked the Home button to the bottom left of Magisk, then the setting icon to the top right, and enabled Zygisk (Beta). And rebooted again.
This time the SafetyNet test succeeded. Another good sign is that the OEM bootloader unlocking option was back — it disappears when the systems detects that the bootloader is unlocked.
But wait, what is this Zygisk thing? To get an idea, let’s first let’s get acquainted with a standard Android component, the Zygote process: It’s the counterpart of common Linux’ init process, in the sense that it’s the father of all processes in the system. Likewise, all processes in the Dalvik VM are the children of the Zygote process. Every new process forks from this one.
Zygisk replaces this process. As I’m not familiar with Android internals, I don’t know what tricks Zygisk pulls, but it’s a common trick in Linux to hijack calls to libraries when launching a new process. So at least the good old Linux trick is to, for example, hijack all library functions that are related to time, and implement a date crack. I guess they did something similar to feed all apps with fake information about the phone. Odds are that the apps will add checks to bypass those that are fake, and then the fix module will cover up these two. Tom and Jerry.
Update 7.10.22: Suddenly, the app of Israeli Post’s authority requested an update, and after the update it refused to run, because it won’t run on a “compromised device” (screenshot of the popup below). I upgraded safetynet-fix to 2.3.1 (latest). Safetynet was happy even before that upgrade, and surely after it, but Israeli Post will still not play ball:
System updates
By default, Android updates itself every now and then by virtue of OTA (Over The Air) updates, which essentially means that the phone downloads a zip file that contains images that are written to several partitions on the flash, the boot partition included (and the fastboot partition too, actually). So if such update is allowed to take place normally, the manipulated boot.img is simply not used anymore, and the phone reboots unrooted. It will still be possible to repeat the root procedure to re-root the phone, as it remains unlocked.
So these automatic updates must be disabled so the phone doesn’t lose its root capabilities all of the sudden. It’s possible to perform manual updates, either by applying a manipulated zip file and write it to the partitions in fastboot mode, or with the method described here and here.
Personally, I’m a very little fan of upgrading things all the time, so odds are that I would have turned off System Upgrade anyhow. I mean, you restart your phone for whatever reason, and poof, something stops working because bug fix X caused a new bug Y. But don’t worry, it will be fixed in the next update, and then there will be bug Z. Regardless of rooting. And I know, security updates and all that. It boils down to either risking getting knocked down by an attack on your phone or by a software update. I would say the latter is much more likely. As for privacy, well, that’s gone long ago, and mainly due to what the phone does per design.
On Android 12, turning off system updates is a bit like turning down a nagging salesman. These are the things I did, and I’m not sure all of them were necessary:
- Under Developer Options (which are enabled for unlocking anyhow). Turn off “Automatic system updates”.
- Go to Settings > Notifications > App Notifications > Google Play Settings and turn off System Update (to silence the “System update paused” notification).
- Go to the Settings > Apps > Google Play Services > Storage and Cache > Clear cache. Possibly also tap “Manage space” > “clear all data”. Not sure either of these two were necessary.
- Disable a few Activities in Android’s guts, as described in this other post of mine. This is the trickiest part, and requires adb.
And from now on, avoid System > System Update (like, why insist?).
Disabling Acitivities is the hardest part to perform, but if you don’t do this, you’ll get popups like this, which evenutally escalating to the phone telling you it’s going to update itself whether you like it or not:
The hot/cold system images for updates
And in case an update is done deliberately: Pixel 6 Pro does System Update with an A/B setting. That means that there’s an “active slot”, A or B, that the phone boots from, and an “inactive slot”, which the updating utilities are free to write to. So the updating process consists of writing the downloaded images into the inactive slot, and then turn the inactive slot into the active one. The existence of A/B partitions is evident by listing the block devices by name:
raven:/ # ls /dev/block/by-name/
abl_a dpm_a gsa_b pbl_b userdata
abl_b dpm_b klog persist vbmeta_a
bl1_a dram_train_a ldfw_a pvmfw_a vbmeta_b
bl1_b dram_train_b ldfw_b pvmfw_b vbmeta_system_a
bl2_a dtbo_a metadata sda vbmeta_system_b
bl2_b dtbo_b mfg_data sdb vbmeta_vendor_a
bl31_a efs misc sdc vbmeta_vendor_b
bl31_b efs_backup modem_a sdd vendor_boot_a
boot_a fips modem_b super vendor_boot_b
boot_b frp modem_userdata tzsw_a
devinfo gsa_a pbl_a tzsw_b
So when A is active, it means that the bootloader uses the *_a partitions, and same goes with B.
I haven’t done any update on my phone yet, so I can’t say anything from personal experience. But in the end, it seems like if a System Update occurs accidentally, it’s not a big deal — the phone will probably start just fine, only with an updated version of Android and without root. So it’s just a matter of going through the rooting process again.
Or better still, if the update is made in a controlled manner, Magisk can manipulate the boot_a or boot_b partition — the one that inactive at that moment — after the regular OTA update, and hence make the device remain rooted when it reboots. That’s why the instructions for this process tell you not to reboot the phone after performing an OTA update.
To drop the resumption notification, possibly remove the already downloaded stuff from the Download/ directory. Apparently, Google Play is taking care of that.
That’s it
For the sake of rooting the phone, there’s no point reading further. Everything that follows is some more or less related information, but nothing necessary to complete the task.
Boot modes
This is kind-of related: The phone has several boot modes, so here’s a brief list of those I know about, along with their Vendor / Product IDs.
- Regular MTP mode. With no data transfer it’s 18d1:4ee7, in File Transfer mode it’s 18d1:4ee2, in tethering (Internet access share) it’s 18d1:4eec, MIDI mode 18d1:4ee9 and in PTP it’s 18d1:4ee6.
- Rescue mode: 18d1:d001
- Fastboot mode, 18d1:4ee0
- Recovery mode, 18d1:4ee0
Fastboot mode
This is like U-boot on embedded Linux (and maybe it’s actually that, I haven’t checked). So it’s the first thing that is loaded from the flash memory. So the point is that as long as Fastboot works and the phone is unlocked, the phone isn’t really bricked even if it doesn’t boot, because everything can be fixed. If Fastboot doesn’t come up, get some mortar.
Recovery mode
Press and hold the the power button, and press the volume up button briefly. The flipped robot image appears.
Going to fastboot mode from there enables fastbootd, which appears to talk with the PC with a partial version of the fastboot protocol.
It’s also possible to view the recovery log (as /tmp/recovery.log) but it’s not clear what it helps for.
Getting out of this mode — I guess there are ways for that, maybe through fastbootd? So far I just waited long enough.
Rescue mode
In rescue mode, the phone presents itself as a host:
$ adb devices
List of devices attached
12345FDEE000S9 host
Not sure what that means, actually. Is it really a USB host? If so, how did adb manage to query it?
The screen shows the flipped robot and “No command”, just like Recovery Mode.
But if you press the power and volume up in this mode, it says “waiting for rescue commands…”.
Now try to get out of that. Rumor has it that pressing the power button for 20 seconds will do the trick. Or just wait long enough. Alternatively, connect to the computer with the USB plug and go
$ adb reboot
There’s also this page discussing Rescue Mode. My best guess is that this mode is intended to allow writing to flash partitions without unlocking the phone, or else how can authorized personnel can fix a phone with a messed up boot chain without taking it apart? So this could be the key to rooting the phone without unlocking it. Maybe.
Mounts
Checking the mounts of a Linux system is one of the first things I do when I begin exploring a system. It turns out even more interesting in the context of rooting.
These are the mounts before rooting the phone with Magisk. That’s one heck of a list on this machine even at this state:
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/block/dm-6 838M 836M 0 100% /
tmpfs 5.6G 1.2M 5.6G 1% /dev
tmpfs 5.6G 0 5.6G 0% /mnt
/dev/block/dm-7 237M 236M 0 100% /system_ext
/dev/block/dm-8 2.5G 2.5G 0 100% /product
/dev/block/dm-9 455M 454M 0 100% /vendor
/dev/block/dm-10 52M 52M 0 100% /vendor_dlkm
tmpfs 5.6G 52K 5.6G 1% /apex
/dev/block/loop4 35M 35M 0 100% /apex/com.android.i18n@1
/dev/block/loop7 232K 24K 204K 11% /apex/com.android.telephony@1
/dev/block/dm-13 732K 704K 16K 98% /apex/com.android.ipsec@311612010
/dev/block/dm-12 2.7M 2.7M 0 100% /apex/com.android.tethering@311616020
/dev/block/loop10 7.5M 7.5M 0 100% /apex/com.android.runtime@1
/dev/block/dm-14 18M 18M 0 100% /apex/com.android.media.swcodec@311610000
/dev/block/loop11 1.6M 1.6M 0 100% /apex/com.google.mainline.primary.libs@311020000
/dev/block/dm-15 6.8M 6.7M 0 100% /apex/com.android.adbd@311611002
/dev/block/loop13 2.0M 1.9M 0 100% /apex/com.android.appsearch@300000000
/dev/block/loop14 232K 88K 140K 39% /apex/com.android.apex.cts.shim@1
/dev/block/dm-16 1.6M 1.6M 0 100% /apex/com.google.mainline.primary.libs@311626000
/dev/block/dm-18 832K 804K 12K 99% /apex/com.android.tzdata@311311000
/dev/block/dm-17 232K 152K 76K 67% /apex/com.android.scheduling@310733000
/dev/block/loop20 40M 40M 0 100% /apex/com.android.vndk.v31@1
/dev/block/dm-19 4.8M 4.7M 0 100% /apex/com.android.conscrypt@310911000
/dev/block/dm-20 6.0M 6.0M 0 100% /apex/com.android.extservices@311610040
/dev/block/dm-21 62M 62M 0 100% /apex/com.android.art@311510000
/dev/block/dm-23 7.8M 7.7M 0 100% /apex/com.android.neuralnetworks@311610000
/dev/block/dm-22 13M 13M 0 100% /apex/com.android.permission@311611030
/dev/block/dm-24 12M 12M 0 100% /apex/com.android.cellbroadcast@311612020
/dev/block/loop27 256M 134M 117M 54% /apex/com.google.pixel.camera.hal@208030436
/dev/block/dm-25 6.3M 6.3M 0 100% /apex/com.android.media@311610000
/dev/block/dm-26 8.3M 8.3M 0 100% /apex/com.android.mediaprovider@311616020
/dev/block/dm-27 1.8M 1.7M 0 100% /apex/com.android.os.statsd@311510000
/dev/block/dm-28 3.7M 3.7M 0 100% /apex/com.android.resolv@311610020
/dev/block/dm-29 5.5M 5.5M 0 100% /apex/com.android.wifi@311612010
/dev/block/dm-30 704K 676K 16K 98% /apex/com.android.sdkext@311511000
/dev/fuse 110G 14G 97G 13% /storage/emulated
and mounts:
$ mount
/dev/block/dm-6 on / type ext4 (ro,seclabel,relatime)
tmpfs on /dev type tmpfs (rw,seclabel,nosuid,relatime,mode=755)
devpts on /dev/pts type devpts (rw,seclabel,relatime,mode=600,ptmxmode=000)
none on /dev/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
none on /dev/cpuctl type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
none on /dev/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset,noprefix,release_agent=/sbin/cpuset_release_agent)
binder on /dev/binderfs type binder (rw,relatime,max=1048576,stats=global)
adb on /dev/usb-ffs/adb type functionfs (rw,relatime)
mtp on /dev/usb-ffs/mtp type functionfs (rw,relatime)
ptp on /dev/usb-ffs/ptp type functionfs (rw,relatime)
proc on /proc type proc (rw,relatime,gid=3009,hidepid=invisible)
sysfs on /sys type sysfs (rw,seclabel,relatime)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
none on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,seclabel,relatime)
none on /sys/fs/fuse/connections type fusectl (rw,relatime)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,seclabel,nosuid,nodev,noexec,relatime)
tmpfs on /mnt type tmpfs (rw,seclabel,nosuid,nodev,noexec,relatime,mode=755,gid=1000)
tmpfs on /mnt/installer type tmpfs (rw,seclabel,nosuid,nodev,noexec,relatime,mode=755,gid=1000)
/dev/fuse on /mnt/installer/0/emulated type fuse (rw,lazytime,nosuid,nodev,noexec,noatime,user_id=0,group_id=0,allow_other)
/dev/block/dm-11 on /mnt/installer/0/emulated/0/Android/data type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/dm-11 on /mnt/installer/0/emulated/0/Android/obb type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
tmpfs on /mnt/androidwritable type tmpfs (rw,seclabel,nosuid,nodev,noexec,relatime,mode=755,gid=1000)
/dev/fuse on /mnt/androidwritable/0/emulated type fuse (rw,lazytime,nosuid,nodev,noexec,noatime,user_id=0,group_id=0,allow_other)
/dev/block/dm-11 on /mnt/androidwritable/0/emulated/0/Android/data type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/dm-11 on /mnt/androidwritable/0/emulated/0/Android/obb type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/platform/14700000.ufs/by-name/persist on /mnt/vendor/persist type ext4 (rw,seclabel,nosuid,nodev,noatime,nodioread_nolock,nodelalloc,commit=1,data=journal)
/dev/block/platform/14700000.ufs/by-name/efs on /mnt/vendor/efs type ext4 (rw,seclabel,noatime)
/dev/block/platform/14700000.ufs/by-name/efs_backup on /mnt/vendor/efs_backup type ext4 (rw,seclabel,noatime)
/dev/block/platform/14700000.ufs/by-name/modem_userdata on /mnt/vendor/modem_userdata type ext4 (rw,seclabel,noatime)
/dev/block/platform/14700000.ufs/by-name/modem_a on /mnt/vendor/modem_img type ext4 (ro,context=u:object_r:modem_img_file:s0,relatime)
/dev/fuse on /mnt/user/0/emulated type fuse (rw,lazytime,nosuid,nodev,noexec,noatime,user_id=0,group_id=0,allow_other)
/dev/block/dm-11 on /mnt/user/0/emulated/0/Android/data type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/dm-11 on /mnt/user/0/emulated/0/Android/obb type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/dm-11 on /mnt/pass_through/0/emulated type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/platform/14700000.ufs/by-name/metadata on /metadata type ext4 (rw,seclabel,nosuid,nodev,noatime,nodioread_nolock,nodelalloc,commit=1,data=journal)
/dev/block/dm-7 on /system_ext type ext4 (ro,seclabel,relatime)
/dev/block/dm-8 on /product type ext4 (ro,seclabel,relatime)
/dev/block/dm-9 on /vendor type ext4 (ro,seclabel,relatime)
/dev/block/dm-10 on /vendor_dlkm type ext4 (ro,seclabel,relatime)
tmpfs on /apex type tmpfs (rw,seclabel,nosuid,nodev,noexec,relatime,mode=755)
/dev/block/loop4 on /apex/com.android.i18n@1 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop4 on /apex/com.android.i18n type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop7 on /apex/com.android.telephony@1 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop7 on /apex/com.android.telephony type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-13 on /apex/com.android.ipsec@311612010 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-13 on /apex/com.android.ipsec type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-12 on /apex/com.android.tethering@311616020 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop10 on /apex/com.android.runtime@1 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-12 on /apex/com.android.tethering type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop10 on /apex/com.android.runtime type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-14 on /apex/com.android.media.swcodec@311610000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-14 on /apex/com.android.media.swcodec type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop11 on /apex/com.google.mainline.primary.libs@311020000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-15 on /apex/com.android.adbd@311611002 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-15 on /apex/com.android.adbd type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop13 on /apex/com.android.appsearch@300000000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop13 on /apex/com.android.appsearch type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop14 on /apex/com.android.apex.cts.shim@1 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop14 on /apex/com.android.apex.cts.shim type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-16 on /apex/com.google.mainline.primary.libs@311626000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-18 on /apex/com.android.tzdata@311311000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-17 on /apex/com.android.scheduling@310733000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-17 on /apex/com.android.scheduling type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-18 on /apex/com.android.tzdata type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop20 on /apex/com.android.vndk.v31@1 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-19 on /apex/com.android.conscrypt@310911000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop20 on /apex/com.android.vndk.v31 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-19 on /apex/com.android.conscrypt type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-20 on /apex/com.android.extservices@311610040 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-20 on /apex/com.android.extservices type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-21 on /apex/com.android.art@311510000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-21 on /apex/com.android.art type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-23 on /apex/com.android.neuralnetworks@311610000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-22 on /apex/com.android.permission@311611030 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-23 on /apex/com.android.neuralnetworks type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-22 on /apex/com.android.permission type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-24 on /apex/com.android.cellbroadcast@311612020 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-24 on /apex/com.android.cellbroadcast type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop27 on /apex/com.google.pixel.camera.hal@208030436 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop27 on /apex/com.google.pixel.camera.hal type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-25 on /apex/com.android.media@311610000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-25 on /apex/com.android.media type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-26 on /apex/com.android.mediaprovider@311616020 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-27 on /apex/com.android.os.statsd@311510000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-27 on /apex/com.android.os.statsd type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-26 on /apex/com.android.mediaprovider type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-28 on /apex/com.android.resolv@311610020 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-28 on /apex/com.android.resolv type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-29 on /apex/com.android.wifi@311612010 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-29 on /apex/com.android.wifi type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-30 on /apex/com.android.sdkext@311511000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-30 on /apex/com.android.sdkext type ext4 (ro,dirsync,seclabel,nodev,noatime)
tmpfs on /apex/apex-info-list.xml type tmpfs (rw,seclabel,nosuid,nodev,noexec,relatime,mode=755)
tmpfs on /linkerconfig type tmpfs (rw,seclabel,nosuid,nodev,noexec,relatime,mode=755)
tmpfs on /linkerconfig type tmpfs (rw,seclabel,nosuid,nodev,noexec,relatime,mode=755)
none on /config type configfs (rw,nosuid,nodev,noexec,relatime)
/dev/block/dm-11 on /data type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/dm-11 on /data/user/0 type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
tmpfs on /data_mirror type tmpfs (rw,seclabel,nosuid,nodev,noexec,relatime,mode=700,gid=1000)
/dev/block/dm-11 on /data_mirror/data_ce/null type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/dm-11 on /data_mirror/data_ce/null/0 type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/dm-11 on /data_mirror/data_de/null type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/dm-11 on /data_mirror/cur_profiles type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/dm-11 on /data_mirror/ref_profiles type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
tmpfs on /storage type tmpfs (rw,seclabel,nosuid,nodev,noexec,relatime,mode=755,gid=1000)
/dev/fuse on /storage/emulated type fuse (rw,lazytime,nosuid,nodev,noexec,noatime,user_id=0,group_id=0,allow_other)
/dev/block/dm-11 on /storage/emulated/0/Android/data type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/dm-11 on /storage/emulated/0/Android/obb type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
Most notable are /dev/fuse on /mnt/androidwritable/0/emulated. Like, really? All file access goes through FUSE? Anyhow, the user visible data root is at /storage/emulated/0/, as usual.
Also interesting is how adb, mtp and ptp are mounted as /dev/usb-ffs/* as functionfs. So this is how the phone behaves as a USB gadget.
Mounts after rooting
This wouldn’t normally be so interesting, but it so happens that Magisk plays a lot with mount games. So here’s the df -h, as a regular user on a rooted phone:
Filesystem Size Used Avail Use% Mounted on
/dev/block/dm-0 838M 836M 0 100% /
tmpfs 5.6G 1.2M 5.6G 1% /dev
tmpfs 5.6G 5.3M 5.6G 1% /dev/PNsr9c
tmpfs 5.6G 0 5.6G 0% /mnt
/dev/block/dm-2 237M 236M 0 100% /system_ext
/dev/block/dm-3 2.5G 2.5G 0 100% /product
/dev/block/dm-4 455M 454M 0 100% /vendor
/dev/block/dm-5 52M 52M 0 100% /vendor_dlkm
tmpfs 5.6G 28K 5.6G 1% /apex
/dev/block/dm-6 110G 9.1G 101G 9% /data
/dev/block/loop4 2.6M 2.6M 0 100% /apex/com.android.tethering@311011010
/dev/block/loop5 35M 35M 0 100% /apex/com.android.i18n@1
/dev/block/loop6 232K 24K 204K 11% /apex/com.android.telephony@1
/dev/block/loop7 696K 668K 16K 98% /apex/com.android.ipsec@311010000
/dev/block/loop8 18M 18M 0 100% /apex/com.android.media.swcodec@311015000
/dev/block/loop10 7.5M 7.5M 0 100% /apex/com.android.runtime@1
/dev/block/loop9 6.7M 6.7M 0 100% /apex/com.android.adbd@310852002
/dev/block/loop12 2.0M 1.9M 0 100% /apex/com.android.appsearch@300000000
/dev/block/loop11 1.6M 1.6M 0 100% /apex/com.google.mainline.primary.libs@311020000
/dev/block/loop13 232K 88K 140K 39% /apex/com.android.apex.cts.shim@1
/dev/block/loop15 832K 804K 12K 99% /apex/com.android.tzdata@310733000
/dev/block/loop14 232K 152K 76K 67% /apex/com.android.scheduling@310733000
/dev/block/loop16 4.8M 4.7M 0 100% /apex/com.android.conscrypt@310911000
/dev/block/loop18 60M 60M 0 100% /apex/com.android.art@310924000
/dev/block/loop19 40M 40M 0 100% /apex/com.android.vndk.v31@1
/dev/block/loop20 18M 18M 0 100% /apex/com.android.permission@311014000
/dev/block/loop17 5.8M 5.8M 0 100% /apex/com.android.extservices@310852000
/dev/block/loop23 10M 10M 0 100% /apex/com.android.cellbroadcast@311010000
/dev/block/loop21 7.6M 7.6M 0 100% /apex/com.android.neuralnetworks@310850000
/dev/block/loop24 1.8M 1.7M 0 100% /apex/com.android.os.statsd@311012000
/dev/block/loop22 6.3M 6.2M 0 100% /apex/com.android.media@311012000
/dev/block/loop25 4.2M 4.2M 0 100% /apex/com.android.mediaprovider@311012020
/dev/block/loop26 256M 134M 117M 54% /apex/com.google.pixel.camera.hal@208030436
/dev/block/loop27 3.6M 3.6M 0 100% /apex/com.android.resolv@311011010
/dev/block/loop28 5.6M 5.5M 0 100% /apex/com.android.wifi@311011000
/dev/block/loop29 692K 664K 16K 98% /apex/com.android.sdkext@310912000
tmpfs 5.6G 0 5.6G 0% /system/bin
/dev/fuse 110G 9.1G 101G 9% /storage/emulated
So it’s more or less like before rooting, but with a couple of clues on what’s happening: There’s some odd tmpfs mounted as /dev/PNsr9c, and /system/bin is a tmpfs as well. Hmmm. That’s suspicious.
The same df -h command as root reveals a lot of /dev/block/loopNN mounts at /apex/*. They are all 99-100%, as they aren’t really storage. I’ll skip the output of that, because they’re all listed in the regular mount.
Which reads like this after rooting. Hold tight:
/dev/block/dm-0 on / type ext4 (ro,seclabel,relatime)
tmpfs on /dev type tmpfs (rw,seclabel,nosuid,relatime,mode=755)
devpts on /dev/pts type devpts (rw,seclabel,relatime,mode=600,ptmxmode=000)
tmpfs on /dev/PNsr9c type tmpfs (rw,seclabel,relatime,mode=755)
proc on /proc type proc (rw,relatime,gid=3009,hidepid=invisible)
sysfs on /sys type sysfs (rw,seclabel,relatime)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
tmpfs on /mnt type tmpfs (rw,seclabel,nosuid,nodev,noexec,relatime,mode=755,gid=1000)
tmpfs on /mnt/installer type tmpfs (rw,seclabel,nosuid,nodev,noexec,relatime,mode=755,gid=1000)
tmpfs on /mnt/androidwritable type tmpfs (rw,seclabel,nosuid,nodev,noexec,relatime,mode=755,gid=1000)
/dev/block/sda8 on /metadata type ext4 (rw,seclabel,nosuid,nodev,noatime,nodioread_nolock,nodelalloc,commit=1,data=journal)
/dev/block/dm-2 on /system_ext type ext4 (ro,seclabel,relatime)
/dev/block/dm-3 on /product type ext4 (ro,seclabel,relatime)
/dev/block/dm-4 on /vendor type ext4 (ro,seclabel,relatime)
/dev/block/dm-5 on /vendor_dlkm type ext4 (ro,seclabel,relatime)
tmpfs on /apex type tmpfs (rw,seclabel,nosuid,nodev,noexec,relatime,mode=755)
tmpfs on /linkerconfig type tmpfs (rw,seclabel,nosuid,nodev,noexec,relatime,mode=755)
none on /dev/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
none on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
none on /dev/cpuctl type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
none on /dev/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset,noprefix,release_agent=/sbin/cpuset_release_agent)
tracefs on /sys/kernel/tracing type tracefs (rw,seclabel,relatime)
/dev/block/sda1 on /mnt/vendor/persist type ext4 (rw,seclabel,nosuid,nodev,noatime,nodioread_nolock,nodelalloc,commit=1,data=journal)
none on /config type configfs (rw,nosuid,nodev,noexec,relatime)
binder on /dev/binderfs type binder (rw,relatime,max=1048576,stats=global)
none on /sys/fs/fuse/connections type fusectl (rw,relatime)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,seclabel,nosuid,nodev,noexec,relatime)
/dev/block/sda5 on /mnt/vendor/efs type ext4 (rw,seclabel,noatime)
/dev/block/sda6 on /mnt/vendor/efs_backup type ext4 (rw,seclabel,noatime)
/dev/block/sda7 on /mnt/vendor/modem_userdata type ext4 (rw,seclabel,noatime)
/dev/block/sda12 on /mnt/vendor/modem_img type ext4 (ro,context=u:object_r:modem_img_file:s0,relatime)
tmpfs on /storage type tmpfs (rw,seclabel,nosuid,nodev,noexec,relatime,mode=755,gid=1000)
/dev/block/dm-6 on /data type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
tmpfs on /linkerconfig type tmpfs (rw,seclabel,nosuid,nodev,noexec,relatime,mode=755)
/dev/block/loop4 on /apex/com.android.tethering@311011010 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop5 on /apex/com.android.i18n@1 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop5 on /apex/com.android.i18n type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop4 on /apex/com.android.tethering type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop6 on /apex/com.android.telephony@1 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop6 on /apex/com.android.telephony type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop7 on /apex/com.android.ipsec@311010000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop7 on /apex/com.android.ipsec type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop8 on /apex/com.android.media.swcodec@311015000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop8 on /apex/com.android.media.swcodec type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop10 on /apex/com.android.runtime@1 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop10 on /apex/com.android.runtime type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop9 on /apex/com.android.adbd@310852002 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop9 on /apex/com.android.adbd type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop12 on /apex/com.android.appsearch@300000000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop11 on /apex/com.google.mainline.primary.libs@311020000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop12 on /apex/com.android.appsearch type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop13 on /apex/com.android.apex.cts.shim@1 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop13 on /apex/com.android.apex.cts.shim type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop15 on /apex/com.android.tzdata@310733000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop15 on /apex/com.android.tzdata type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop14 on /apex/com.android.scheduling@310733000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/dm-6 on /data/user/0 type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
tmpfs on /data_mirror type tmpfs (rw,seclabel,nosuid,nodev,noexec,relatime,mode=700,gid=1000)
/dev/block/dm-6 on /data_mirror/data_ce/null type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/dm-6 on /data_mirror/data_ce/null/0 type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/dm-6 on /data_mirror/data_de/null type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/dm-6 on /data_mirror/cur_profiles type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/dm-6 on /data_mirror/ref_profiles type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/loop14 on /apex/com.android.scheduling type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop16 on /apex/com.android.conscrypt@310911000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop16 on /apex/com.android.conscrypt type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop18 on /apex/com.android.art@310924000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop19 on /apex/com.android.vndk.v31@1 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop18 on /apex/com.android.art type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop20 on /apex/com.android.permission@311014000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop19 on /apex/com.android.vndk.v31 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop20 on /apex/com.android.permission type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop17 on /apex/com.android.extservices@310852000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop17 on /apex/com.android.extservices type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop23 on /apex/com.android.cellbroadcast@311010000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop21 on /apex/com.android.neuralnetworks@310850000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop21 on /apex/com.android.neuralnetworks type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop23 on /apex/com.android.cellbroadcast type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop24 on /apex/com.android.os.statsd@311012000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop24 on /apex/com.android.os.statsd type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop22 on /apex/com.android.media@311012000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop25 on /apex/com.android.mediaprovider@311012020 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop22 on /apex/com.android.media type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop25 on /apex/com.android.mediaprovider type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop26 on /apex/com.google.pixel.camera.hal@208030436 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop27 on /apex/com.android.resolv@311011010 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop26 on /apex/com.google.pixel.camera.hal type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop27 on /apex/com.android.resolv type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop28 on /apex/com.android.wifi@311011000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop28 on /apex/com.android.wifi type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop29 on /apex/com.android.sdkext@310912000 type ext4 (ro,dirsync,seclabel,nodev,noatime)
/dev/block/loop29 on /apex/com.android.sdkext type ext4 (ro,dirsync,seclabel,nodev,noatime)
tmpfs on /apex/apex-info-list.xml type tmpfs (rw,seclabel,nosuid,nodev,noexec,relatime,mode=755)
devpts on /dev/PNsr9c/.magisk/pts type devpts (rw,seclabel,nosuid,noexec,relatime,mode=600,ptmxmode=000)
/dev/PNsr9c/.magisk/block/system_ext on /dev/PNsr9c/.magisk/mirror/system_ext type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/product on /dev/PNsr9c/.magisk/mirror/product type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/vendor on /dev/PNsr9c/.magisk/mirror/vendor type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/data on /dev/PNsr9c/.magisk/mirror/data type f2fs (rw,lazytime,seclabel,relatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/PNsr9c/.magisk/block/system_root on /dev/PNsr9c/.magisk/mirror/system_root type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/data on /dev/PNsr9c/.magisk/modules type f2fs (rw,lazytime,seclabel,relatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
tmpfs on /system/bin type tmpfs (rw,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/abb type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/abx type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/abx2xml type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/am type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/apexd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/app_process32 type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/app_process64 type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/appops type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/appwidget type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/atrace type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/audioserver type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/auditctl type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/awk type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/badblocks type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/bc type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/bcc type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/blank_screen type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/blkid type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/bmgr type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/bootanimation type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/bootstat type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/bootstrap type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/boringssl_self_test32 type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/boringssl_self_test64 type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/bpfloader type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/bu type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/bugreport type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/bugreportz type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/bzip2 type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/cameraserver type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/charger type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/clatd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/cmd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/content type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/cppreopts.sh type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/credstore type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/debuggerd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/device_config type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/dmabuf_dump type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/dmctl type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/dnsmasq type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/dpm type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/drmserver type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/dumpstate type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/dumpsys type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/e2freefrag type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/e2fsck type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/e2fsdroid type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/flags_health_check type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/fsck.f2fs type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/fsck_msdos type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/fsverity_init type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/gatekeeperd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/gpuservice type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/gsi_tool type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/gsid type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/heapprofd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/hid type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/hw type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/hwservicemanager type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/idmap2 type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/idmap2d type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/ime type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/incident type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/incident-helper-cmd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/incident_helper type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/incidentd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/init type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/input type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/installd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/iorap.cmd.compiler type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/iorap.cmd.maintenance type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/iorap.inode2filename type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/iorap.prefetcherd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/iorapd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/ip type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/iptables type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/keystore2 type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/keystore_cli_v2 type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/ld.mc type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/ldd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/librank type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/llkd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/lmkd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/locksettings type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/logcat type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/logd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/logwrapper type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/lpdump type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/lpdumpd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/lshal type ext4 (ro,seclabel,relatime)
tmpfs on /system/bin/magisk type tmpfs (rw,seclabel,relatime,mode=755)
tmpfs on /system/bin/magiskinit type tmpfs (rw,seclabel,relatime,mode=755)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/make_f2fs type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/mdnsd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/mediaextractor type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/mediametrics type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/mediaserver type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/migrate_legacy_obb_data.sh type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/mini-keyctl type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/mke2fs type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/mm_events type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/monkey type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/mtpd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/ndc type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/netd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/netutils-wrapper-1.0 type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/newfs_msdos type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/odsign type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/otapreopt type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/otapreopt_chroot type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/otapreopt_script type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/otapreopt_slot type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/perfetto type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/ping type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/ping6 type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/pm type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/pppd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/preloads_copy.sh type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/preopt2cachename type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/procrank type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/racoon type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/reboot type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/recovery-persist type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/recovery-refresh type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/requestsync type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/resize2fs type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/rss_hwm_reset type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/run-as type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/schedtest type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/screencap type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/screenrecord type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/sdcard type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/secdiscard type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/secilc type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/sensorservice type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/service type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/servicemanager type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/set-verity-state type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/settings type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/sgdisk type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/sh type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/showmap type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/simpleperf type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/simpleperf_app_runner type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/sload_f2fs type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/sm type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/snapshotctl type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/snapuserd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/ss type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/storaged type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/surfaceflinger type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/svc type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/tc type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/tcpdump type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/telecom type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/tombstoned type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/toolbox type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/toybox type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/traced type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/traced_perf type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/traced_probes type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/trigger_perfetto type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/tune2fs type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/tzdatacheck type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/uiautomator type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/uinput type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/uncrypt type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/update_engine type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/update_verifier type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/usbd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/vdc type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/viewcompiler type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/vold type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/vold_prepare_subdirs type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/vr type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/wait_for_keymaster type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/watchdogd type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/wificond type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/wm type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/xml2abx type ext4 (ro,seclabel,relatime)
/dev/PNsr9c/.magisk/block/system_root on /system/bin/ziptool type ext4 (ro,seclabel,relatime)
tmpfs on /system/bin/app_process32 type tmpfs (rw,seclabel,relatime,mode=755)
tmpfs on /dev/PNsr9c/.magisk/mirror/system_root/system/bin/app_process32 type tmpfs (rw,seclabel,relatime,mode=755)
tmpfs on /system/bin/app_process64 type tmpfs (rw,seclabel,relatime,mode=755)
tmpfs on /dev/PNsr9c/.magisk/mirror/system_root/system/bin/app_process64 type tmpfs (rw,seclabel,relatime,mode=755)
adb on /dev/usb-ffs/adb type functionfs (rw,relatime)
mtp on /dev/usb-ffs/mtp type functionfs (rw,relatime)
ptp on /dev/usb-ffs/ptp type functionfs (rw,relatime)
/dev/fuse on /mnt/installer/0/emulated type fuse (rw,lazytime,nosuid,nodev,noexec,noatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /mnt/androidwritable/0/emulated type fuse (rw,lazytime,nosuid,nodev,noexec,noatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /mnt/user/0/emulated type fuse (rw,lazytime,nosuid,nodev,noexec,noatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /storage/emulated type fuse (rw,lazytime,nosuid,nodev,noexec,noatime,user_id=0,group_id=0,allow_other)
/dev/block/dm-6 on /mnt/pass_through/0/emulated type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/dm-6 on /mnt/androidwritable/0/emulated/0/Android/data type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/dm-6 on /mnt/installer/0/emulated/0/Android/data type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/dm-6 on /mnt/user/0/emulated/0/Android/data type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/dm-6 on /storage/emulated/0/Android/data type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/dm-6 on /mnt/androidwritable/0/emulated/0/Android/obb type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/dm-6 on /mnt/installer/0/emulated/0/Android/obb type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/dm-6 on /mnt/user/0/emulated/0/Android/obb type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
/dev/block/dm-6 on /storage/emulated/0/Android/obb type f2fs (rw,lazytime,seclabel,nosuid,nodev,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,acl,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,reserve_root=32768,resuid=0,resgid=1065,inlinecrypt,alloc_mode=default,checkpoint_merge,fsync_mode=nobarrier,compress_algorithm=lz4,compress_log_size=2,compress_extension=apk,compress_extension=apex,compress_extension=so,compress_mode=fs,atgc)
The thing that stands out in this mount list is that individual files under /system/bin are mounted, one by one, from /dev/PNsr9c/.magisk/block/system_root, which is clearly a hack for rooting. When they say that Magisk is “just a layer”, I guess they refer to these mounts that override the original files without changing the original filesystem. It’s also often said that Magisk “doesn’t change system”, so once again, it doesn’t change it, but overrides some files by virtue of tons of mounts.
This is time to mention that on my own phone “overlay” is listed in /proc/filesystems, so apparently that would be the more elegant solution to overriding a lot of files. That would allow having a single mount over /system/bin and let it selectively override some of the files. Once again, without touching the original.
This post is about fetching mail. For sending emails through OAuth2-enabled SMTP servers, see this post.
Introduction
After a long time that Google’s smtp server occasionally refused to play ball with fetchmail, tons of Critical Alerts on “someone knowing my password” and requests to move away from “Less Secure Apps” (LSA) and other passive-aggressive behaviors, I eventually got the famous “On May 30, you may lose access” mail. That was in the beginning of March.
My reaction to that was exactly the same as to previous warnings: I ignored it. Given the change that was needed to conform to Google’s requirements, I went on with the strategy that “may” doesn’t necessarily mean “will”, and that I don’t move until they pull the plug. Which they didn’t on May 30th, as suggested. Instead, it happened on June 8th.
Since I don’t intend to ditch my Gmail address, it follows that OAuth2 is going to be part of my life from now on, and that I’ll probably have to figure out why this or that doesn’t work out every now and then. So I might as well learn the darn thing once and for all. Which I just did.
This post consists of my own understanding on this matter. More than anything, it’s intended as a reminder to myself each time I’ll need to shove my hands deep into some related problem.
None of the covered subjects have anything to do with my professional activity as an engineer.
Spoiler: My decisions on my own mail processing
This is my practical conclusion of everything written below, so the TL;DR is that I decided to stop using Fetchmail with Google’s mail servers. Instead, I’ve set Gmail to forward all emails to one of my other email accounts, to which I have pop3 access with Fetchmail (it’s actually on a server I control). I just added another transmission leg.
This is mainly because of the possibility that continuing to use Fetchmail with Google’s server will require my personal attention every now and then, for reasons I elaborate on below. It’s not impossible, but I’m not sure Fetchmail with Gmail is going to be a background thing anymore.
Mail forwarding is a solid solution to this. It doesn’t create a single point of failure, because I can always access the mails with Gmail’s web interface, should there be a problem with the forward-to-Fetchmail route. The only nasty thing that can happen is that the forwarding’s destination email address may be disclosed to the sender, if the delivery fails for some reason: It appears in the bounce message.
So if you want to use the forwarding method for an email address that you keep for the sake of anonymity, you’ll have to use a destination email address that says nothing about you either. There are plenty of email services with POP3 support at a fairly low cost.
The only reason I still need to live with OAuth2 support is that emails that I send with my Gmail address must go through Google’s servers, or else they are rejected by a whole lot of mail servers out there by virtue of DMARC.
So I upgraded Thunderbird to a version that supports OAuth2, and it works nicely with Google. I could have fetched the emails with Thunderbird too, but I still want to run my own spam filter, which I why I want fetchmail to remain in the loop for arriving mails.
And now, to the long story.
What is OAuth2?
To make a long story short, it’s the mechanism behind “Login with Google / Facebook / whatever”. Rather than having the user maintain a username and password for every service it accesses, there’s one Authorization Server, say Google, that maintains the capability to verify the actual user.
The idea is that when the user wants to use some website with “Login with Google”, the website doesn’t need to check the user’s identity itself, but instead it relies on the authentication made by Google. As a bonus, the fact that the user has logged into the site with Google, allows the site’s back-end (that is, the web server) to access some Google services on behalf of the user. For example, to add an entry in the user’s Google calendar.
To make this work, the site’s back-end needs to be able to prove that it’s eligible to act on behalf of the said user. For this purpose, it obtains an access token from the Authorization Server. In essence, this access token is a short-lived password for performing certain tasks on behalf of a certain user.
So an access token is limited in three ways:
- It’s related to a specific Google user
- It’s limited in time
- It gives its owner only specific permissions to carry out operations, or as they’re called, scopes.
For the sake of fetching emails, the recent change was that Gmail moved from accepting username + password authentication to only accepting an access token that allows the relevant user to perform pop / imap operations.
Interactions with the Authorization Server
The Authorization Server is responsible mainly for two tasks:
- The initial authentication, which results in obtaining an access token, a refresh token and various information (in JSON format).
- The refreshing of the access token, which is performed to replace an expired or soon-to-expire access token with a valid one.
So the overall picture is that it starts with some initial authentication, and then the owner of the access token keeps extending its validity by recurring refresh requests.
The initial authentication is done by a human using a web browser. That’s the whole point. This allows the Authorization Server to control the level of torture necessary to obtain the access token. It may not require any action if the user is already safely logged in, and it may suddenly decide to ask silly questions and/or perform two-factor authentication and whatnot.
Refreshing the access token is a computer-to-computer protocol that requires no human interaction. In principle, access can be granted forever based upon that initial authentication by refreshing the token indefinitely. But the Authorization Server is nevertheless allowed to refuse a refresh request for any or no reason. In fact, this is the way Google can force us humans to pay attention. The documentation tends to imply that tokens are always refreshed, but at the same time clearly state that the requester of a refresh should handle a refusal gracefully by reverting to the browser thing.
Remember those “suspicious activity” notifications from Google, begging us to confirm that it was us doing something on an uknown device? No need to beg anymore. If Google wants us to confirm something, it just denies the token refresh request. The only way to resume access is going back to initial authentication. This brings the human user to a browser soon enough to re-authenticate, which is a good opportunity to sort out whatever needs sorting out.
For example, if Thunderbird is used to access mail from Gmail with OAuth2, it must have the capability to open a browser window in order to perform the initial authentication (which it does nowadays). Hence if a refresh requests fails, this browser window will be opened again for further action. So there’s a means to talk with the human user. This possibility didn’t exist with the old password authentication, because if that failed, the user was prompted for a new password. So there was no reasonable way to initiate communication with the human user by refusing access.
How obnoxious service providers intend to be with this new whip is yet to be seen, but it’s clear that OAuth2 opens that possibility. The fact that access tokens are currently refreshed forever without the need to re-authenticate, doesn’t say how it’s going to be in the future.
As a bit of a side note, it’s common practice that access to cloud services can be made with an initial authentication that doesn’t involve a web browser. This makes sense, as software that consumes these services typically runs on servers with no human around. Today, this can be used to obtain tokens for Gmail access, but I doubt that will go on for long.
The authentication handshake in a nutshell
There are plenty of resources on OAuth2: To be begin with, there’s RFC 6749, which defines OAuth2, and several tutorials on the matter, for example this one. And there’s Google’s page on using OAuth2 for accessing Google APIs, which is maybe the most interesting one, as it walks through the different usage scenarios, including devices that can’t run a web browser.
This way or another, it boils down to the following stages for a website with “Login with X”:
It may be required to add additional credentials in requests for an access token (i.e. along with an authorization code or a refresh token), namely the client_id and client_secret parameters. These credentials are relevant in particular with cloud applications, and they are obtained when registering for such.
So this was the scenario for a website. What about fetching mails with Thunderbird and alike? It’s basically the same principle, only that the redirection with the authorization code is handled differently. There are several other variations, depending on the capabilities of the device that needs access. Among others, there’s a browser-less option for cloud applications, which is once again a variant of the above.
As for Thunderbird and other MUAs, they take the role of the back-end server: If they don’t have a valid access token, they open a browser window with the Authorization Server’s URL, with all necessary parameters. The redirection to the website is done differently, but it boils down to Thunderbird obtaining the authorization code and subsequently using it to obtain the access token. And then refreshing it as necessary.
So to summarize: There’s a browser session that ends with an authorization code, and the application uses this authorization code to get an access token. This access token is effectively a short-lived password that is used with Google’s API servers, Google’s smtp server included.
And by the way, there’s a maintained Perl module for OAuth2. I don’t know if I should be surprised about that.
fetchmail and OAuth2
Fetchmail 7 is apparently going to to support OAuth2, but there’s little enthusiasm for supporting it on the long run. It also appears like OAuth2 will not be backported to fetchmail-6.x.x.
To Fetchmail, the authentication tokens are just a replacement for the password. It’s another secret to send away to the server. So the entry in .fetchmailrc goes something like this:
poll <imap_server> protocol imap
auth oauthbearer username <your_email>
passwordfile "/home/yourname/.fetchmail-token"
is yourname here
[ ... ]
For this to work, there must be a mechanism for keeping the token valid. The mechanism suggested in fetchmail’s own git repository is that a cronjob first invokes a Python script that refreshes the token if necessary (and updates .fetchmail-token). Fetchmail is then called (in non-daemon mode) as part of this cronjob, and does its thing.
The approach for making this work automatically is to rely on the API for Google and Microsoft’s cloud services, which is intended for allowing scripts to access these services in a safely authenticated way. It seems to be an attempt to avoid the browser session at all costs. Which is understandable, given that fetchmail is traditionally a daemon that works silently in the background.
However using fetchmail like this requires registering the user as a Google cloud API user, which is quite difficult and otherwise annoying. So I can definitely understand the lack to of enthusiasm expressed by Fetchmail’s authors (more on that below).
But I beg to differ on this approach. The browser session is what Google really wants, so there’s no choice but to embrace it. Since my own motivation to use fetchmail is zero at this point, I didn’t implement anything, but this is what I would have done. And maybe will do, if it becomes relevant in the future:
A simple systemd-based daemon keeps track on when tokens expire, and issues refresh requests as necessary. If a valid token for a Gmail account is missing (because the refresh requests failed, or because an account was just added), this daemon draws the user’s attention to the need for an authentication session. Maybe a popup, maybe an icon on the system tray. When the user responds to that alert, a browser window opens with the relevant URL, and the authentication process takes place, ending with an authorization code, which is then turned into a valid token.
As for Fetchmail itself, it keeps running as usual as a daemon, only using access tokens instead of passwords. If a token is invalid, Google’s server will reject it, and if that goes on for too long, Fetchmail issues the warning mail message we’re probably all familiar with. Nothing new.
This doesn’t require any registration to any service. Just to enter the username and password the first time the daemon is launched, and then possibly go through whatever torture Google requires when it gets paranoid. But this is the way Google probably wants it to work, so no point trying to fight it. Frankly, I don’t quite understand why the Fetchmail guys didn’t go this way to begin with.
Future of OAuth2 support
Personally, I think Fetchmail should support OAuth2 authentication to the extent that it’s capable of using an access token for authentication. As for obtaining and maintaining the tokens, I can’t see why that has anything to do with Fetchmail.
The authors’ view is currently somewhat pessimistic. To cite the relevant entry in the NEWS file:
OAuth2 access so far seems only to be supported by providers who want to exert control over what clients users can use to access their very own personal data, or make money out of having clients verified. There does not appear to be a standard way how service end-points are configured, so fetchmail would have to carry lots of provider-specific information, which the author cannot provide for lack of resources.
OAuth2 is therefore generally considered as experimental, and unsupported, OAuth2 may be removed at any time without prior warning.
As for their affection for OAuth2, see the preface in README.OAUTH2 file. This file nevertheless explains how to obtain an OAuth2 client id and client secret from Google and Microsoft. Something I suggested to skip, but anyhow.
App passwords
This isn’t really related, but it’s often mentioned as a substitute for OAuth2, so here are a few words on that.
It seems like there’s a possibility to generate a 16-digit password, which is specific to an app. So at least in theory, this app password could be given to Fetchmail in order to perform a regular login.
I didn’t pursue this direction, mainly because the generation of an app password requires two-step verification. Forwarding sounds so much nicer all of the sudden.
Besides, I will not be surprised if Google drops App passwords sooner or later, in particular for Gmail access.
Summary
I can’t say that I’m happy with OAuth2 becoming mandatory, but I guess it’s here to stay. My personal speculation is that it has become mandatory to allow Google to re-authenticate humans gracefully, possibly with increasingly annoying means. This is a fight against spammers, scammers and account hijackers, so paranoia is the name of the game.
Apparently, forcing the owner of the Google account into an authentication session, either with a browser on the desktop or on the mobile phone, possibly both combined, is the future weapon in this fight. It’s quite annoying indeed, but I guess there are worse problems on this planet.
