Thunderbird: Upgrade notes

Introduction

These are my notes as I upgraded Thunderbird from version 3.0.7 (released September 2010) to 91.10.0 on Linux Mint 19. That’s more than a ten year’s gap, which says something about what I think about upgrading software (which was somewhat justified, given the rubbish issues that arose, as detailed below). What eventually forced me to do this was the need to support OAuth2 in order to send emails through Google’s Gmail server (supported since 91.8.0).

Thunderbird is essentially a Firefox browser which happens to be set up with a GUI that processes emails. So for example, the classic menubar is hidden, but can be revealed by pressing Alt.

Using the correct profile

When attempting to run a new version of Thunderbird, be sure to rename ~/.thunderbird into something else, or else the current profile will be upgraded right away. With some luck, the suffixes (e.g. -release) might make Thunderbird ignore the old information, but don’t trust that.

Actually, it seems like this is handled gracefully anyhow. When I installed exactly the same version on a different position on the disk, it ignored the profile with -release suffix, and added one with -release-1. So go figure.

To select which profile to work with, invoke Thunderbird with Profile Manager with

$ thunderbird -profilemanager &

For making the upgrade, first make a backup tarball from the original profile directory.

To adopt in into the new version of Thunderbird, invoke the Profile Manager and pick Create Profile…, create a new directory (I called it “mainprofile”), and pick that as the place for the new profile. Launch Thunderbird, quit right away, and then delete the new directory. Rename the old directory with the new deleted directory’s name. Then launch Thunderbird again.

Add-ons

Previously, I had the following add-ons:

• BiDi Mail UI (apparently still necessary)
• Clippings. Just import the previous clippings from ~/clipdat2.rdf. Unlike the old version, the data is kept in a database file inside the profile, so the old file can be deleted.
• Gnome Integration Options for calling a command on mail arrival. It was deprecated. So I went for Mailbox Alert, which allows adding specific actions: Sound, a message and/or command. With mail folder granularity, in fact.
• Mail Tweak. It’s really really old, and probably unnecessary since long.
• Outgoing Message Format (for text vs. HTML messages). Deprecated since long, as these options are integrated into Thunderbird itself.

So I remained with the first two only.

Installing Thunderbird

The simplest Thunderbird installation involves downloading it from their website and extract the tarball somewhere in the user’s own directories. For a proper installation, I installed it under /usr/local/bin/ with

# tar -C /usr/local/bin -xjvf thunderbird-91.10.0.tar.bz2

as root. And then reorganize it slightly:

# cd /usr/local/bin
# mv thunderbird thunderbird-91.10.0
# ln -s thunderbird-91.10.0/thunderbird

Composing HTML messages

Right-click the account at the left bar, pick Settings and select the Composition & Addressing item. Make sure Compose messages in HTML is unchecked: Messages should be composed as plain text by default.

Then go through each of the mail identities and verify that Compose messages in HTML is unchecked under the Composition & Addressing tab.

However if Shift is pressed along with clicking Write, Reply or whatever for composing a new message, Thunderbird opens it as HTML.

Recover old contacts

Thunderbird went from the old *.mab format to SQLite for keeping the address books. So go Tools > Import… > Pick Address Books… and pick Monk Database, and from there pick abook.mab (and posssibly repeat this with history.mab, but I skipped this, because it’s too much).

Silencing update notices

Thunderbird, like most software nowadays, wants to update itself automatically, because who cares if something goes wrong all of the sudden as long as the latest version is installed.

I messed around with this for quite long until I found the solution. So I’m leaving everything I did written here, but it’s probably enough with just adding policies.json, as suggested below.

So to the whole story (which you probably want to skip): Under Preferences > General > Updates I selected “check for updates” rather than install automatically (it can’t anyhow, since I’ve installed Thunderbird as root), but then it starts nagging that there are updates.

So it’s down to setting the application properties manually by going to Preferences > General > Config Editor… (button at the bottom).

I changed app.update.promptWaitTime to 31536000 (365 days) but that didn’t have any effect. So I added an app.update.silent property and set it true, but that didn’t solve the problem either. So the next step was to change app.update.staging.enabled to false, and that did the trick. Well, almost. With this, Thunderbird didn’t issue a notification, but its tab on the system tray gets focus every day. Passive aggressive.

As a side note, there are other suggestions I’ve encountered out there: To change app.update.url so that Thunderbird doesn’t know where to look for updates, or set app.update.doorhanger false. Haven’t tried either.

So what actually worked: Create a policies.json in /usr/local/bin/thunderbird/distribution/, with “DisableAppUpdate“: true, that is:

{
 "policies": {
  "DisableAppUpdate": true
 }
}

Note that the “distribution” directory must be in the same the directory as the actual executable for Thunderbird (that is, follow the symbolic link if such exists). In my case, I had to add this directory myself, because of a manual installation.

And, as suggested on this page, the successful deployment can be verified by restarting Thunderbird, and then looking at Help > About inside Thunderbird, which now says (note the comment on updates being disabled):

In hindsight, I can speculate on why this works: The authors of Thunderbird really don’t want us to turn off automatic updates, mainly because if people start running outdated software, that increases the chance of a widespread attack on some vulnerability, which can damage the software’s reputation. So Thunderbird is designed to ignore previous possibilities to turn the update off.

There’s only one case where there’s no choice: If Thunderbird was installed by the distribution. In this case, it’s installed as root, so it can’t be updated by a plain user. Hence it’s the distribution’s role to nag. And it has the same interest to nag about upgrades (reputation and that).

So I guess that’s why Thunderbird respects this JSON file only.

Folders with new mails in red

Exactly like 10 years ago, the trick is to create a “chrome” directory under .thunderbird/ and then add the following file:

$ cat ~/.thunderbird/sdf2k45i.default/chrome/userChrome.css
@namespace
url("http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"); /* set default namespace to XUL */

/* Setting the color of folders containing new messages to red */

treechildren::-moz-tree-cell-text(folderNameCol, newMessages-true) {
 font-weight: bold;
 color: red !important;
}

But unlike old Thunderbird, this file isn’t read by default. So to fix that, go to Preferences > General > Config Editor… (button at the bottom) and there change toolkit.legacyUserProfileCustomizations.stylesheets to true.

New mail icon in system tray

Thunderbird sends a regular notification when a new mail arrives, but exactly like last time, I want a dedicated icon that is dismissed only when I click it. The rationale is to be able to see if a new mail has arrived at a quick glance of the system tray. Neither zenity –notification nor send-notify were good for this, since they send the common notification (zenity used to just add an icon, but it “got better”).

But then there’s yad. I began with “apt install yad”, but that gave me a really old version that distorted the icon in the system bar. So I installed it from the git repository’s tag 1.0. I first attempted v12.0, but I ended up with the problem mentioned here, and didn’t want to mess around with it more.

Its “make install” adds /usr/local/bin/yad, as well as a lot of yad.mo under /usr/local/share/locale/*, a lot of yad.png under /usr/local/share/icons/*, yad.m4 under /usr/local/share/aclocal/ and yad.1 + pfd.1 in /usr/local/share/man/man1. So quite a lot of files, but in a sensible way.

With this done, the following script is kept (as executable) as /usr/local/bin/new-mail-icon:

#!/usr/bin/perl
use warnings;
use strict;
use Fcntl qw[ :flock ];

my $THEDIR="$ENV{HOME}/.thunderbird";
my $ICON="$THEDIR/green-mail-unread.png";

my $NOW=scalar localtime;

open(my $fh, "<", "$ICON")
  or die "Can't open $ICON for read: $!";

# Lock the file. If it's already locked, the icon is already
# in the tray, so fail silently (and don't block).

flock($fh, LOCK_EX | LOCK_NB) or exit 0;

fork() && exit 0; # Only child continues

system('yad', '--notification', "--text=New mail on $NOW", "--image=$ICON", '--icon-size=32');

This script is the improved version of the previous one, and it prevents multiple icons in the tray much better: It locks the icon file exclusively and without blocking. Hence if there’s any other process that shows the icon, subsequent attempts to lock this file fail immediately.

Since the “yad” call takes a second or two, the scripts forks and exits before that, so it doesn’t delay Thunderbird’s machinery.

With this script in place, the Mailbox Alert is configured as follows. Add a new item to the list as in this dialog box:

The sound should be set to a WAV file of choice.

Then right-click the mail folder to have covered (Local Folders in my case), pick Mailbox Alert and enable “New Mail” and “Alert for child folders”.

Then right-click “Inbox” under this folder, and verify that nothing is checked for Mailbox Alert for it (in particular not “Default sound”). That except for the Outbox and Draft folders, for which “Don’t let parent folders alert for this one” should be checked, or else there’s a false alarm on autosaving and when using “send later”.

Later on, I changed my mind and added a message popup, so now all three checkboxes are ticked, and the Message tab reads:

I picked the icon as /usr/local/bin/thunderbird-91.10.0/chrome/icons/default/default32.png (this depends on the installation path, of course).

I’m not 100% clear why the original alert didn’t show up, even though “Show an alert” was still checked under “Incoming Mails” at Preferences > General. I actually preferred the good old one, but it seems like Mailbox Alert muted it. I unchecked it anyhow, just to be safe.

Refusing to remember passwords + failing to sent through gmail

It’s not a real upgrade if a weird problem doesn’t occur out of the blue.

So attempting to Get Messages from pop3 server at localhost failed quite oddly: Every time I checked the box to use Password Manager to remember the password, it got stuck with “Main: Connected to 127.0.0.1…”. But checking with Wireshark, it turned out that Thunderbird asked the server about its capabilities (CAPA), got an answer and then did nothing for about 10 seconds, after which it closed the connection.

On the other hand, when I didn’t request remembering the password, it went fine, and so did subsequent attempts to fetch mail from the pop3 server.

Another thing was that when attempting to use Gmail’s server, I went through the entire OAuth2 thing (the browser window, and asking for my permissions) but then the mail was just stuck on “Sending message”. Like, forever.

So I followed the advice here, and deleted key3.db, key4.db, secmod.db, cert*.db and all signon* files with Thunderbird not running of course. Really old stuff.

And that fixed it.

The files that were apparently created when things got fine were logins.json, cert9.db, key4.db and pkcs11.txt. But I might have missed something.

The GUI stuck for a few seconds every now and then

This happened occasionally when I navigated from one mail folder to another. The solution I found somewhere was to delete all .msf files from where Thunderbird keeps the mail info, and that did the trick. Ehm, just for a while. After a few days, it was back.

As a side effect, it forgot the display settings for each folder, i.e. which columns to show and in what order.

These .msf files are apparently indexes to the files containing the actual messages, and indeed it took a few seconds before something appeared when I went to view each mail folder for the first time. At which time the new .msf files went from zero bytes to a significant figure.

Since the problem remains, I watched “top” when the GUI got stuck. And indeed, Thunderbird’s process was at 100%, but so was a completely different process: caribou. Which is a virtual keyboard. Do I need one? No. So to get rid of this process (which runs all the time, but doesn’t eat a lot of CPU normally), go Accessibility settings, the Keyboard tab and turn “Enable the on-screen keyboard” off. The process is gone, and so is the problem with the GUI? Nope. It’s basically the same, but instead of two processes taking 100% CPU, now it’s Thunderbird alone. I have no idea what to do next.