fetchmail, openssl and a sudden failure to authenticate certificates
Since around the beginning of December 2017, fetchmail stopped retrieving mails form Gmail servers silently, without issuing any kind of error message. Only when starting fetchmail in the foreground, I got
fetchmail: Server certificate verification error: unable to get local issuer certificate
fetchmail: This means that the root signing certificate (issued for /C=US/O=Google Trust Services/CN=Google Internet Authority G3) is not in the trusted CA certificate locations, or that c_rehash needs to be run on the certificate directory. For details, please see the documentation of --sslcertpath and --sslcertfile in the manual page.
140703549138760:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:s3_clnt.c:1060:
fetchmail: SSL connection failed.
fetchmail: socket error while fetching from ...@pop.gmail.com
fetchmail: Query status=2 (SOCKET)
That’s really cute. The SSL connection fails, but fetchmail doesn’t think it should drop me a note about it (like it does when the server refuses for a long time, for example). I should mention that I’m using an old 6.3.17 fetchmail release.
So let’s try the connection following this post (or “man s_client”):
Connect to a secure POP server:
$ openssl s_client -connect pop.gmail.com:995
CONNECTED(00000003)
depth=2 OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
verify return:1
depth=1 C = US, O = Google Trust Services, CN = Google Internet Authority G3
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google Inc, CN = pop.gmail.com
verify return:1
---
Certificate chain
0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=pop.gmail.com
i:/C=US/O=Google Trust Services/CN=Google Internet Authority G3
1 s:/C=US/O=Google Trust Services/CN=Google Internet Authority G3
i:/OU=GlobalSign Root CA - R2/O=GlobalSign/CN=GlobalSign
[ ... ]
Huh? No problem? So why doesn’t fetchmail accept the certificate? Because in my case, certificates were read from a local directory: In .fetchmailrc, I had the “sslcertpath /home/…/.certs/” directive for each and every entry related to Gmail. That made fetchmail accept certificate authorities only from the local directory, which failed suddenly, probably due to a change in Gmail’s certificate chain.
Where did I get this directive from? Probably from the automatic configuration made by fetchmailconf. Hurray.
So the obvious solution was to drop all those “sslcertpath” directives, and all was fine again.
Not really relevant, but…
Since I was playing with openssl, it’s also possible to talk with an HTTPS server directly this way (note the GET / request close to the end):
$ openssl s_client -connect www.google.com:443
CONNECTED(00000003)
depth=3 C = US, O = Equifax, OU = Equifax Secure Certificate Authority
verify return:1
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify return:1
depth=1 C = US, O = Google Inc, CN = Google Internet Authority G2
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google Inc, CN = www.google.com
verify return:1
---
Certificate chain
0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
i:/C=US/O=Google Inc/CN=Google Internet Authority G2
1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEdjCCA16gAwIBAgIIWv4BLr9C/xUwDQYJKoZIhvcNAQELBQAwSTELMAkGA1UE
BhMCVVMxEzARBgNVBAoTCkdvb2dsZSBJbmMxJTAjBgNVBAMTHEdvb2dsZSBJbnRl
cm5ldCBBdXRob3JpdHkgRzIwHhcNMTcxMjEzMTMyOTExWhcNMTgwMzA3MTMwMTAw
WjBoMQswCQYDVQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwN
TW91bnRhaW4gVmlldzETMBEGA1UECgwKR29vZ2xlIEluYzEXMBUGA1UEAwwOd3d3
Lmdvb2dsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDh3H8L
+wzppO5DrRgLvOOF8nM692sA/aaFv0Hr5pQuuDluOwEV+ocY8iKwBWH2XMNTDTDz
KIuqyqAbe/DvUQ+a8HjuEGepWdBe/VY0vuxrjKc7yJs6QImmSt9dUF01LI6zKjyR
B7MNyMCtHHn2DFvD9uGocNqFXAeJrWCs2VbIqP+jj2QdVJ2WK/gV0ybyGmdyZfbw
SdDKSYKt3KK3depWai7CKeYHHNpMY8OFBLi4uWIWA28ZTzIxqb2Ar7aiZUUEzWgf
8Ak4fLsNzEuiCmouhFdBwwxDGGqDgrM+3NFk7kGoBOf2mTH6qcQ+sg2G/rVylE94
2mUhlw2viW/bAzN7AgMBAAGjggFBMIIBPTATBgNVHSUEDDAKBggrBgEFBQcDATAZ
BgNVHREEEjAQgg53d3cuZ29vZ2xlLmNvbTBoBggrBgEFBQcBAQRcMFowKwYIKwYB
BQUHMAKGH2h0dHA6Ly9wa2kuZ29vZ2xlLmNvbS9HSUFHMi5jcnQwKwYIKwYBBQUH
MAGGH2h0dHA6Ly9jbGllbnRzMS5nb29nbGUuY29tL29jc3AwHQYDVR0OBBYEFMY4
s3yyzEnrwiJ/NsB1uEETOXq6MAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUSt0G
Fhu89mi1dvWBtrtiGrpagS8wIQYDVR0gBBowGDAMBgorBgEEAdZ5AgUBMAgGBmeB
DAECAjAwBgNVHR8EKTAnMCWgI6Ahhh9odHRwOi8vcGtpLmdvb2dsZS5jb20vR0lB
RzIuY3JsMA0GCSqGSIb3DQEBCwUAA4IBAQAxixLtjbT18w3iXYj6psMVK2uK21Is
Z4Oi2y5533nHqZXhb3z9K7m1ejwgL/s+bb5+D4HQdKhopO81oBf2Li9ztQ255Q24
nA1p4xkTdPV3UvFPA6R6G4muFZQmJUvIgrH/uZAXQ36K9/8aI8SgawLo1RPDWOxW
pCw4/1SfQ8FgUQmvqb+OkQ5bCXXySRhidZkCUg4DXUNsJ++HATlSEeSb4kdu4Fny
mwRBZma5muwbwRxTHCnbs1A82Ehxi0+DBjifgJy0NMyZbsgiWnyuexNAxoDVIC0I
u2xlpTBqaS1HzPpJ7K9tJ7i6AYe2YCrKo/fYBYBWaw/Q7VNDWk7NvlTs
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
issuer=/C=US/O=Google Inc/CN=Google Internet Authority G2
---
No client certificate CA names sent
---
SSL handshake has read 3481 bytes and written 439 bytes
---
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : AES128-SHA
Session-ID: A455441C733C2CDD844EEC39DEE96FA7C6D47D38F4A6021097891DF198E60C68
Session-ID-ctx:
Master-Key: EA9D1453E32865C2C4F04E969103ADA4E59C8ECC2FA09F6D8F2EDAE5E75E1DE40A18FF60CED7459A4F60679BB230C663
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
TLS session ticket lifetime hint: 100800 (seconds)
TLS session ticket:
0000 - 00 26 5c 16 79 2f e1 46-a4 5b 34 30 72 a1 64 b7 .&\.y/.F.[40r.d.
0010 - 79 20 15 87 62 cd 2a 05-8f 05 ac f7 d7 38 40 66 y ..b.*......8@f
0020 - 87 0a a9 50 55 ba 5e d2-8f 90 c0 d0 83 25 b8 3e ...PU.^......%.>
0030 - b7 7b eb 5a ff 30 27 aa-1b c9 a1 d9 54 c3 aa 7e .{.Z.0'.....T..~
0040 - 05 96 83 76 49 90 fe 8e-d9 d7 55 e0 a3 0b 5b df ...vI.....U...[.
0050 - aa 28 12 81 02 84 b9 47-97 cd b8 81 b8 ee 2a 1c .(.....G......*.
0060 - c2 8b e0 e6 92 ae 4b a3-fb 2a 8e f3 eb f5 43 7b ......K..*....C{
0070 - a8 e9 58 c9 22 3a 15 3d-81 a7 0b a8 1a e4 3a 55 ..X.":.=......:U
0080 - cd 72 04 8d 0e 70 5e 60-5d 19 d7 18 a1 1b ce d9 .r...p^`].......
0090 - 87 60 78 ec c0 f7 6e 0f-c4 5c e7 06 1a e2 c1 d0 .`x...n..\......
00a0 - e1 46 df c2 98 d0 da fd-87 eb 9b 0f 93 8a 4c e4 .F............L.
00b0 - 95 db da 63 b3 e2 78 08-09 75 53 b7 d1 e3 6c d2 ...c..x..uS...l.
00c0 - b6 6f 02 26 bd 16 e4 ae-a8 01 fa 81 3f e4 55 0d .o.&........?.U.
Start Time: 1515438093
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
GET /
HTTP/1.0 302 Found
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Location: https://www.google.co.il/?gfe_rd=cr&dcr=0&ei=E8BTWoO6GqfP8AeV05m4Cg
Content-Length: 272
Date: Mon, 08 Jan 2018 19:01:39 GMT
Alt-Svc: hq=":443"; ma=2592000; quic=51303431; quic=51303339; quic=51303338; quic=51303337; quic=51303335,quic=":443"; ma=2592000; v="41,39,38,37,35"
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="https://www.google.co.il/?gfe_rd=cr&dcr=0&ei=E8BTWoO6GqfP8AeV05m4Cg">here</A>.
</BODY></HTML>
read:errno=0
So it’s a bit like “nc”, only with encryption.