The SPF, DKIM and DMARC trio: Making your email appear decent

This post was written by eli on March 9, 2019
Posted Under: email,Internet

Intro

Whether you just want your non-Gmail personal email to get through, or you have a website that produces transactional emails (those sent by your site or web app), there’s a long fight with spam filters ahead.

The war against unsolicited emails will probably go on as long as email is used, and it’s an ongoing battle where one leak is sealed and another is found. Those responsible for mail delivery constantly tweak their spam detectors’ parameters to minimize complaints. There are no general rules for what is detected as spam and what isn’t. What passes Gmail’s tests may very well fall on Security Industries’ mail server and vice versa. Each have their own experience.

But you want your message to reach them all. Always.

This is a guide to the main ideas and concepts behind the trio of mechanisms mentioned in the title. The purpose is to focus on the delicate and sometimes crucial details that are often missed in howtos everywhere. And also try to understand the rationale behind each mechanism, even though it might not be relevant when spam detector X is tuned to achieve the best results, given a current flow of spam mails with a certain pattern.

Howtos usually tell you to employ a DKIM signing software on the mail server, and make SPF and DKIM DNS records for “your domain”. Which one? Not necessarily trivial, as discussed below. And then possibly add a DMARC record as well. Will it really help? Also discussed.

Here’s the thing: Employing these elements will most likely do something good, even if you get it wrong. Setting up things without understanding what you’re doing can solve an immediate problem. This post focuses on understanding the machinery, so the best possible setting can be achieved.

Get your tie knot right.

Rationale: Domains cost money

There are different ideas behind each of the trio’s mechanisms, but there’s one solid idea behind them all: The reputation of a domain name.

If you’re a spammer, you can’t send thousands of emails that are linked to a domain name without wrecking its reputation rather quickly. So let’s make sure each domain name’s owner stands behind the mails sent on its behalf, and maintains its reputation. This requires a way to tell whether this owner really sent each mail, and not just a spammer abusing it. SPF and DKIM supply these mechanisms.

The cost of domain names makes it unworthy to purchase domains just for the sake of a few thousand mails until its reputation is dead meat. Well, sort of. There are .bid domains at $1.75 today. But .com and .org are still rather expensive.

DMARC takes this one step further, and allows a domain name owner to prevent the delivery of emails that weren’t sent on its behalf. It also puts the focus on the the sender given in the”From:” header, instead of other domains, which SPF and DKIM might relate to. This makes the junk domain concept even less eligible.

Despite all said above, I still get spam messages (of the random recipient type) with this trio perfectly set up. But they’re relatively rare.

The trio in short

These three techniques are fundamentally different in what they do. In brief for now, in more detail further below:

  • SPF: Defines the set of server IP addresses that are authorized to use a domain name to identify itself (HELO/EHLO) and/or the mail’s sender (MAIL FROM) in the SMTP exchange. Note that this doesn’t directly relate to the “From:” mail header, even though it does in many practical cases.
  • DKIM: A method to publish a public key in a DNS record for the digital signature of some parts of an email message, so this signature can be verified by any recipient. The domain name of this DNS record, which is given explicitly in the signature, doesn’t need to have any relation to the mail’s author, sender or any relaying server involved (even though it usually has). It’s just a placeholder for the accumulating reputation of mails that are signed with it.
  • DMARC: A mechanism to prevent the domain name from being abused by spammers. It basically tells the recipient than an email with a certain Author Domain (as it appears in “From:”) should pass an SPF and/or DKIM test, and what to do if not.

In essence, SPF authenticates the use of some mail relay servers, DKIM authenticates the message carrying its signature, and DMARC says what to do if the authentication(s) fail.

The DNS records

All three techniques rely on a DNS lookups for a TXT entry, which has the domain name included (let’s say we have example.com.):

  • SPF records are found as a TXT record for the domain itself (that is, example.com.).
  • DKIM records are the TXT records for the “selector._domainkey” subdomain, where “selector” is given in the mail message’s DKIM header. So it’s like default._domainkey.example.com (for selector=default).
  • DMARC record are the TXT entry for the “_dmarc” subdomain (i.e. _dmarc.example.com).

So it’s crucial which domain it is that the spam filter software considers to be “the domain”. Spoiler: DKIM and DMARC have this sorted out nicely. It’s SPF that is tricky.

Note that given an email message, the recipient can easily check whether it has SPF and DMARC records, but (without DMARC) it can’t know if there’s a relevant DKIM record available, because of the selector part. Consequently, adding a DKIM record and signing only part of the emails won’t backlash on those that aren’t signed.

Which domain is “the domain”?

Quite often, guides in these topics just say “the domain”, making it sound as if there’s only one domain involved. In fact, there are several to be aware of.

Let’s say that myself@example.com sends a mail by connecting to its ISP’s mail server mx.isp.com, which in turn relays it to the destination mail server. We then have four different domains involved.

  1. The domain of the author, appearing in the From header, shown to the human recipient as the sender. example.com in this case.
  2. The “envelope sender”, appearing in the MAIL FROM part of the SMTP conversation of the relay transmission. This could be example.com (the simple approach), but also something like bounce.isp.com. This is because the envelope sender is the bounce address, and some mail relays make up some kind of bogus bounce address so they can track the bouncing mails.
  3. The domain used in the HELO/EHLO part of the SMTP conversation of the relay transmission. Probably something like mail23.isp.com, as the ISP has many servers for relaying out.
  4. The rDNS domain entry of the IP address of the sender on the relay transmission. If this entry doesn’t exist, or isn’t exactly as the HELO/EHLO domain, hang the postmaster. Some mail servers won’t even talk with you unless they match.

I use the term “relay transmission” for the connection between two mail servers: Going from the server that accepted your message for transmission when you pressed “Send” to the server that holds the mail account of the mail’s recipient (i.e. destination of the MX record of the recipient’s full domain).

But oops. Mails are often relayed more than once before reaching their final station. Except for the first item in the list above, the domains are different on each such transmission. Which one counts? When does it count?

Luckily, this dilemma is pretty much limited to SPF. And with DMARC, it’s nonexistent.

SPF

At times, people just add an SPF record for their mail address’ domain with their relay servers’ IP range, and think they’ve covered themselves SPF-wise. Sometimes they did, and sometimes they didn’t. No escape from the gory details.

If you’re not familiar with the HELO/EHLO and MAIL FROM: SMTP tokens, I warmly suggest taking a quick look on another post of mine. It’s nearly impossible to understand SPF without it.

The SPF mechanism is quite simple: The server that receives the email looks up the TXT DNS record(s) for the domain name given in the envelope sender, that is in “MAIL FROM:”. If an SPF record exists, it checks if the IP address of the sender is in the allowed set, and if so, the SPF test is passed.

The domain name that is checked is the “domain portion” of the “MAIL FROM” identity (see RFC7208 section 4.1), or in other words, everything after the “@” character of the MAIL FROM. Or so it’s commonly understood: The RFC doesn’t define this term.

The receiver is likely to perform the same check on the HELO/EHLO identification of the sender. In fact, RFC7208 section 2.3 recommend performing it even before the MAIL FROM check. The SPF test will pass if either of the HELO/EHLO or the MAIL FROM check passes (the RFC doesn’t say this explicitly, but it’s clear from the argument for beginning with the more definite HELO/EHLO check).

This is important: Any mail server can ensure all mails that go through it pass the (non-DMARC) SPF test, just by having a DNS record on its full HELO/EHLO domain name. It’s silly not to have one. So if you’re setting up a mail server called mx.theisp.com, be sure to add SPF records for mx.theisp.com, allowing the IP of that server. This SPF test won’t count for DMARC purposes, but the “Received-SPF: pass” line in among the mail headers surely doesn’t hurt.Except for when DMARC is applied in one of its enforcing modes, there is no clear rule on what to do if this test fails or passes with one of the SMTP tokens or both. This is raw material for the spam detection software.

It’s however important to note that it’s perfectly normal that envelope address is made up completely by the mail relay, because it functions as a bounce address. So an email sent from myself@example.com may have the same envelope address, but it’s also perfectly normal that the MAIL FROM: would be bounce-3242535@bounce.isp.com. This allows the ISP to detect massive bouncing of emails, and possibly do something about it. In this case, the relaying server’s domain can be used to pass the (non-DMARC) SPF test instead.

Well, with the reputation per domain rationale, it actually does makes sense. But with DMARC, this won’t cut. The SPF record must belong to the “From:” sender. See below.

Now, the formal rules are nice, but if you just wrote a spam filter, would you check for the SPF record of the “From:” sender’s domain, even though it’s not really relevant according to the RFC? Of course you would. If the domain owner of the Author Address has given permission for a server to relay emails on its behalf, it’s a much stronger indication. So it’s probably a good idea to make such a record, even if makes no sense directly. And it makes you better prepared for DMARC.

As a matter of fact, it’s recommended to add SPF records for any domain and subdomain that may somehow appear in the mail, to the extent possible, of course. A DNS record is cheap, and you never know if a spam detector expects it to be there, whether it should or not.

Bottom line: We don’t really know how many points spam filter X gives an SPF record of this type or another. It depends on the history of previous spam. So try to cover all options, even those that aren’t required per RFC.

Information on setting up an SPF record is all around the web. I suggest starting with Wikipedia’s great entry and if you want to be accurate about it, in RFC7208.

DKIM

This is easiest explained through a real signature, taken from the header of a real mail message:

Dkim-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;        d=gmail.com;
 s=20161025;        h=mime-version:references:in-reply-to:from:date:message-id:subject:to;
        bh=vx0LAOXz3Mr8zS/Jy2ayKOep6NlflK3t+BpJyi78v9A=;
 b= [ ... ]

To verify this signature, a lookup for the DNS TXT entry for 20161025._domainkey.gmail.com is made.Note that except for the _domainkey part, the domain comprises of the s= (selector) and the d= (domain) assignments in the signature. The answer should contain an RSA public key for verifying that the hash of some selected headers (selected by h= ) is indeed signed by the blob in the b= assignment. That’s it. If the signature is OK, the DKIM test is passed.

Note that DKIM doesn’t (usually) sign the message body, so signing a message with DKIM doesn’t make you accountable for its content, only the fact that you sent that mail.

Also note that no other domains, that are related to the email, make any difference for passing the DKIM test itself. Not the sender’s not the mail relays’, nothing. Passing the DKIM test just means that the signing domain (gmail.com) has signed this message (actually, some of its headers) and therefore puts its own reputation on it. It doesn’t say anything on who sent the message.

The common practice is however that the signing domain is the From: header domain. Probably because DMARC can’t be applied otherwise, maybe also because the goal is to impress spam filters. Passing the DKIM test is nice formally, but if the spam filter thinks it fishy, it can backlash.

As for Spamassassin, it doesn’t care much about DKIM so far. As of now, passing the DKIM test doesn’t change the score. Or more precisely, the existence of a DKIM signature increases the spam score (more spammy) with 0.1, but if the signature is correct, the score is reduced with 0.1. So we’re back to zero. If the signature belongs to the author (matching From: domain) the score is reduced (i.e. towards non-spam) by 0.1. All in all, a DKIM signature wins a score of 0.1 on Spamassassin. May not seem to be worth the efforts, but Spamassassin is not the only filter in the world. And it may change over time.

Another reason, from RFC4871, section 6.3: “If the message is signed on behalf of any address other than that in the From: header field, the mail system SHOULD take pains to ensure that the actual signing identity is clear to the reader.” Yeah right. I’ve seen Gmail verifying a DKIM signature of a domain which had nothing to do with anything in that message, surely not the sender. It just went “dkim=pass”.

Finally a question: The MUA (e.g. Thunderbird) is allowed to put a DKIM signature, which would actually make sense: It allows a human end user to sign the emails directly, with no need for anything special on the relaying infrastructure. And there’s no problem with multiple RSA key pairs for multiple users of the domain, since the “s=” selector allows a virtually unlimited number of DKIM DNS records. Why there isn’t a plugin for at least Thunderbird is unclear to me. Maybe the answer lies in Spamassassin’s indifferent response to it.

DMARC

Suppose that you own company Example Ltd. with domain example.com, and you’ve decided that all mails from (as in header From:) that domain will be DKIM signed. Now some spam mail arrives from someone else, without a DKIM signature and fails the SPF test. But the recipient has no way to tell that it should pass such tests.

DMARC is the mechanism that tells the recipient what to expect, and what to do if the expectation isn’t met. This allows the owner of the domain to ensure only mails arriving from its own machines are accepted. Spam pretending to come from its domain is dropped.

This is what Gmail did to force emails from all its users (i.e. having a gmail.com address) to be relayed through their servers only. The TXT for _dmarc.gmail.com goes:

"v=DMARC1; p=none; sp=quarantine; rua=mailto:mailauth-reports@google.com"

In other words, if it isn’t proven to come from Gmail’s server, hold the message. Most servers just junk it.

And now to how the test is done. Spoiler II: DMARC isn’t interested in a domain test if it isn’t tightly linked with the “From:” header’s domain. Or as they call it: Aligned with the RFC5322.From field. This is huge difference.

Let’s take it directly from RFC7489, section 4.2:

A message satisfies the DMARC checks if at least one of the supported authentication mechanisms:

  1. produces a “pass” result, and
  2. produces that result based on an identifier that is in alignment, as defined in Section 3.

The “supported authentication mechanisms” for DMARC version 1 are SPF and DKIM, as listed in section 4.1 of the same RFC.

The first thing we learn is that it’s enough to pass one of SPF or DKIM. No need to have both for passing DMARC.

Second, the term “is in alignment” above. It’s defined in the RFC itself, and essentially means that the domain for which the SPF or DKIM passed is the same as the one in the From: header, possibly give or take subdomains. The only reason they didn’t just say that the domains must be equal is because of the possibility of “relaxed mode”, allowing an email from myself@mysubdomain.example.com to be approved by passing tests with the example.com domain. This is what “being in alignment” means in relaxed mode. In “strict mode” alignment occurs only when they’re perfectly equal.

If the email passes the DMARC test, there isn’t much to fuss about. If it fails, the decision what to do depends on the policy, as given in the relevant domain. Which, according to RFC7489 section 3 is: “Author Domain: The domain name of the apparent author, as extracted from the RFC5322.From field”. And then in section 4.3, item 7: “The DMARC module attempts to retrieve a policy from the DNS for that domain” (referring to the Author Domain).

So it’s a DNS query for the TXT record of the From: domain, with the “_dmarc” subdomain prepended. As in the example above for gmail.com.

Finally, a tricky point. If a mail server, for which the SPF test is made, didn’t use the Author Domain in its MAIL FROM nor in the HELO/EHLO, the SPF test is worthless for DMARC purposes. It’s however quite tempting to check the Author Domain for its SPF record nevertheless. I mean, if the Author Domain allows the IP address of the mail relay server, isn’t it good enough to pass a DMARC test? Doing this goes against the SPF’s RFC, and isn’t mentioned in any way in DMARC’s RFC. But it makes a lot of sense. I won’t be surprised if it’s common practice already.

Will DMARC make my email delivery better?

TL;DR: Surprisingly enough, yes.

The irony about DMARC is that it bites on the spam messages, and does very little on the legit ones. After all, if an email passed both the SPF and DKIM tests on the Author Domain, what is there left to say?

And if the same email passed only one of the tests, why would a DMARC record add reassurance?

Of course, if you want to fake mails pretending to be you, definitely apply DMARC.

But once again, noone knows how spam filter X behaves. Maybe someone found out that DMARC signed domains carry less spam, and tuned the filter in favor of them. And maybe the rejection of spam mails thanks to the DMARC record helped with the domain’s spam statistics. Even though I would expect any machine that maintains statistics to count the emails that pass SPF / DKIM tests separately.

And here comes the big surprise. Gmail refused to accept messages from my server until I added a DMARC record. Once I did it, I was all welcome. It makes no sense, but somehow, Google seems to like the very existence of a DMARC record. Maybe a coincidence, most likely not. So do yourself a favor, and add a TXT record to _dmarc.yourdomain.com:

v=DMARC1; p=none; sp=none; ruf=mailto:mailreports@yourdomain.com

This record tells the recipient to do nothing with a mail message that fails the DMARC test, so it’s harmless. But it will send an email to tell you about it to the email address given. Which can be useful in itself.

Conclusion

There might be official rules for entering a club, but in the end of the day, you can’t know what the doorkeeper looks at. So try to get everything as tidy as possible, and hope you won’t be mistaken for the bad guys.

And don’t wait for the first time you won’t be let in. It might be too late to fix it then.

Add a Comment

required, use real name
required, will not be published
optional, your blog address