The SPF, DKIM and DMARC trio: Making your email appear decent

Intro

Whether you just want your non-Gmail personal email to get through, or you have a website that produces transactional emails (those sent by your site or web app), there’s a long fight with spam filters ahead.

The war against unsolicited emails will probably go on as long as email is used, and it’s an ongoing battle where one leak is sealed and another is found. Those responsible for mail delivery constantly tweak their spam detectors’ parameters to minimize complaints. There are no general rules for what is detected as spam and what isn’t. What passes Gmail’s tests may very well fall on Security Industries’ mail server and vice versa. Each have their own experience.

But you want your message to reach them all. Always.

This is a guide to the main ideas and concepts behind the trio of mechanisms mentioned in the title. The purpose is to focus on the delicate and sometimes crucial details that are often missed in howtos everywhere. And also try to understand the rationale behind each mechanism, even though it might not be relevant when spam detector X is tuned to achieve the best results, given a current flow of spam mails with a certain pattern.

Howtos usually tell you to employ a DKIM signing software on the mail server, and make SPF and DKIM DNS records for “your domain”. Which one? Not necessarily trivial, as discussed below. And then possibly add a DMARC record as well. Will it really help? Also discussed.

Here’s the thing: Employing these elements will most likely do something good, even if you get it wrong. Setting up things without understanding what you’re doing can solve an immediate problem. This post focuses on understanding the machinery, so the best possible setting can be achieved.

Get your tie knot right.

Rationale: Building a reputation for the domain

There are different ideas behind each of the trio’s mechanisms, but there’s one solid idea behind them all: The reputation of a domain name.

If you’re a spammer, you can’t send thousands of emails that are linked to a domain name without wrecking its reputation rather quickly. So let’s make sure each domain name’s owner stands behind the mails sent on its behalf, and maintains its reputation. This requires a way to tell whether this owner really sent each mail, and not just a spammer abusing it. SPF and DKIM supply these mechanisms.

As domain names are rather cheap today, spammers may very well buy a new domain name for each shower of emails. But you can’t do that over time. This is why some spam filters don’t give any points to passing domain-related tests, even if they check them. However large email services (Gmail in particular) do seem to collect statistics on domains, and treat new emails according to existing reputation.

DMARC takes this one step further, and allows a domain name owner to prevent the delivery of emails that weren’t sent on its behalf. It also puts the focus on the the sender given in the”From:” header, instead of other domains, which SPF and DKIM might relate to. This seems to be important to prevent spammers from using your domain and wreck its reputation. More about that below.

Despite all said above, I still get spam messages (of the random recipient type) with this trio perfectly set up. But they’re relatively rare.

The trio in short

These three techniques are fundamentally different in what they do. In brief for now, in more detail further below:

• SPF: Defines the set of server IP addresses that are authorized to use a domain name to identify itself (HELO/EHLO) and/or the mail’s sender (MAIL FROM) in the SMTP exchange. Note that this doesn’t directly relate to the “From:” mail header, even though it does in many practical cases.
• DKIM: A method to publish a public key in a DNS record for the digital signature of some parts of an email message, so this signature can be verified by any recipient. The domain name of this DNS record, which is given explicitly in the signature, doesn’t need to have any relation to the mail’s author, sender or any relaying server involved (even though it usually has). It’s just a placeholder for the accumulating reputation of mails that are signed with it.
• DMARC: A mechanism to prevent the domain name from being abused by spammers. It basically tells the recipient than an email with a certain Author Domain (as it appears in “From:”) should pass an SPF and/or DKIM test, and what to do if not.

In essence, SPF authenticates the use of some mail relay servers, DKIM authenticates the message carrying its signature, and DMARC says what to do if the authentication(s) fail.

The DNS records

All three techniques rely on a DNS lookups for a TXT entry, which has the domain name included (let’s say we have example.com.):

• SPF records are found as a TXT record for the domain itself (that is, example.com.).
• DKIM records are the TXT records for the “selector._domainkey” subdomain, where “selector” is given in the mail message’s DKIM header. So it’s like default._domainkey.example.com (for selector=default).
• DMARC record are the TXT entry for the “_dmarc” subdomain (i.e. _dmarc.example.com).

So it’s crucial which domain it is that the spam filter software considers to be “the domain”. Spoiler: DKIM and DMARC have this sorted out nicely. It’s SPF that is tricky.

Note that given an email message, the recipient can easily check whether it has SPF and DMARC records, but (without DMARC) it can’t know if there’s a relevant DKIM record available, because of the selector part. Consequently, adding a DKIM record and signing only part of the emails won’t backlash on those that aren’t signed.

Which domain is “the domain”?

Quite often, guides in these topics just say “the domain”, making it sound as if there’s only one domain involved. In fact, there are several to be aware of.

Let’s say that myself@example.com sends a mail by connecting to its ISP’s mail server mx.isp.com, which in turn relays it to the destination mail server. We then have four different domains involved.

1. The domain of the author, appearing in the From header, shown to the human recipient as the sender. example.com in this case.
2. The “envelope sender”, appearing in the MAIL FROM part of the SMTP conversation of the relay transmission. This could be example.com (the simple approach), but also something like bounce.isp.com. This is because the envelope sender is the bounce address, and some mail relays make up some kind of bogus bounce address so they can track the bouncing mails.
3. The domain used in the HELO/EHLO part of the SMTP conversation of the relay transmission. Probably something like mail23.isp.com, as the ISP has many servers for relaying out.
4. The rDNS domain entry of the IP address of the sender on the relay transmission. If this entry doesn’t exist, or isn’t exactly as the HELO/EHLO domain, hang the postmaster. Some mail servers won’t even talk with you unless they match.

Dkim-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;        d=gmail.com;
 s=20161025;        h=mime-version:references:in-reply-to:from:date:message-id:subject:to;
        bh=vx0LAOXz3Mr8zS/Jy2ayKOep6NlflK3t+BpJyi78v9A=;
 b= [ ... ]

DMARC

Suppose that you own company Example Ltd. with domain example.com, and you’ve decided that all mails from (as in header From:) that domain will be DKIM signed. Now some spam mail arrives from someone else, without a DKIM signature and fails the SPF test. But the recipient has no way to tell that it should pass such tests.

DMARC is the mechanism that tells the recipient what to expect, and what to do if the expectation isn’t met. This allows the owner of the domain to ensure only mails arriving from its own machines are accepted. Spam pretending to come from its domain is dropped.

This is what Gmail did to force emails from all its users (i.e. having a gmail.com address) to be relayed through their servers only. The TXT for _dmarc.gmail.com goes:

"v=DMARC1; p=none; sp=quarantine; rua=mailto:mailauth-reports@google.com"

In other words, if it isn’t proven to come from Gmail’s server, hold the message. Most servers just junk it.

And now to how the test is done. Spoiler II: DMARC isn’t interested in a domain test if it isn’t tightly linked with the “From:” header’s domain. Or as they call it: Aligned with the RFC5322.From field. This is huge difference.

Let’s take it directly from RFC7489, section 4.2:

A message satisfies the DMARC checks if at least one of the supported authentication mechanisms:

1. produces a “pass” result, and
2. produces that result based on an identifier that is in alignment, as defined in Section 3.

The “supported authentication mechanisms” for DMARC version 1 are SPF and DKIM, as listed in section 4.1 of the same RFC.

The first thing we learn is that it’s enough to pass one of SPF or DKIM. No need to have both for passing DMARC.

Second, the term “is in alignment” above. It’s defined in the RFC itself, and essentially means that the domain for which the SPF or DKIM passed is the same as the one in the From: header, possibly give or take subdomains. The only reason they didn’t just say that the domains must be equal is because of the possibility of “relaxed mode”, allowing an email from myself@mysubdomain.example.com to be approved by passing tests with the example.com domain. This is what “being in alignment” means in relaxed mode. In “strict mode” alignment occurs only when they’re perfectly equal.

If the email passes the DMARC test, there isn’t much to fuss about. If it fails, the decision what to do depends on the policy, as given in the relevant domain. Which, according to RFC7489 section 3 is: “Author Domain: The domain name of the apparent author, as extracted from the RFC5322.From field”. And then in section 4.3, item 7: “The DMARC module attempts to retrieve a policy from the DNS for that domain” (referring to the Author Domain).

So it’s a DNS query for the TXT record of the From: domain, with the “_dmarc” subdomain prepended. As in the example above for gmail.com.

Finally, a tricky point. If a mail server, for which the SPF test is made, didn’t use the Author Domain in its MAIL FROM nor in the HELO/EHLO, the SPF test is worthless for DMARC purposes. It’s however quite tempting to check the Author Domain for its SPF record nevertheless. I mean, if the Author Domain allows the IP address of the mail relay server, isn’t it good enough to pass a DMARC test? Doing this goes against the SPF’s RFC, and isn’t mentioned in any way in DMARC’s RFC. But it makes a lot of sense. I won’t be surprised if it’s common practice already.

Will DMARC make my email delivery better?

TL;DR: Surprisingly enough, yes.

The irony about DMARC is that it bites on the spam messages, and does very little on the legit ones. After all, if an email passed both the SPF and DKIM tests on the Author Domain, what is there left to say?

And if the same email passed only one of the tests, why would a DMARC record add reassurance?

Of course, if you want to fake mails pretending to be you, definitely apply DMARC.

But once again, noone knows how spam filter X behaves. Maybe someone found out that DMARC signed domains carry less spam, and tuned the filter in favor of them. And maybe the rejection of spam mails thanks to the DMARC record helped with the domain’s spam statistics. Even though I would expect any machine that maintains statistics to count the emails that pass SPF / DKIM tests separately.

And here comes the big surprise. Gmail refused to accept messages from my server until I added a DMARC record. Once I did it, I was all welcome. It makes no sense, but somehow, Google seems to like the very existence of a DMARC record. Maybe a coincidence, most likely not. So do yourself a favor, at the very least add a TXT record to _dmarc.yourdomain.com:

v=DMARC1; p=none; sp=none; ruf=mailto:mailreports@yourdomain.com

This record tells the recipient to do nothing with a mail message that fails the DMARC test, so it’s harmless. But it will send an email to tell you about it to the email address given. Which can be useful in itself.

Actually, make that “p=reject” and “sp=reject” as soon as you’re sure it won’t bite back on your own mails. Once again, it seems to improve deliverability with Gmail, and possible also the overall statistics: Even through spam emails don’t meet the DMARC criterion, if they’re not discarded, they might circulate and wreck the domain’s reputation. So kill them while they’re small.

Conclusion

There might be official rules for entering a club, but in the end of the day, you can’t know what the doorkeeper looks at. So try to get everything as tidy as possible, and hope you won’t be mistaken for the bad guys.

And don’t wait for the first time you won’t be let in. It might be too late to fix it then.