Inability to define DNS_NAME in django.core.mail results in e-mail messages being rejected or marked as spam

[Franklin]

Summary
Being the case that the application server's hostname may not necessarily be a valid fully qualified domain name (FQDN) the use of socket.getfqdn() in defining DNS_NAME may yield a number of e-mail messages either rejected or marked as spam. Rejection issues arise if settings.EMAIL_USE_TLS is True - meaning smtplib.ehlo() will be called - and if a mail server is particularly strict (or misconfigured) and rejects any ehlo command followed by an invalid FQDN. Spam issues arise with DNS_NAME being used as the local_hostname argument to smtplib.SMTP() within the SMTPConnection.open() method, meaning the initial "Received:" e-mail header may be from an invalid FQDN. Further increasing the chance of being marked as spam is DNS_NAME being used by the make_msgid() method, leaving the "Message-ID:" e-mail header ending in an invalid FQDN.

Real World Scenario
A shared server has a hostname, which you may not and can not change, of "star-wars-reference". The socket.getfqdn() method fails to find info on its IP address and by default simply returns said hostname. E-mail is then sent by Django with a Message-ID of "200804....@star-wars-reference" from a server claiming to be "star-wars-reference ([server's IP address])" and is, of course, marked as spam.

Solution
Provide an optional settings variable such as EMAIL_LOCAL_HOSTNAME where Django developers may define a FQDN which maps to their application's server. EMAIL_LOCAL_HOSTNAME will be used by the class defining DNS_NAME in lieu of socket.getfqdn() if EMAIL_LOCAL_HOSTNAME exists in the settings file (this has the added benefit of preventing a call to socket.getfqdn() - further reducing overhead on restarts).

[stilldodge]

This still appears to be an issue. Possibly this was unresolved because email servers used to accept an invalid FQDN. I can confirm that Exchange 2007 rejects the connection if the local host "calculated" name is not truly a valid FQDN

[Jeremy Grosser]

I've attached an updated and documented patch (mail-local-hostname2.diff) allowing the user to specify EMAIL_LOCAL_HOSTNAME. Please review and let me know if any changes are necessary to get this merged.

[Adrian Holovaty]

I'm just skimming this, but my immediate reaction is: introducing a new setting is a non-starter. Is there a way to accomplish what you want without a setting?

[Jeremy Grosser]

We could force anybody needing this to subclass the SMTP EmailBackend, but that seems a bit excessive to me.

[Aymeric Augustin]

Change UI/UX from NULL to False.

[Aymeric Augustin]

Change Easy pickings from NULL to False.

[Edwin]

Here is a workaround.

Python's socket.getfqdn() resolution process works like this on Linux:

1. Get hostname of server.
2. Resolve IP address for hostname.
3. Resolve name for IP address.
4. If 2 or 3 fail, just use the hostname.

Therefore, if you add a line like this to your /etc/hosts, socket.getfqdn() will resolve myhostname to 127.0.0.1 and resolve 127.0.0.1 to myfqdn since that is the first name in the list. You also have to make sure that there are no other entries for 127.0.0.1 on previous lines.

127.0.0.1       myfqdn myhostname

[anonymous]

That workaround only makes sense if the server is hosting one django project. I have one Linux server with several django projects. Each uses a different fqdn but the same IP.

[davidgrant@…]

Can we get this in? Relying on socket.get_fqdn() seems very wrong and patch looks good.

[davidgrant@…]

So where do we go from here? Instead of a setting we could use an environment variable. Other than that I don't see any other way.

[Russell Keith-Magee]

@davidgrant - Where do we go? We go the same way we do with any other ticket that has stalled: Start a discussion on django-dev. Summarize the problem. Summarize the alternatives. Propose a solution.

I'll also note that Adrian's comment about settings -- we've historically avoided adding new settings wherever possible. But if it truly isn't possible (at all, or for all practical purposes) to avoid a setting, it's not completely off the cards.

[Tim Graham]

I think a setting is fine considering we added other mail settings recently. The patch needs to be updated, checked against the PatchReviewChecklist, and turned into a pull request.

[Diederik van der Boor]

Python's smtplib.SMTP works around this problem with a fallback to the IP address::

            # RFC 2821 says we should use the fqdn in the EHLO/HELO verb, and
            # if that can't be calculated, that we should use a domain literal
            # instead (essentially an encoded IP address like [A.B.C.D]).
            fqdn = socket.getfqdn()
            if '.' in fqdn:
                self.local_hostname = fqdn
            else:
                # We can't find an fqdn hostname, so use a domain literal
                addr = '127.0.0.1'
                try:
                    addr = socket.gethostbyname(socket.gethostname())
                except socket.gaierror:
                    pass
                self.local_hostname = '[%s]' % addr

This is overwritten as Django's django.core.mail.backens.smtp.EmailBackend overwrites this to:

connection_params = {'local_hostname': DNS_NAME.get_fqdn()}

One possible extra fix would be to support the same fallback to IP-addresses when there is no dot in the hostname, just like smtplib.SMTP.open() does.

[Diederik van der Boor]

To add, this issue also happens on Kubernetes pods.

A standard Pod on Kubernetes has the following configuration:

/etc/hosts

# Kubernetes-managed hosts file.
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
fe00::0	ip6-mcastprefix
fe00::1	ip6-allnodes
fe00::2	ip6-allrouters
10.244.1.204	examplepod-77f687b9b9-5hlsg

/etc/resolv.conf

nameserver 10.96.0.10
search NAMESPACE.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Hence email is submitted using EHLO examplepod-77f687b9b9-5hlsg and thus triggering spam filtering.

[Diederik van der Boor]

I bumbed into this again, in the end I had to fix this by patching:

django.core.mail.utils.DNS_NAME._fqdn = "my.desired.host"

[Jacob Rief]

Pull request to add this feature:
​https://github.com/django/django/pull/13728

However as a configuration directive I used EMAIL_MESSAGEID_FQDN because to me, that's more meaningful.
I'm open to discussions on the naming of this.

[Alexander Lazarević]

I'm trying to understand the status of this ticket and the problem. I traced the discussion here, on the PR and google groups ...

The end of the discussion in google groups seems to be:

Jacob Rief 31 Jan 2022, 15:36:40 to Django developers (Contributions to Django itself)
On Monday, January 31, 2022 at 7:55:47 AM UTC+1 f.apo...@gmail.com wrote:

    Okay then,

    some of the things like sender reputation and different bounce hooks came to my mind as well, but it is good to hear confirmation from others. I think the next steps would be to create a new ticket to add support for *multiple* email backends and then work from that (I would only link the old ticket since it's scope was mainly putting the config into a dict as opposed to multiple backends). Given that there are plenty of +1 here already I think we already have our implementors? :)

    Some items that I like to see addressed in a PR:
     * Backwards compat
     * Similarity to Caches & Databases (ie so we don't invent yet another syntax)
     * Support for connection aliases (default/…) in send_email % friends (basically everything taking a connection now should probably take aliases as well)

OK. Since I made the proposal, it then is probably up to me to create the ticket.  


Mariusz Felisiak 1 Jun 2023, 21:48:04 to Django developers (Contributions to Django itself)

It seems we have a consensus here to support multiple email backends. Jacob, feel-free to create a new ticket.

Best, Mariusz

I couldn't find the Ticket by Jacob or anyone related to this.

A related ticket #31885 got fixed and closed. In the description it is said that "We will then recommend subclassing in the docs for more control. "

Is subclassing now also the proposed method for setting a custom DNS_NAME (or other option)? (I does not seemed to be mentioned in the docs)

Is the support for multiple email backends abandoned for now?

[Mariusz Felisiak]

Replying to Alexander Lazarević:

Is subclassing now also the proposed method for setting a custom DNS_NAME (or other option)? (I does not seemed to be mentioned in the docs)

Yes, subclassing is basically a way to go for any fancy change to the EmailBackend behavior.

Is the support for multiple email backends abandoned for now?

Not really, it looks like no one had time to create a ticket. Please do if you want.

[Mike Edmunds]

support for multiple email backends

See #35514 Dictionary based EMAIL_PROVIDERS settings