Django crashes on unicode characters in the local part of an e-mail address

[Sergei Maertens]

With Python 3.5 and Django 1.9 I'm running into trouble with internationalized e-mail addresses. According to RFC 6532 it is possible to have unicode characters in the e-mail address: ​https://tools.ietf.org/html/rfc6532.html and this RFC *should* be supported in Python 3.5: ​https://docs.python.org/3/whatsnew/3.5.html#email. Steps to reproduce are at the bottom of the ticket.

Now, for validating e-mail addresses in various places, Django calls stdlib mail.formataddr, which doesn't seem to respect this RFC - not even if you explicitly set the EmailPolicy.utf8 (see ​https://docs.python.org/3/library/email.policy.html#email.policy.EmailPolicy.utf8) as I see no reference to that in the source code.

This function in the stdlib blatantly calls address.encode('ascii'). Luckily, it's quite short, and I would suggest rolling 'our own' formataddr function (for the time being). I'll bring this issue up on the Python bug tracker as well. I think it's possible, as it's a relative simple function of only 40 LoC, of which 12 lines docstring.

Steps to reproduce
Basically shell output:

mkdir bug_email
cd bug_email
mkdir bug_email -p python3.5
(bug_email)➜  bug_email  python --version
Python 3.5.1
(bug_email)➜  bug_email  pip install Django==1.9
Collecting Django==1.9
  Using cached Django-1.9-py2.py3-none-any.whl
Installing collected packages: Django
Successfully installed Django-1.9
(bug_email)➜  bug_email  django-admin.py startproject bug_email .

# regular shell is enough to test
(bug_email)➜  bug_email  ./manage.py shell

Shell session

>>> from django.core.mail.message import sanitize_address
>>> sanitize_address(('dummy', u'juan.lópez@abc.com'), 'utf8')
Traceback (most recent call last):
  File "/home/bbt/coding/.virtualenvs/bug_email/lib/python3.5/site-packages/django/core/management/commands/shell.py", line 69, in handle
    self.run_shell(shell=options['interface'])
  File "/home/bbt/coding/.virtualenvs/bug_email/lib/python3.5/site-packages/django/core/management/commands/shell.py", line 61, in run_shell
    raise ImportError
ImportError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/bbt/coding/.virtualenvs/bug_email/lib/python3.5/site-packages/django/core/mail/message.py", line 118, in sanitize_address
    return formataddr((nm, addr))
  File "/usr/lib64/python3.5/email/utils.py", line 91, in formataddr
    address.encode('ascii')
UnicodeEncodeError: 'ascii' codec can't encode character '\xf3' in position 6: ordinal not in range(128)
>>> 
>>> sanitize_address(('dummy', u'juan.lópez@abc.com'), 'idna')
Traceback (most recent call last):
  File "/home/bbt/coding/.virtualenvs/bug_email/lib/python3.5/site-packages/django/core/management/commands/shell.py", line 69, in handle
    self.run_shell(shell=options['interface'])
  File "/home/bbt/coding/.virtualenvs/bug_email/lib/python3.5/site-packages/django/core/management/commands/shell.py", line 61, in run_shell
    raise ImportError
ImportError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/bbt/coding/.virtualenvs/bug_email/lib/python3.5/site-packages/django/core/mail/message.py", line 118, in sanitize_address
    return formataddr((nm, addr))
  File "/usr/lib64/python3.5/email/utils.py", line 91, in formataddr
    address.encode('ascii')
UnicodeEncodeError: 'ascii' codec can't encode character '\xf3' in position 6: ordinal not in range(128)
>>> 

[Claude Paroz]

Reporting the issue in Python would be the first step. Could you please report here the ticket number as soon as it is done?

[Sergei Maertens]

Python bug tracker issue: ​http://bugs.python.org/issue25955

[Sergei Maertens]

The upstream ticket has been closed, with the following comment:

formataddr is part of the legacy interface and has no knowledge of the current policy. So it doesn't support RFC 6532. For that you need to use the new API: just assign your address to the appropriate field, or create a headerregistry.Address object.

Looks like this will have to be solved on Django's end after all. I'm not too familiar with the e-mail library yet, but I could look into it when I get some more spare time.

[Sergei Maertens]

As I was implementing this, some extra information:

• The RFC 6532 is a red herring. The actual issue was that the unicode characters were not properly ​MIME word-encoded. To support said RFC, an extra mail-server plugin must be enabled, so I went the safe way for Django where that's not needed.

• There are major differences between Python 2 and 3. This is the case in:
  • the FakeSMTPServer in the testcases, where the mailfrom variable might not be MIME-word-encoded all the way.
  • the usage of Header(<string>, <encoding), where the str representation calls the encode method on Python 2 and on Python 3 it simply returns the initial <string> that was passed in.

Potential issue:
Because of the difference in str representation, that has been altered to always call the encode method. This causes simple ascii local parts to look garbled, for instance: to@example.com becomes =?utf-8?q?to?=@example.com. When django users test the e-mail messages generated by Django, they may have failing tests because they're not expecting the encoded version. I have not personally confirmed this yet though.

[Tim Graham]

Left comments for improvement on the PR.

[Sergei Maertens]

Remarks were processed, up for review again (if the build succeeds).

[Tim Graham <timograham@…>]

In ec009ef1:

Fixed #25986 -- Fixed crash sending email with non-ASCII in local part of the address.

On Python 3, sending emails failed for addresses containing non-ASCII
characters due to the usage of the legacy Python email.utils.formataddr()
function. This is fixed by using the proper Address object on Python 3.

[Mike Edmunds]

Does anyone know of an email server or service where this sort of non-ASCII mailbox works?

Sorry to unearth this ancient ticket. I'm hoping one of the original participants or cc's might have more info.

This ticket attempted to add support for non-ASCII characters in the local part (username) of an email address: e.g., j.lópez@example.com . But the implementation used an RFC 2047 encoded-word (=?utf-8?b?ai5sw7NwZXo=?=@example.com). RFC 2047 specifically prohibits that in an addr-spec (email address). And I've been unable to find any email systems that actually support this behavior. At best, the message bounces—and sometimes it just disappears without a trace.

If anyone has an example where RFC 2047 encoded-word email addresses work, I'll put some effort into preserving the current behavior as part of updating to Python's modern email API (#35581). Or if nobody is actually using this successfully, I'm going to treat it as a bug.

More info:

• ​RFC 2047 section 5(3) specifies that: "An 'encoded-word' MUST NOT appear in any portion of an 'addr-spec'." (And I can't find any later RFC that removes this restriction.)
• I can't find any email software or services that support RFC 2047 in an email localpart. The closest is Exim, which can ​convert a localpart to/from an A-Label (an ​RFC 5891 IDNA encoding, a.k.a. "punycode": j.xn--lpez-qqa@example.com). But that's a different encoding, and this behavior seems to be unique to Exim.
• From what I can tell, the only spec compliant way to use a non-ASCII localpart involves keeping utf-8 un-encoded in the headers, and then sending with the SMTPUTF8 extension (​RFC 6530 / ​RFC 6531 / ​RFC 6532). There doesn't seem to be any spec that allows a non-ASCII localpart with 7-bit header encoding.
• The Python behavior seems to be a bug. Note that the same Python code also converts a non-ASCII domain name to an RFC 2047 encoded-word, which is definitely not deliverable: juan@lópez.example.mx → juan@=?utf-8?q?l=C3=B3pez?=.example.mx. This is an ​acknowledged bug in Python's email package.

(Incidentally, RFC 2047 encoded words are allowed in a display-name: J. López <jlopez@example.com> → J. =?utf-8?q?L=C3=B3pez?= <jlopez@example.com>. That worked before and after this ticket, and won't be changed by #35581.)

[Sergei Maertens]

It's been a long time, but I *think* this had to do with a test suite of a third party package (I want to say django-yubin?) that started failing because of the Python 2 -> 3 differences in behaviour, and that made me dive into it. I've never actually come across such an email, and for some project recently looked up the address-part-with-non-ascii-characters possibility and reported to them that no additional work is needed because it shouldn't be allowed in the first place.

Likely I assumed at the time that the "new" Python behaviour was correct.

[Mike Edmunds]

Thanks, Sergei, that's very helpful. I'll plan to remove this behavior.