Incorrect folding of long address headers with special characters when using 7bit Content-Transfer-Encoding in EmailMessage

[andres]

Hi!

I'm getting errors when trying to send email messages when the "to" header has non-ASCII chars.
The problem seems to happen when all these conditions are true at the same time:

• the readable name part of the recipient is big (e.g. "A very long and big name for this recipient" <to@…>)
• there is at least one non-ASCII char
• there is at least one "special char" (only seems to happen with comma or parenthesis)
• the special char isn't close to or between no-ASCII chars

Code example:

import email
from django.core.mail import EmailMessage

recipient = '"A véry long name with non-ASCII char and, comma" <to@example.com>'

msg = EmailMessage(from_email='from@example.com', to=[recipient]).message()
msg.policy = email.policy.default.clone(cte_type='7bit')
print(msg.as_bytes())

That prints a bytes string with:

\nTo: A =?utf-8?q?v=C3=A9ry?= long name with non-ASCII char and, comma\n <to@example.com>\n

It has an unprotected comma, and the message is rejected by ESPs.
Using default policy fix this problem, but possibly causes other bad consequences:

\nTo: =?utf-8?q?A_v=C3=A9ry_long_name_with_non-ASCII_char_and=2C_comma?=\n <to@example.com>\n

More details here: ​https://github.com/anymail/django-anymail/issues/369

[Sarah Boyce]

Thank you for the report!
I can replicate and the discussion in ​https://github.com/anymail/django-anymail/issues/369 is also very useful, thank you.

I'm pretty sure the problem only occurs with Python email's legacy email.message.Message, which uses a ​different folding algorithm (via email.policy.compat32) than modern email.message.EmailMessage (email.policy.default) uses.
My understanding is the Compat32 legacy layer is there specifically to replicate Python 2's email behavior (including any bugs), so there's not much point in reporting bugs against it.

Agree that #31784 is also somewhat related.

[Lufafa Joshua]

​PR

[andres]

Thanks, Lufafa Joshua! I manually reproduced your fix here, did a quick test and it seems to work!

[Mike Edmunds]

[I'm responsible for the example cited above. Setting an existing message's policy attribute like that isn't documented anywhere, so probably shouldn't be considered a valid test case. Apologies for that.]

I've come to the conclusion that any attempt to use modern email.policy.default to serialize an email.message.Message (which is constructed with legacy email.policy.compat32) is prone to errors, and probably just isn't supported by the Python email package. Additional information in ​https://github.com/anymail/django-anymail/issues/369#issuecomment-2184252097

I would suggest not trying to fix this particular issue within Django. Long term, there would be a lot of advantages to upgrading django.core.mail to use modern email.message.EmailMessage (and email.policy.default). But that's a separate issue, and I think needs to happen all at once; trying to mix legacy and modern email code seems problematic.

Here's an updated test, using only documented Python email package features (generator policy override). It has the same problem, but I think clarifies this is more of a Python email bug than a Django problem:

import email.generator
import email.policy
import io
import django.core.mail

to = '"Người nhận a very very long, name" <to@example.com>'
mime_message = django.core.mail.EmailMessage(to=[to]).message()

fp = io.BytesIO()
g = email.generator.BytesGenerator(
    fp, 
    policy=email.policy.default.clone(cte_type="7bit"),
)
g.flatten(mime_message)
print(fp.getvalue().decode("ascii"))
# [... other headers ...]
# To: =?utf-8?b?TmfGsOG7nWkgbmjhuq1u?= a very very long, name <to@example.com>

[Sarah Boyce]

Thank you for the update Mike, updated the ticket accordingly
Also following the discussion of ​https://groups.google.com/u/2/g/django-developers/c/2zf9GQtjdIk which is nice to see 👍

[Mike Edmunds]

(Reported Python email bug as ​python/cpython#121284 )