Support Unicode email addresses in SMTP EmailBackend (EAI, SMTPUTF8)

[Mike Edmunds]

This ticket proposes allowing use of the SMTPUTF8 extension in Django's SMTP EmailBackend, as part of fully supporting email address internationalization (EAI).

Among other things, EAI allows non-ASCII email addresses like jlópez@example.mx, where the local-part (username) can include Unicode characters. As of Django 5.1 the current behavior for that address is to send it using an invalid RFC 2047 encoding, which typically results in a bounced message. (#35713 would change that to the behavior described below for smtputf8=False.)

For more background on Internationalized Email see ​RFC 6530 or Wikipedia's ​International email. The SMTPUTF8 extension is defined by ​RFC 6531.

Much of the popular ​SMTP server software supports SMTPUTF8. However, deployment is not yet widespread: a July, 2023 analysis found only ~22% of unique MX servers advertised SMTPUTF8 support: ​UASG 047 UA-Readiness Report FY23 (PDF), sections 6.11–6.12.

Related tickets

Fully enabling EAI in Django requires three separate changes:

• Supporting SMTPUTF8 in the SMTP EmailBackend: this ticket.
• Allowing EmailMessage.message() to generate messages with ​RFC 6532 8-bit, utf-8 encoded headers. Updating to Python's modern email API will enable this: #35581. (Prerequisite for this ticket.)
• Allowing non-ASCII local-parts to pass Django's EmailValidator: #27029 and some discussion in #26423. (Not required for this ticket.)

To avoid creating a new setting, this ticket leverages:

• Dictionary based EMAIL_PROVIDERS settings, #35514. (Prerequisite for this ticket.)

(Also, #33969 requested something similar and was closed "needsinfo.")

Proposal

Add a new smptutf8 option to backends.smtp.EmailBackend.__init__(). Valid values would be:

• True: unconditionally use SMTPUTF8. Generate messages with 8-bit headers using email.policy.SMTPUTF8, and request the SMTPUTF8 extension in smtplib's sendmail(..., mail_options)—raising an error if the server doesn't support it.
• False: never use SMTPUTF8. Raise an error when attempting to send a message with non-ASCII local-parts in any email address.
• None: ("auto mode", the default) if an individual message includes any non-ASCII local-parts in any email address, try to use SMTPUTF8 for that message—and raise an error if the server doesn't support it. For all other messages, don't try to use SMTPUTF8.

The smtputf8 backend option would not be exposed as a separate setting. It would only be available through kwargs to get_connection() or the upcoming EMAIL_PROVIDERS dict setting.

The True setting is intended for use with an SMTP host that is known to support SMTPUTF8. It offers a slight performance advantage over the None option. Also, it would be the only way to use SMTPUTF8 (rather than RFC 2047 encoded headers) for messages that have non-ASCII characters in headers like Subject, but only ASCII local-parts in addresses.

When the SMTP backend uses SMTPUTF8 for a message (either via True or auto mode), it will also pass any non-ASCII domains (IDNs) in email addresses directly to the SMTP server, rather than performing its own IDNA encoding.

[Claude Paroz]

Thanks for the thorough proposal. Setting it as Someday/Maybe because it depends on other tickets. Once all blockers are merged, we can pass the ticket to Accepted.