Upgrade django.core.mail to use Python's modern email API

[Mike Edmunds]

django.core.mail.EmailMessage currently uses ​Python's "legacy" email API. Updating that code to Python's newer ​"modern" email API would fix bugs and substantially simplify it.

​django-developers discussion

The update should be mostly transparent for users of Django's documented mail APIs, but will rework the internals of django.core.mail.message in ways that may require changes to some third-party email backends and other code that borrows undocumented (but popular) functions from that module. In addition, parts of Django's mail test suite will need reworking to avoid dependence on legacy implementation specifics.

Some relevant posts from django-developers:

• ​Background and rationale for switching to Python's modern email API
• ​Proposed implementation details (some of these were modified in later discussion—in particular, we ​don't expect to combine EmailMultiAlternatives into EmailMessage as part of this ticket, and we ​won't deprecate MIMEBase attachments just yet)
• ​Deprecation details for the documented BadHeaderError, SafeMIMEText and SafeMIMEMultipart, and the undocumented forbid_multi_line_headers() and sanitize_address()

[YashRaj1506]

I will try to solve this.

[Mike Edmunds]

@YashRaj1506 I have a bunch of work underway on this and will open a PR soon. Don't want you to needlessly duplicate effort. Code review on the PR would be very welcome once it's ready.

Part of the delay is I've run into a handful of open issues in Python's modern email API that may require further discussion:

• Modern set_content() forces a trailing newline on text content (​https://github.com/python/cpython/issues/121515). I think this is probably fine, but it touches a lot of tests.
• Security issues in modern header serialization (​https://github.com/python/cpython/issues/80222, ​https://github.com/python/cpython/issues/121284). I think these would block this change.

Also trying to decide how best to handle Django's EmailMessage.encoding. It's not documented, but a bunch of tests cover setting it and checking the results. (The only relevant docs are ​this note about the DEFAULT_CHARSET setting also applying to django.core.mail.)

[Mike Edmunds]

I've opened two sequential PRs, as suggested in the django-developers discussion. Both are ready for review:

• ​https://github.com/django/django/pull/18502 improves the test suite for django.core.mail while staying on the existing (legacy) API.
• ​https://github.com/medmunds/django/pull/2 implements the changeover to the modern email API.

(The second PR is in my own Django fork so that it can be based on the first PR without overlap. I'll reopen it upstream once the first PR is closed. But reviews are welcome now if you don't mind working in my fork; just note any discussion history there won't transfer to the eventual upstream PR.)

Some additional notes are in the PR comments.

[Sarah Boyce <42296566+sarahboyce@…>]

In 889be2f:

Refs #35581 -- Clarified some test names and comments in mail tests.

[Sarah Boyce <42296566+sarahboyce@…>]

In 4d76adfa:

Refs #35581 -- Verified attachments in the generated message in mail tests.

This also removed send() calls, as this doesn't check the serialized content, and
the backend tests cover sending.

[Sarah Boyce <42296566+sarahboyce@…>]

In 00861c4c:

Refs #35581 -- Identified mail tests that check for Python 2 behavior.

This also removed a duplicate CTE case (that used to be distinct in Python 2).

[Sarah Boyce <42296566+sarahboyce@…>]

In cf4d902:

Refs #35581 -- Reduced boilerplate in mail tests.

[Sarah Boyce <42296566+sarahboyce@…>]

In bddd35c:

Refs #35581 -- Improved reporting for failing tests in mail tests.

• Converted HeadersCheckMixin to MailTestsMixin for all shared helpers:
  • Hoisted assertStartsWith() from BaseEmailBackendTests.
  • Added matching assertEndsWith().
  • Hoisted get_decoded_attachments() from MailTests.
  • Improved failure reporting in assertMessageHasHeaders().
• Used unittest subTest() to improve handling of compound test cases.
• Replaced assertTrue(test on string) with custom assertions, so that failure reporting is more informative than True != False.

[Sarah Boyce <42296566+sarahboyce@…>]

In 5d7001b:

Refs #35581 -- Used modern email parser and helpers in mail tests.

• Used modern email API (policy.default) for tests that reparse generated messages, and switched to modern accessors where helpful.
• Split get_raw_attachments() helper out of get_decoded_attachments(), and used modern iter_attachments() to avoid finding nested attachments in attached message/* emails.
• Stopped using legacy parseaddr.

[Sarah Boyce <42296566+sarahboyce@…>]

In ea34de3b:

Refs #35581 -- Added tests for email parameters, attachments, MIME structure, bcc header, encoding and sending.

[Mike Edmunds]

I've opened ​PR 18966 that implements the change to Python's modern email APIs.

Unfortunately, it's blocked by a couple of (public) Python security issues. Absent some sort of holiday miracle, I think it's unlikely those will be fixed before the Django 5.2 feature freeze. So realistically, this is probably targeting 6.0. (Or later.)

[Mike Edmunds]

The PR includes a few changes that weren't covered by the original ​deprecation plan from the django-developers discussion:

1. Deprecated MIMEBase attachments: I had ​originally proposed continuing support for legacy MIMEBase attachments without deprecation. But it turns out the modern API's email.message.MIMEPart is a good replacement, so the PR puts MIMEBase on the usual deprecation path. I've also added a docs example of using MIMEPart for inline images.

2. Dropped undocumented subtype attributes: Removed support for the undocumented EmailMessage.mixed_subtype and EmailMultiAlternatives.alternative_subtype attributes. (But it will raise an error on attempts to use them.)

3. Dropped undocumented encoding=Charset(): The legacy code allows setting the undocumented EmailMessage.encoding property to a legacy email.charset.Charset object. There's no way to make that work with the modern APIs, so I've dropped it. (You can still set encoding to a string charset name, and that works properly.)