SafeMIMEText.set_payload() crash on Python 3.11.9

[Sarah Boyce]

A recent bug fix to Python ​gh-76511 has been back ported to Python 3.11 and 3.12.
See the release notes of ​Python 3.11.9 and the soon to be released ​Python 3.12.3.

As part of the work to prepare Django for Python 3.13, a fix for this has already been merged to main in b231bcd19e57267ce1fc21d42d46f0b65fdcfcf8.

Believe this needs to be back ported to Django 4.2 and Django 5.0 to support the latest (or soon to be latest) Python 3.11 and Python 3.12 versions.

Confirmed when testing Django 4.2 against Python 3.11.9, we know have the following test failure:

======================================================================
ERROR [0.034s]: test_dont_base64_encode (tests.mail.tests.MailTests.test_dont_base64_encode)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tests/django/tests/mail/tests.py", line 902, in test_dont_base64_encode
    s = msg.message().as_string()
        ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tests/django/django/core/mail/message.py", line 133, in as_string
    g.flatten(self, unixfrom=unixfrom, linesep=linesep)
  File "/usr/local/lib/python3.11/email/generator.py", line 115, in flatten
    self._write(msg)
  File "/usr/local/lib/python3.11/email/generator.py", line 180, in _write
    self._dispatch(msg)
  File "/usr/local/lib/python3.11/email/generator.py", line 217, in _dispatch
    meth(msg)
  File "/usr/local/lib/python3.11/email/generator.py", line 246, in _handle_text
    msg.set_payload(msg._payload, charset)
  File "/tests/django/django/core/mail/message.py", line 168, in set_payload
    has_long_lines = any(
                     ^^^^
  File "/tests/django/django/core/mail/message.py", line 169, in <genexpr>
    len(line.encode()) > RFC5322_EMAIL_LINE_LENGTH_LIMIT
        ^^^^^^^^^^^^^
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 28-33: surrogates not allowed

[Natalia Bidart]

Related forum discussion: ​https://forum.djangoproject.com/t/send-mail-throws-unicodeencodeerror-under-python-3-11-9/29871/3

[Natalia Bidart]

PR for main adding a test (to be backported along with the fix from b231bcd19e57267ce1fc21d42d46f0b65fdcfcf8):
​https://github.com/django/django/pull/18057/

[Sarah Boyce <42296566+sarahboyce@…>]

In 338ec052:

Refs #35361 -- Added test for Email line length checks when dealing with surrogate pairs.

Refs #33173, #34118 and #34900.

[Sarah Boyce <42296566+sarahboyce@…>]

In 42435fc5:

Fixed #35361 -- Added release notes for 4.2.12 and 5.0.5 for backport of b231bcd19e57267ce1fc21d42d46f0b65fdcfcf8.

[Sarah Boyce <42296566+sarahboyce@…>]

In 2e6ae1e9:

[5.0.x] Refs #34900, Refs #35361 -- Fixed SafeMIMEText.set_payload() crash on Python 3.13.

Payloads with surrogates are passed to the set_payload() since
​https://github.com/python/cpython/commit/f97f25ef5dfcdfec0d9a359fd970abd139cf3428

Backport of b231bcd19e57267ce1fc21d42d46f0b65fdcfcf8 from main.

[Sarah Boyce <42296566+sarahboyce@…>]

In 23d605d3:

[5.0.x] Fixed #35361 -- Added release notes for 4.2.12 and 5.0.5 for backport of b231bcd19e57267ce1fc21d42d46f0b65fdcfcf8.

Backport of 42435fc55cbf7c04c1389ee46cc50e2565b40e37 from main.

[Sarah Boyce <42296566+sarahboyce@…>]

In 0d3ddcaf:

[4.2.x] Refs #34900, Refs #35361 -- Fixed SafeMIMEText.set_payload() crash on Python 3.13.

Payloads with surrogates are passed to the set_payload() since
​https://github.com/python/cpython/commit/f97f25ef5dfcdfec0d9a359fd970abd139cf3428

Backport of b231bcd19e57267ce1fc21d42d46f0b65fdcfcf8 from main.

[Sarah Boyce <42296566+sarahboyce@…>]

In 27c32cc:

[4.2.x] Fixed #35361 -- Added release notes for 4.2.12 for backport of b231bcd19e57267ce1fc21d42d46f0b65fdcfcf8.

Backport of 42435fc55cbf7c04c1389ee46cc50e2565b40e37 from main.

[Sarah Boyce <42296566+sarahboyce@…>]

In cd823778:

Refs #35361 -- Clarified release notes for 4.2.12 and 5.0.5.

[Sarah Boyce <42296566+sarahboyce@…>]

In a1baaddb:

[5.0.x] Refs #35361 -- Clarified release notes for 4.2.12 and 5.0.5.

Backport of cd823778e66307b82469858cfd8d1aa75613b49a from main.

[Sarah Boyce <42296566+sarahboyce@…>]

In 1d85b416:

[4.2.x] Refs #35361 -- Clarified release notes for 4.2.12.

Backport of cd823778e66307b82469858cfd8d1aa75613b49a from main.

[Anders Kaseorg]

A disadvantage of the current May 6 schedule for the release with these Python 3.11.9 and 3.12.3 compatibility fixes is that Ubuntu 24.04 LTS will be releasing with Python 3.12.3 on ​April 25.