django.core.mail.EmailMessage._create_mime_attachment incorrectly encodes message/rfc822 attachments in base64, breaks in Evolution and Thunderbird

[Michael]

Django appears to insert message/rfc822 attachments in base64 encoding. This breaks display in Novell Evolution ​#651197 and Mozilla Thunderbird ​#333880.

This behaviour is similar to some other mail clients (namely Gmail and Microsoft Outlook), but is non-standard according to ​RFC2046:

5.2.1. RFC822 Subtype
No encoding other than "7bit", "8bit", or "binary" is permitted for the body of a "message/rfc822" entity.

The Mozilla Thunderbird bug entry seems to indicate that with certain versions of Thunderbird, this behaviour is accepted, but this seems to change sporadically. They believe that encoding message/rfc822 attachments in this fashion is not acceptable behaviour.

[Michael]

I've created a patch and pull request: ​https://github.com/django/django/pull/373

This patch is against Django master.

There are some changes in the behaviour of django.core.mail.EmailMessage.attach:

• message/rfc822 attachments may be a email.message.Message object (not just a string)
• When passing in a string attachment, Django will now automatically convert it to email.message.Message internally for email.mime.message.MIMEMessage. This means that if the message/rfc822 attachment is not well-formed, an exception may be thrown.
• Per RFC2046, the attached message headers must be ASCII encoded. As a result SafeMIMEMessage checks headers with ASCII encoding.

This also implements SafeMIMEMessage (like the other SafeMIME* classes).

The tests currently pass, however there are no specific tests in this patch for this functionality.

[Michael]

I've added regression tests for the bug, and fixed an issue where it was applying the transforms to all message/* mimetypes, and not just message/rfc822. I've also allowed the EmailMessage.attach method to take in Django's EmailMessage object as an argument for message/rfc822 objects.

I've updated my pull request (above) with these changes.

[Michael]

I notice the patch for this bug was marked as "Accepted" 7 months ago, however the patch for the issue has not been merged. I'm confused to what is going on.

This is still an outstanding bug, as Django continues to produce non-RFC2046 s5.2.1-compliant mail, which breaks in multiple popular email clients when attempting to send message/rfc822 attachments from Django.

I've now rebased this patch against the stable 1.5 tree as at today, however this has now got GitHub very confused, as it's pulled in hundreds of commits in the last 8 months. I've since created a new pull request that fixes this:

​https://github.com/django/django/pull/1202

If there is something needed from me to make this proceed please let me know. Thanks.

[Aymeric Augustin]

This ticket was marked as "Accepted" because the issue was recognized as valid.

Once someone has written a patch (with tests and docs if appropriate), someone else can review it, and mark the ticket as "Ready for Checkin" if the patch is good.

Then the patch can be merged.

A pull request is an appropriate way to provide a patch, however, it must be done against the master branch, not a stable branch.

[Michael]

I will write some documentation and rebase this against master.

How can I target the stable versions of Django for getting this patch in future bugfix releases? This bug affects Django 1.3, 1.4 and 1.5... though probably also other releases as well. I'm a bit upset that this was missed in the lead-up to 1.5.

I'm going to have to backport this patch to 1.5.x anyway, because it causes my application to fail in many of it's email-related tasks, as it is sending emails attached to emails quite frequently.

Trying to get this fixed in other email clients is an uphill battle, as it's non-RFC and has been an open bug in Thunderbird for 7 years.

[Michael]

I've written documentation for the change. I've got another pull request against django master:

​https://github.com/django/django/pull/1222

As this fixes a bug in Django, it would be great if this was also pulled into Django 1.5.2. Pull request 1202 has been updated against current stable/1.5.x.

This already has unit tests which identify the buggy behaviour.

[Luke Plant]

I'm afraid that are policy with bug fixes is that this can only go into master - it won't be backported. To ensure it gets into 1.6 it would help if you can find someone to review your patch.

[cuerty]

IMHO: This patch seems good.

I was able to reproduce the bug with the test provided and the patch fix it. Also the last version of the patch merges without problems with master.

[Ramiro Morales <cramm0@…>]

In f9d1d5dc1377cb21b39452b0897e7a79a3d02844:

Fixed #18967 -- Don't base64-encode message/rfc822 attachments.

Thanks Michael Farrell for the report and his work on the fix.

[Andrew Godwin <andrew@…>]

In 01223840f34ff2eacf1425bc133c347564fe2614:

Fixed #18967 -- Don't base64-encode message/rfc822 attachments.

Thanks Michael Farrell for the report and his work on the fix.