Can't send unicode e-mail

[carneiro.be@…]

Hello guys,

I'm on django dev and having troubles sending unicode emails. Here's the relevant code:

# person is a normal django user
subject = u"Welcome to Odeio - %s" % person.name
message = render_to_string('accounts/registration_email.txt', ctx_dict)
person.email_user(subject, message)

Error:

'ascii' codec can't encode character u'\xe1' in position 2: ordinal not in range(128)

Stacktrace:

File "/odeio/django/core/mail/__init__.py", line 62, in send_mail
    connection=connection).send()
 
  File "/odeio/django/core/mail/message.py", line 251, in send
    return self.get_connection(fail_silently).send_messages([self])
 
  File "/odeio/django/core/mail/backends/smtp.py", line 91, in send_messages
    sent = self._send(message)
   
  File "/odeio/django/core/mail/backends/smtp.py", line 106, in _send
    self.connection.sendmail(from_email, recipients, email_message.message().as_string())
 
  File "/odeio/django/core/mail/message.py", line 141, in as_string
    g.flatten(self, unixfrom=unixfrom)
 
  File "/opt/bitnami/python/lib/python2.6/email/generator.py", line 84, in flatten
    self._write(msg)
 
  File "/opt/bitnami/python/lib/python2.6/email/generator.py", line 109, in _write
    self._dispatch(msg)
 
  File "/opt/bitnami/python/lib/python2.6/email/generator.py", line 135, in _dispatch
    meth(msg)
 
  File "/opt/bitnami/python/lib/python2.6/email/generator.py", line 178, in _handle_text
    self._fp.write(payload)
 
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 2: ordinal not in range(128)

Thanks in advance for the help!

[Claude Paroz]

Can you isolate if the encoding error is due to unicode in subject, in message, or both? If it is related to message, can you also give us the output of the render_to_string call?

[carneiro.be@…]

Hello claudep,
thank you for your reply. I believe the encoding error is due to unicode in the message, because the character at position 2 in the message is "á", which is not ASCII. It seems like the subject is ok, even though it might contain unicode characters (person.name might be unicode). Here's the output of render_to_string:

Olá My name,

seja bem-vindo ao sistema de administração jurídica online Odeio. O escritório Departamento Jurídico cadastrou-lhe no sistema. Os seus dados são:

Nome: My name
Usuário / Email: my-email@gmail.com

Para validar o seu cadastro e criar uma senha, entre no seguinte link:
http://odeio.com.br/pessoas/validar/130281cba7106776845efc1ab274776b02ff5206/

Qualquer dúvida, sugestão ou crítica não hesite em entrar em contato. Seu feedback é muito bem-vindo.

Obrigado,
Equipe Odeio

Thanks again for your help.

[carneiro.be@…]

I have just double checked the subject and message. Removing all unicode characters from the message results in the message being sent, even when I added unicode characters to the subject.

[Claude Paroz]

Thanks, can you still tell us the exact version of Python, the encoding of the file, the value of settings.FILE_CHARSET, settings.DEFAULT_CHARSET and sys.getdefaultencoding()?

[carneiro.be@…]

Hi claudep,

thanks for the quick reply. My python version is Python 2.6.5. I haven't set FILE_CHARSET neither DEFAULT_CHARSET, so I assume it's utf-8. I'm going to set it manually and try again. sys.getdefaultencoding() returns 'ascii'. For the file I did this: file -i registration_email.txt
registration_email.txt: text/plain; charset=utf-8. So I guess it should be okay.

I've tried again setting FILE_CHARSET and DEFAULT_CHARSET to utf-8 but it still fails with the same stacktrace.

If you want I can change the code to introduce any tests you wish. Many thanks again for the help.

[Claude Paroz]

I think i get it. This issue is probably related to ​http://bugs.python.org/issue1368247 that has been fixed in Python 2.6.6 (​http://hg.python.org/cpython/rev/dd378b885b94/). I will attach a patch with a workaround for Python < 2.6.6. Can you test it?

[Claude Paroz]

Workaround for python bug 1368247

[carneiro.be@…]

OH. MY. GOD. Seriously, are you a wizard? Because that's some serious black magic you just performed. That did fix it. How did you manage to find such bug in python? Wow... many many many many many thanks. That'd have taken me 10 years to figure out.

[Claude Paroz]

Great :-) It is not so magic. I couldn't reproduce the failure. The only obvious difference between your setup and mine was the Python version. So I looked for the NEWS file of recent Python 2.6 releases and searched for issues related to MIME. I admit I was lucky to find a match so quickly.

[Claude Paroz <claude@…>]

In 501d793398c3ecc3c8a54475d07609cf34643553:

Fixed #19107 -- Workarounded message-encoding bug on Python < 2.6.6

Thanks Bernardo Pires for the report.

[Ben Kornrath]

The fix seems to have been removed from the latest version of Django as I'm getting this problem with Django 1.6.5 on Python 2.6.5.

Code that demonstrates the problem (replace with valid emails):

from core.models import Setting
from django.core.mail import send_mail
send_mail('hello', u'マウス', 'noreply@example.com', ('test@example.com',))

System information:

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=10.04
DISTRIB_CODENAME=lucid
DISTRIB_DESCRIPTION="Ubuntu 10.04.4 LTS"

$ python --version
Python 2.6.5

I'll attach the patch I'm using to fix the issue. Should I re-open this issue? Should I create a pull request for this? Thanks.

[Ben Kornrath]

Patch to fix this issue on Django 1.6.5.

[Tim Graham]

Looks like the code was removed in 35a447a263115b22a7040a9f4a16ff4bd2078466. I'm not sure it was intentional, but I think we should probably add it back since Ubuntu 10.04 server with that version of Python is supported until April 2015. I will check with Florian.

[Tim Graham <timograham@…>]

In e4b2bea743944b67a55d0ef3d17982a2319f866d:

[1.6.x] Fixed #19107 -- Restored bug fix for sending unicode email with Python 2.6.5 and below.

[Greg Humphreys]

I'm still encountering this bug on Ubuntu 10.04 with Django 1.6.6, even though your patch has landed there. In fact, upgrading to Django 1.6.6 (from 1.4.1) was what caused the issue.

The stack trace is the same as in the OP. What information would be useful to help reproduce?

[Greg Humphreys]

Here's the portion of the email that appears to be causing issues (printed with 'print email.dict'):

Hi\r\n\r\nWhen browsing with chrome on my Android tablet (Samsung Galaxy 12 Note Pro), the suit symbols don\u2019t show *in the diagrams*, which is of course only a problem for the bidding (it\u2019s ok in the text, although without colors).

[Tim Graham]

A working, tested patch would be needed as I don't have motivation to reproduce and fix the issue myself.

[Claude Paroz]

Our FAQ is clear: ​https://docs.djangoproject.com/en/dev/faq/install/#what-python-version-can-i-use-with-django.
You should really try to install the most recent 2.6 Python version. Unfortunately, it seems Ubuntu doesn't care much (​https://bugs.launchpad.net/ubuntu/+source/python2.6/+bug/788525).

[Ben Kornrath]

The fix in Django 1.6.6 using Python 2.6.5 in Ubuntu 10.04 is working for me. @Humper: You could try updating to the latest version of the Python 2.6.5 package. I think the Ubuntu devs have back-ported some fixes from later versions of Python to the 2.6.5 package and that might be why things are working for you. Just an idea though.