i18n crash on non-ASCII (UTF-8 encoded) doctrings

[AV <av0000@…>]

The Unicode decoding error occured in i18n.py when source file contains non-ASCII docstrings.

Example:

Navigate to yoursite/documentation/models using following model example (the source file is in utf-8 encoding)

# -*- coding: utf-8 -*-
# ...
class TestModel(models.Model):
    """
    Test description with non-ASCII symbols.
    АаБбВвГг
    """

Workaround #1: use u"""....""" format but not on TAGS - tag parser is different :(

Workaround/patch #2: patch or change django/templatetags/i18n.py TranslateNode.render: from translation.ugettext(value) to translation.ugettext(smart_unicode(value))

[Marc Garcia]

I can't reproduce the error. Adding non ascii characters to the docstrings on the model doesn't make django raise any encoding error. Could you provide more information about this issue please?

[Karen Tracey <kmtracey@…>]

I was able to recreate using the TestModel exactly as shown. However I don't believe that is a valid testcase. Per PEP 257 (​http://www.python.org/dev/peps/pep-0257/):

For Unicode docstrings, use u"""Unicode triple-quoted strings""".

That is, the model should be specified like so:

# -*- coding: utf-8 -*-
# ...
class TestModel(models.Model):
    u"""
    Test description with non-ASCII symbols.
    АаБбВвГг
    """

I believe that is the correct way to avoid the problem, since the model's __doc__ attribute will then be a Unicode object instead of a bytestring. Django can't really fix up a bytestring returned by __doc__ after the fact because the correct encoding is buried in the 'coding:' specification on the first line of the file containing the Model definition. The patch provided probably seemed to work because the system default locale encoding matched the file encoding, but you cannot count on that. I did verify that the recommended way of specifying the docstring works correctly on newforms-admin.