Exception with utf8 message leads to DjangoUnicodeDecodeError

[bjourne]

The psycopg2 driver raises exceptions with i18n:ized messages. Django is not able to diplay them and will raise a DjangoUnicodeDecodeError. Example:

File "/var/lib/python-support/python2.6/django/views/debug.py", line 39, in technical_500_response
    html = reporter.get_traceback_html()
  File "/var/lib/python-support/python2.6/django/views/debug.py", line 97, in get_traceback_html
    'exception_value': smart_unicode(self.exc_value, errors='replace'),
  File "/var/lib/python-support/python2.6/django/utils/encoding.py", line 35, in smart_unicode
    return force_unicode(s, encoding, strings_only, errors)
  File "/var/lib/python-support/python2.6/django/utils/encoding.py", line 70, in force_unicode
    raise DjangoUnicodeDecodeError(s, *e.args)
DjangoUnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 7: ordinal not in range(128). You passed in DataError('FEL:  v\xc3\xa4rdet f\xc3\xb6r l\xc3\xa5ngt f\xc3\xb6r typen character varying(511)\n',) (<class 'psycopg2.DataError'>)

[Walter Doekes]

Hmyes.. I've tested this on my debian/lenny install and can confirm this.

psycopg2 uses the client_encoding (as set through self.connection.set_client_encoding('UTF8') in django/db/backends/postgresql_psycopg2/base.py) for its error messages.

Thus, to get this fixed, all DatabaseErrors raised by psycopg2 should have their .message.decode('utf-8')'d before they're exposed to the world.

[Russell Keith-Magee]

I can't reproduce this problem. I don't deny that the problem exists (since at least one person has been able to reproduce it), but the stack trace and comment doesn't give enough detail to explain how to reproduce the problem. If you can provide instructions on how to reproduce the problem, please reopen.

[Karen Tracey]

I think I know what this is. It's a recurrence of the problem fixed by r8588 that happens when running under Python 2.6. Prior to 2.6 (or with 2.5 at any rate), Exceptions did not have unicode methods, with 2.6 they do. The r8588 special-casing for Exceptions was made only in the path that applies to objects without unicode methods, thus it is no longer working under 2.6. (Having a unicode method under 2.6 is also why the original traceback is now completely absent in TemplateSyntaxErrors that wrap other exceptiosn -- the code to include the original traceback in the exception value is in an __str__ method, and 2.6 no longer calls that. But that's a different problem than this one, and there's also another ticket open for that.)

[Karen Tracey]

Not sure why you put this in the ORM component? It's a general problem when running under 2.6 with any exception that has non-ASCII data and does not provide a working unicode method. Strictly speaking I think the exceptions are broken, but Malcolm eventually agreed in #6353 that having Django's debug info code generate a 2nd exception because the 1st exception is broken is something that should be avoided. The fix probably goes into force_unicode, though I haven't had a chance to look at it closely yet.

[Walter Doekes]

Hm. I cannot reproduce my reproduction. Must've been smoking crack at the time. force_unicode seems to do what it should correctly:

Set lc_messages and restart postgresql:

$ grep lc_messages /etc/postgresql/8.3/main/postgresql.conf 
lc_messages = 'sv_SE.UTF-8'                     # locale for system error message

Start up a django shell:

$ ./manage shell
Python 2.5.2 (r252:60911, Jan  4 2009, 21:59:32) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django import VERSION
>>> VERSION
(1, 1, 1, 'final', 0)
>>> from django.db import connection
>>> c = connection.cursor()
>>> c.execute('SELECT 123 FROM;')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/opt/django11/django/db/backends/util.py", line 19, in execute
    return self.cursor.execute(sql, params)
ProgrammingError: FEL:  syntaxfel vid eller nära ";"
LINE 1: SELECT 123 FROM;
                       ^

>>> try: c.execute('SELECT 123 FROM;')
... except Exception, e: pass
...
>>> unicode(e)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 27: ordinal not in range(128)
>>> from django.utils.encoding import force_unicode
>>> force_unicode(e)
u'FEL:  syntaxfel vid eller n\xe4ra ";"\nLINE 1: SELECT 123 FROM;\n                       ^\n'

I do not deny that there is the problem you've analyzed. But I retract my reproduction.

[Karen Tracey]

The shell session you show is using Python 2.5. The original traceback is using Python 2.6. (I can't tell what you were using in the first comment where you could reproduce it.) The problem only occurs under 2.6 and higher; it is due to a change in Python that the fix made in r8588 is not kicking in. Specifically, exceptions in 2.6+ have unicode methods, where they did not in 2.5, and the r8588 fix was made in a path that deals only with objects that do not have unicode methods.

[Karen Tracey]

(In [12621]) Fixed #12302: Modified force_unicode to avoid raising unicode errors when
handed exceptions with non-ASCII bytestring data and no working unicode method
under Python 2.6 and higher.

[Karen Tracey]

(In [12623]) [1.1.X] Fixed #12302: Modified force_unicode to avoid raising unicode errors when
handed exceptions with non-ASCII bytestring data and no working unicode method
under Python 2.6 and higher.

r12621 and r12622 from trunk.

[Jacob]

Milestone 1.2 deleted

[GitHub <noreply@…>]

In 2d899ce:

Refs #23919 -- Removed a Python 2 code path in force_text().

Reverted the obsolete fix and tests for refs #12302.