Localized PostgreSQL error messages break loaddata

[gerard.henry@…]

Doing loaddata:

python manage.py loaddata -v 3 --traceback /tmp/mc1.json 
...
Traceback (most recent call last):
  File "/Users/me/python/virtualenv/django15x/lib/python2.7/site-packages/django/core/management/base.py", line 222, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/me/python/virtualenv/django15x/lib/python2.7/site-packages/django/core/management/base.py", line 255, in execute
    output = self.handle(*args, **options)
  File "/Users/me/python/virtualenv/django15x/lib/python2.7/site-packages/django/core/management/commands/loaddata.py", line 226, in handle
    e.args = ("Problem installing fixtures: %s" % e,)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 126: ordinal not in range(128)

changing line 226 in loaddata.py (replace %s by %r), i see the error:

IntegrityError: Problem installing fixtures: IntegrityError('insert or update on table "eventsmgmt_mycalendar" violates foreign key constraint "eventsmgmt_mycalendar_created_by_id_fkey"\nD\xc3\x89TAIL : Key (created_by_id)=(46) is not present in table "auth_user".\n',)

in the terminal, the locale is:

$localeLANG="fr_FR.UTF-8"
LC_COLLATE="fr_FR.UTF-8"
LC_CTYPE="fr_FR.UTF-8"
LC_MESSAGES="fr_FR.UTF-8"
LC_MONETARY="fr_FR.UTF-8"
LC_NUMERIC="fr_FR.UTF-8"
LC_TIME="fr_FR.UTF-8"
LC_ALL=

the workaround is to change the locale to en_US:

$ export LANG=en_US.UTF-8

and it works

[Claude Paroz]

Note that I can currently reproduce the error with fixtures_regress.tests.TestFixtures.test_loaddata_raises_error_when_fixture_has_invalid_foreign_key

[Claude Paroz]

Also note that this is a Python2-only issue.

Some possible solutions:

• using %r instead %s. Drawback: the resulting message is not quite the same:

  e.args = ("Problem installing fixtures: %s" % e,)
  
  (u"Problem installing fixtures: The row in table 'fixtures_regress_book' with primary key '1' has an invalid
  foreign key: fixtures_regress_book.author_id contains a value '3' that does not have a corresponding value in
  fixtures_regress_person.id.",)
  
  e.args = ("Problem installing fixtures: %r" % e,)
  
  (u'Problem installing fixtures: IntegrityError(u"The row in table \'fixtures_regress_book\' with primary key
  \'1\' has an invalid foreign key: fixtures_regress_book.author_id contains a value \'3\' that does not have a
  corresponding value in fixtures_regress_person.id.",)',)
  

  And we should still track this pattern all over Django code.

• using force_text in DatabaseErrorWrapper to convert message to unicode.
  Drawback: non-ascii unicode args for exceptions are not well supported, notably in unittest2 assertRaisesRegexp where str(exc_value) will raise a UnicodeEncodeError.

• using force_text(s, 'ascii', errors='ignore') in DatabaseErrorWrapper to strip out any non-ascii char from the exception message.
  Drawback: the error message will be altered.

Opinions or other solutions welcome.

[EmilStenstrom]

There are lots of other cases where printing error messages in ascii could lead to problems. I'm not sure if there are more bugs covering this, so I'll add a small description here:

I have a model who's unicode() method returns:

<MailComment: \n\nK\xc3\xb6r jobbet nu och det verkar bli fel...>

When loading my fixture I get: ValueError('Cannot assign [unicode representation of MailComment here]') which get propagated to fixture loading and makes that crash with:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 32: ordinal not in range(128)

Using %r instead of %s seems reasonable, as it doesn't remove any of the offending characters (they are often the reason for the exception in the first place), and it will likely work well with all sorts of environments.

[Claude Paroz]

#22882 reports a similar issue with the flush command.

[Tim Graham]

Can we add from __future__ import unicode_literals to dumpdata.py? This strategy seems to fixed the issue for flush.

[Claude Paroz]

#25719 was a duplicate.

[Claude Paroz]

Django master is now Python 3 only, therefore I'll close this ticket.