[unicode] UnicodeDecodeError with on postgresql 7.4 using psycopg1 backend on python 2.3

[Øyvind Saltvik <oyvind@…>]

Traceback (most recent call last):
File "/var/www/vhosts/3kanten.it/httpdocs/trunk/django/core/handlers/base.py" in get_response
  77. response = callback(request, *callback_args, **callback_kwargs)
File "/var/www/vhosts/3kanten.it/httpdocs/trekanten/kontakt/views.py" in contact
  50. return object_list(request, **kwargs)
File "/var/www/vhosts/3kanten.it/httpdocs/trunk/django/views/generic/list_detail.py" in object_list
  75. }, context_processors)
File "/var/www/vhosts/3kanten.it/httpdocs/trunk/django/template/context.py" in __init__
  100. self.update(processor(request))
File "/var/www/vhosts/3kanten.it/httpdocs/trekanten/meny/context_processors.py" in meny
  9. menyliste = list(Side.objects.filter(i_meny=True)) + list(Element.objects.all())
File "/var/www/vhosts/3kanten.it/httpdocs/trunk/django/db/models/query.py" in __iter__
  109. return iter(self._get_data())
File "/var/www/vhosts/3kanten.it/httpdocs/trunk/django/db/models/query.py" in _get_data
  471. self._result_cache = list(self.iterator())
File "/var/www/vhosts/3kanten.it/httpdocs/trunk/django/db/models/query.py" in iterator
  188. rows = cursor.fetchmany(GET_ITERATOR_CHUNK_SIZE)
File "/var/www/vhosts/3kanten.it/httpdocs/trunk/django/db/backends/postgresql/base.py" in typecast_string
  273. return smart_unicode(s, client_encoding)
File "/var/www/vhosts/3kanten.it/httpdocs/trunk/django/utils/encoding.py" in smart_unicode
  25. return force_unicode(s, encoding, strings_only, errors)
File "/var/www/vhosts/3kanten.it/httpdocs/trunk/django/utils/encoding.py" in force_unicode
  42. s = unicode(s, encoding, errors)

  UnicodeDecodeError at /kontakt/
  'ascii' codec can't decode byte 0xc3 in position 134: ordinal not in range(128)

the variable encoding is ascii on line 42

[Øyvind Saltvik <oyvind@…>]

Switching to psycopg2 solved the problem.

[Malcolm Tredinnick]

So for some reason your default client encoding is ASCII (possibly configured to be non-international friendly in postgresql.conf). You, or any other user, may not be in control of that, so we have to work around. It's going to be annoying to have to try and change this value, though. We don't know what is available on the server-side and things will go badly wrong (Django will crash, most likely) if we get it wrong.

I don't have PostgreSQL 7.4 on any machine at the moment. Can you add a line just before line 97 (at the same indentation level that as line 97) that says

cursor.execute("SET CLIENT_ENCODING to 'UNICODE'")

So you will have a "set client encoding" followed by "show client encoding". I want to see if the "set" takes effect. If so, we can try to set the encoding to UTF-8 if the client reports ASCII.

(I may have the "UNICODE" string wrong. It isn't listed as an explicit option ​in table 20.2, here, but it looks like it should be supported. Try a few alternative strings if that doesn't work.)

[Øyvind Saltvik <oyvind@…>]

that solved it

[Malcolm Tredinnick]

(In [5535]) Fixed #4664 -- Forced the client character set encoding to UTF-8 for PostgreSQL
(via the psycopg backend). The previous version was causing problems on some
setups, particularly PostgreSQL 7.x. Current code should work with 7.x and 8.x,
no matter what the default client encoding is.