Problem following the tutorial when using non English characters

[gradha@…]

I'm following ​http://www.djangoproject.com/documentation/tutorial1/ and its second part. Everything goes OK. Just, to try things, I modify most of the examples of code. So when I modified the polls/models.py to create the Poll object, instead of using pub_date's tutorial provided description, I wrote:

pub_date = models.DateTimeField('Fecha de publicación')

To make the script a valid python script, I added at the top:

# -*- coding: latin1 -*-

So later, much much later, when I'm adding the administrative interface, I reach the point where I have added the "class Admin: pass" to Poll and web browsed my way to see the first poll. Here, the poll is displayed, and the date description shows a broken character (the accented o): "Fecha de publicaci�n:" (not sure if that will get through, it shows a question mark inside a rotated square).

Well, I guess maybe Django wants unicode characters or stuff. So I modified my Poll class to read (note the unicode string):

pub_date = models.DateTimeField(u'Fecha de publicación')

Reloading the admin page I get:

UnicodeDecodeError at /admin/polls/poll/1/
'ascii' codec can't decode byte 0xc3 in position 75: ordinal not in range(128)

...along with a hell long of a callstack and environment dump. Since I'm used to these kind of things, I tried the following change:

pub_date = models.DateTimeField(u'Fecha de publicación'.encode("utf8"))

...which worked and didn't crash, but strikes me as unusually cumbersome to do. AFAICS the model doesn't properly treat unicode strings and the reason I get this OK with the utf8 encoding is just because that's what the output page charset is, and the result is processed by the browser. I guess if I switched that to latin1 now it would break visually as well, right? Is Django unable to handle internationalised stuff in the model?

If this is the case, I would add a note on the kind of strings you can use for the model.

[asmodai@…]

I cannot reproduce this.

I used:

# -*- coding: latin1 -*-

with a verbose name of:

models.CharField('áóéíàèìò', maxlength = 64)

and in my admin I see:

áóéíàèìò:

Did you check your browser to see if it selected/auto-detected the right page encoding? It should be UTF-8.

Note that I am using the latest SVN version and you didn't specify yours though.

[Adrian Holovaty]

Looks like this isn't a bug. (See previous comment.)

[Ramiro Morales]

I'm reopening this ticket because I'm experiencing the same issue and to post some additional information:

SVN revision 3065

first line of models.py has:

  # -*- coding: iso-8859-1 -*-

a model:

  class Project(models.Model):
    ...
    description = models.CharField('descripcion', blank=True, maxlength=255, db_column='DESCRIPTION')
    ...
    class Admin:
        fields = ((None, { 'fields': ('title', 'description', 'client', 'start_date', 'leader', 'status')} ), )
        list_display=('title', 'client', 'status', 'leader')

Django conf/global_settings.py has

  DEFAULT_CHARSET = 'utf-8'

Project settings.py has

  DEBUG = True
  LANGUAGE_CODE = 'es-ar'

$ file models.py
models.py: ASCII English text

If I edit the field verbose name to add a non-ASCII char

    description = models.CharField('descripción', blank=True, maxlength=255, db_column='DESCRIPTION')

$ file models.py
models.py: ISO-8859 English text

And then whe trying to access the edit page for that model by selecting it from the change list page of the admin app, I get a 'Descripci�n:' field label.

I'm using the bundled Django development Web server.

Server-side info:
Linux, python 2.3.5

Client-side info:

Mozilla Firefox 1.5.0.4 / Win32 / Windows 2000 + SP4 English

View -> Character Encoding -> UTF-8 is checked, if I change it to Western (ISO-8859-1) the
problem goes away.

When I change the verbose name to include the u prefix

    description = models.CharField(u'descripción', blank=True, maxlength=255, db_column='DESCRIPTION')

I also get the exception + back trace debug page:

Request Method:  	GET
Request URL: 	http://<my server DNS name>:8000/admin/intro/project/81/
Exception Type: 	UnicodeDecodeError
Exception Value: 	'ascii' codec can't decode byte 0xc3 in position 222: ordinal not in range(128)
Exception Location: 	/usr/lib/python2.3/site-packages/django/template/__init__.py in render, line 686

Traceback (most recent call last):
File "/usr/lib/python2.3/site-packages/django/template/__init__.py" in render_node
  701. result = node.render(context)
File "/usr/lib/python2.3/site-packages/django/template/defaulttags.py" in render
  113. nodelist.append(node.render(context))
File "/usr/lib/python2.3/site-packages/django/template/defaulttags.py" in render
  115. return nodelist.render(context)
File "/usr/lib/python2.3/site-packages/django/template/__init__.py" in render
  686. return ''.join(bits)

  UnicodeDecodeError at /admin/intro/project/81/
  'ascii' codec can't decode byte 0xc3 in position 222: ordinal not in range(128)

Why is the text getting decoded with the ascii codec?.

Feel free to ask for further info bacause I have the environment at hand.

Regards,

--

Ramiro

[Ramiro Morales]

More info:

"Request information - META" section of the backtrace:

CONTENT_LENGTH
CONTENT_TYPE	'text/plain'
DJANGO_SETTINGS_MODULE	'tang.settings'
GATEWAY_INTERFACE	'CGI/1.1'
HOME	'/home/ramiro'
HTTP_ACCEPT	'text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'
HTTP_ACCEPT_CHARSET	'ISO-8859-1,utf-8;q=0.7,*;q=0.7'
HTTP_ACCEPT_ENCODING	'gzip,deflate'
HTTP_ACCEPT_LANGUAGE	'en-us,en;q=0.7,es-ar;q=0.3'
HTTP_CONNECTION	'keep-alive'
HTTP_COOKIE	'sessionid=145006e2d5e41bb0ae6e740c5c3e4811; SITESERVER=ID=f9aa77d7c40371371daebd95787b0042'
HTTP_HOST	'<my Django server DNS name>:8000'
HTTP_IF_MODIFIED_SINCE	'Fri, 02 Jun 2006 18:46:50 GMT'
HTTP_IF_NONE_MATCH	'1a18096f877784f328f0045a20f26e9a'
HTTP_KEEP_ALIVE	'300'
HTTP_REFERER	'​http://<my Django server DNS name>:8000/admin/intro/project/'
HTTP_USER_AGENT	'Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.0.4) Gecko/20060508 Firefox/1.5.0.4'
LANG	'en_US'
LANGUAGE	'en_AR:en_US:en_GB:en'
LOGNAME	'ramiro'
PATH_INFO	'/admin/intro/project/81/'
PWD	'/home/ramiro/src/tang'
QUERYSTRING
REMOTE_ADDR	'<my workstation IP address>'
REMOTE_HOST	'<my workstation DNS name>'
REQUEST_METHOD	'GET'
RUN_MAIN	'true'
SCRIPT_NAME
SERVER_NAME	'<my Django server DNS name>'
SERVER_PORT	'8000'
SERVER_PROTOCOL	'HTTP/1.1'
SERVER_SOFTWARE	'WSGIServer/0.1 Python/2.3.5'
TZ	'America/Cordoba'
USER	'ramiro'
_	'./manage.py'
wsgi.errors	<open file '<stderr>', mode 'w' at 0xb7d890a0>
wsgi.file_wrapper	<class django.core.servers.basehttp.FileWrapper at 0xb7a9e7dc>
wsgi.input	<socket._fileobject object at 0xb7031c34>
wsgi.multiprocess	False
wsgi.multithread	True
wsgi.run_once	False
wsgi.url_scheme	'http'
wsgi.version	(1, 0)

[Ramiro Morales]

I'm seeing now that a solution to this problem was posted in one of the lasts comments of ticket #170 (I've tested it and it works).

So I'm closing this ticket again. Perhaps I will open another targeted at the documentation component because it seems a somewhat common pitfall for non English users (see #1715, ​http://groups.google.com/group/django-users/browse_thread/thread/d536902362a36b9c/0d5b920adc628005#0d5b920adc628005). If the recommended solution is in fact the right one then maybe a note could be added to the models docs stating that for non ASCII languages it is beter to encode the models.py file in UTF-8 and flag it so by using magic comments as per ​PEP 0263

Regards,