Opened 18 years ago

Closed 18 years ago

Last modified 18 years ago

#3115 closed defect (fixed)

Postgresql backend should convert Unicode input to bytestrings

Reported by: Manuel Saelices <msaelices@…> Owned by: Adrian Holovaty
Component: Database layer (models, ORM) Version: dev
Severity: major Keywords:
Cc: Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

ORM fails in PostgreSQL when you filter with unicode strings. For example:

>>> from django.contrib.auth.models import User
>>> User.objects.filter(username=u'admin')
...
ProgrammingError: ERROR:  column "admin" does not exist

SELECT "auth_user"."id","auth_user"."username","auth_user"."first_name","auth_user"."last_name","auth_user"."email","auth_user"."password","auth_user"."is_staff","auth_user"."is_active","auth_user"."is_superuser","auth_user"."last_login","auth_user"."date_joined" FROM "auth_user" WHERE ("auth_user"."username" = admin) ORDER BY "auth_user"."username" ASC

The problem is on psycopg library, here are an example:

>>> from django.db import connection
>>> cursor = connection.cursor()
>>> cursor.execute('SELECT * from auth_user WHERE username = %s', [u'admin'])
...
ProgrammingError: ERROR:  column "admin" does not exist

SELECT * from auth_user WHERE username = admin

Ok, it's true... it's not a django error, but for example, in newforms all is unicode... ¿what we do? ¿Use psycopg2? ¿convert all unicodes to ascii in lookups? ¿convert all params on postgresql?. Call is made on django/db/models/query.py, on this sentences:

     cursor = connection.cursor()
     select, sql, params = self._get_sql_clause()
     cursor.execute("SELECT " + (self._distinct and "DISTINCT " or "") + ",".join(select) + sql, params)

Change History (5)

comment:1 by Adrian Holovaty, 18 years ago

Yes, we ought to change the postgresql Django database backend to check whether each string is Unicode before adding it to the statement.

This also ties into another suggestion people have had, which is to have a DATABASE_CHARSET setting. This would describe which character set the database uses.

comment:2 by Adrian Holovaty, 18 years ago

Summary: ORM problems with unicode in postgresql backendPostgresql backend should convert Unicode input to bytestrings

comment:3 by Adrian Holovaty, 18 years ago

See also #952.

comment:4 by Adrian Holovaty, 18 years ago

Resolution: fixed
Status: newclosed

(In [4244]) Fixed #3115 -- Changed postgresql backend to convert all Unicode strings to bytestrings according to DEFAULT_CHARSET. This is necessary because psycopg1 does not apply database quoting to Unicode strings

comment:5 by (none), 18 years ago

milestone: Version 1.0

Milestone Version 1.0 deleted

Note: See TracTickets for help on using tickets.
Back to Top