sqlite and unicode problems

[hugo <gb@…>]

sqlite3 and pysqlite2 allways returns unicode stuff for values. This poses problems in django. To fix, just use a cursor factory and a row factory to patch out all unicode elements to utf8 bytestrings. This is the patch:

Index: core/db/backends/sqlite3.py
===================================================================
--- core/db/backends/sqlite3.py (revision 341)
+++ core/db/backends/sqlite3.py (working copy)
@@ -14,6 +14,13 @@
 Database.register_converter("date", typecasts.typecast_date)
 Database.register_converter("datetime", typecasts.typecast_timestamp)
 
+# a row factory that removes unicode #########################################
+def utf8row(cursor, row):
+        def utf8(s):
+                if type(s) == unicode: return s.encode('utf-8')
+                else: return s
+        return [utf8(el) for el in row]
+
 # Database wrapper ############################################################
 
 class DatabaseWrapper:
@@ -30,7 +37,7 @@
             self.connection.create_function("django_date_trunc", 2, _sqlite_date_trunc)
         if DEBUG:
             return base.CursorDebugWrapper(FormatStylePlaceholderCursor(self.connection), self)
-        return FormatStylePlaceholderCursor(self.connection)
+        return self.connection.cursor(factory=FormatStylePlaceholderCursor)
 
     def commit(self):
         self.connection.commit()
@@ -50,7 +57,11 @@
     This fixes it -- but note that if you want to use a literal "%s" in a query,
     you'll need to use "%%s" (which I belive is true of other wrappers as well).
     """
-    
+
+    def __init__(self, *args, **kw):
+        Database.Cursor.__init__(self, *args, **kw)
+        self.row_factory = utf8row
+
     def execute(self, query, params=[]):
         query = self.convert_query(query, len(params))
         return Database.Cursor.execute(self, query, params)

[yeiazel <yeiazel@…>]

This patch works perfectly here ! Thanks

(problem was in the admin area, when modifying an object with unicode characters, and when displaying any titles list (like in the comments admin page) when a title had unicode char(s), with french characters (the usual é, è, ê, à, etc.))

I changed the severity to "major" because it really blocks things, and makes it impossible to use unless accessing to the sqlite database with the command-line tool.

[Jacob]

(In [383]) Fixed #227 -- the sqlite backend now correctly typecasts unicode objects to bytestrings (thanks hugo)