MySQL Collations/Charsets and Engines

[Armin Ronacher]

Django should really have a DATABASE_CHARSET / DATABASE_COLLATION setting in the settings.py as well as a setting for the mysql database engine. It's stunning that unittests can enforce a database collation but tables not. Also an argument in the Meta section of models that allows to define the collation, charset and engine would be an important thing to have.

I know that there are workarounds for those limitations but they have all disadvantages.

[Dmitry Gorbunow]

-0 to this. I think that Unicode should be used everythere by default.

[Armin Ronacher]

Unicode yes. But collations do not affect unicode but the way it deals with unicode data (german sorting, french sorting, greek sorting etc.). And other encodings in the database are required if you deal with old database data. And currently django creates latin1 tables which is the database default.

[Dmitry Gorbunow]

Anyways this is a database-specific thing.

[Armin Ronacher]

So is the Oracle db_tablespace parameter which is exposed to django. So what?

[Malcolm Tredinnick]

Before the comments really get out of hand...

This is probably worth doing in some limited form. I'd guess using the existing DATABASE_OPTIONS setting is probably the most logical way, similar to how MySQL's engine setting is supported now. It's not worth adding it as an option to each model, since it's unlikely to vary between models (and for those cases where it does, the initial SQL support can be used). Two extra settings just for MySQL support is a little excessive, too.

I'd want to rejig DATABASE_OPTIONS a little bit first, though, so that, for example, we can differentiate between creation time options (engine, collation, etc) and general connection options. That will require some design work, but that's my thinking at the moment.

[Armin Ronacher]

@mtredinnick: Sounds like a good idea.

[Armin Ronacher]

Damn. That was not a very friendly written ticket. :)

[Armin Ronacher]

In hindsight that should be implemented database backend independent and unicode does not appear to specify names for collations, so that should probably not be fixed in the way I originally recommended.

[Shai Berger]

For the record:

All core backends (except Oracle, it seems) can set collation per column; collation names are not standard (and SQLite has only 3 built-in collations, none of them reasonable for any native language). Except (I think) for Oracle, they can also set collation for a specific query (the collation can be used in ordering comparisons or in ORDER BY). This means that any reasonable collation support in core would beget backend-specific usages, which I wouldn't like.

However, I think core should make it possible to add collation support via external libraries. I would like to see the API for such extension.

Of all core backends, the only one which can set the character encoding at a level lower than the whole database is MySQL. Since Django does not usually create databases (except for test databases), I think this feature should not be added (except, with the API mentioned above, in a MySQL-specific library outside core).

[Clifford Gama]

Related to #18392