Add support for translation backends other than gettext

[Marinho Brandão]

Using gettext has been a problem on many projects. It's weird to translate and is limited to (static) potfiles (nothing convince me that Pootle Server and Rosetta are great ideas, actually they are just quick fixes for a bad implemented tool).

So, I'd like to suggest you to make a contrib app to have all i18n functions.

Not only the existing functions but provide backend system for alternative ways to make translations, to replace gettext for database stored message strings, or dynamic/smart translations.

Would be helpful also if the same application has a way to translate database objects, like django-multilingual (1) and polyglot (2) do.

I can work on it if the idea get approved.

• (1) ​http://code.google.com/p/django-multilingual/
• (2) ​http://reddes.bvsalud.org/projects/clinical-trials/browser/trunk/opentrials/polyglot

[Jannis Leidel]

With all due respect, calling gettext problematic for non-static translation seems a bit off since it clearly wasn't made for dynamic translations. In any case, this requires thorough discussion on the mailing list before any approval.

[Marinho Brandão]

Of course... so, because it is not for dynamic translations, it's a problem to do it if the only way that the framework offers to do is using gettext.

This is why I believe a backend system would help a lot, because who want to work with non-static translations (and at the level of model objects) can avoid gettext and use database or cache to do it.

I know that gettext is mature and fast, yet a limited sollution.

[Jannis Leidel]

Replying to marinho:

Of course... so, because it is not for dynamic translations, it's a problem to do it if the only way that the framework offers to do is using gettext.

You're basically trying to justify a new feature by saying the existing feature doesn't cover your use case, which is a moot point in my opinion.

This is why I believe a backend system would help a lot, because who want to work with non-static translations (and at the level of model objects) can avoid gettext and use database or cache to do it.

Right, I don't quite understand what you mean by backend system though. An explanation (on the mailing list) might help to clarify what you mean. The big issue I see here (at least from what I understand) is that content translation and translation of UI elements are completely different stories because the way those content types differ in structure and storage. Adding a backend system implies having a common API, but I don't see any. Can you elaborate?

I know that gettext is mature and fast, yet a limited sollution.

It's not limited but really just not supposed to be used like you use.

[Marinho Brandão]

Replying to jezdez:

I didn't revise the text below, please understand I'm doing it as faster as I can (today I'm working).

Ok, I tried to be clear on the mail list, but I'm going to right a fast story with more details below:

You know, most of i18n functions are in:

1. utils/translation
2. core/management/commands/compilemessages.py
3. core/management/commands/makemessages.py
4. views/i18n.py
5. conf/locale

What I'm saying is about to change the points 1, 2, 3 and 4.

Keep in my the following structure:

• contrib/i18n
• contrib/i18n/init.py
• contrib/i18n/models.py <--- here we have the model classes for DB storage
• contrib/i18n/views.py <--- here we have the current "views/i18n.py"
• contrib/i18n/backends/init.py
• contrib/i18n/backends/base.py <--- here we have the backend base class
• contrib/i18n/backends/potfiles.py <--- here we have the backend class for gettext (with most of current "utils/translation" code)
• contrib/i18n/backends/db.py <--- here we have the backend class for database (using the model class for DB storage, etc)
• contrib/i18n/management/init.py
• contrib/i18n/management/commands/init.py
• contrib/i18n/management/commands/compilemessages.py <--- we move the current to here
• contrib/i18n/management/commands/makemessages.py <--- we move the current to here

Now a prototype of "contrib/i18n/models.py":

class MessageString(models.Model):
    class Meta:
        unique_together = (
            ('hash','language'),
        )
    hash = models.CharField()
    language = models.CharField()
    message_id = models.TextField() # if necessary
    message_str = models.TextField()

Now a prototype of "contrib/i18n/backends/base.py":

class BaseBackend(object):
    def gettext(self, msg):
        raise NotImplemented

    def ugettext(self, msg):
        raise NotImplemented

    def gettext_noop(self, msg):
        raise NotImplemented

    def get_language(self):
        return self._language # Of course, different

    def makemessages(self):
        raise NotImplemented

    def compilemessages(self):
        raise NotImplemented

    # etc...

Now a prototype of "contrib/i18n/backends/potfiles.py":

class GettextBackend(BaseBackend):
    potfile = 'django.po' # etc...

    def gettext(self, msg):
        return do_translate(message, 'gettext') # Like it does nowadays

    def ugettext(self, msg):
        return do_translate(message, 'ugettext') # Like it does nowadays

    def gettext_noop(self, msg):
        return message

    def makemessages(self):
        # do the same that makemessages.py actually does

    def compilemessages(self):
        # do the same that compilemessages.py actually does

    # etc...

Now a prototype of "contrib/i18n/backends/db.py":

class DatabaseBackend(BaseBackend):
    def get_hash(self, msg):
        return hashlib.sha1(msg).hexdigest()

    def gettext(self, msg):
        try:
            obj = MessageString.objects.get(
                    language = self.get_language(),
                    hash = self.get_hash(msg),
                    )
            return obj.message_str
        except MessageString.DoesNotExist:
            return msg

    def ugettext(self, msg):
        return do_translate(message, 'ugettext') # Like it does nowadays

    def gettext_noop(self, msg):
        return message

    def makemessages(self):
        # do the same that makemessages.py actually does, but stores using
        # MessageString instead of using the gettext tools

    def compilemessages(self):
        # do nothing, probably

    # etc...

Now the setting to set the current backend (settings.py):

I18N_BACKEND = 'django.contrib.i18n.backends.DatabaseBackend'

So, the template tags and ugettext functions must be changed to use the current
backend instead of just use gettext.

[Marinho Brandão]

Just fixing: ignore the method "ugettext" from "DatabaseBackend". It should be like the method "gettext" from the same class.

[Aymeric Augustin]

The original report proposes three main changes:

• 1) a django.contrib.i18n application — which is basically a large refactoring
• 2) support for translation backends (other file formats / string databases)
• 3) support for dynamic translations (translation of database content)

Refactorings are complicated, because they introduce backwards incompatibilities that only disappear after three releases — that's the length of our deprecation path. They're also very hard to review. We tend to accept them only when they're a pre-requisite to add interesting features. Anyway, the amount of effort needed for the refactoring is dwarfed by the the amount needed for the features, so it doesn't really matter at this step.

Features (2) and (3) are rather large — as a wild guess, defining, developing, testing and documenting them for Django would take a few hundred hours.

Reading ​the thread you started on this topic, I understand that your proposal is mostly about (2). As you say in your last message, what we need now is a more detailed proposal, explaining the consequences of the change:

• which backends would you include initially?
• how much would this affect performance?
• how would this affect distribution of translations for Django itself, and for pluggable third-party applications?
• etc.

A proof of concept in a fork on GitHub would be interesting too.

To sum up:

• I agree that pluggable translation backends and dynamic translations are interesting ideas. They're separate ideas, so let's limit this ticket to translation backends. I'm accepting the ticket with this scope.
• Please don't focus too much on refactoring — it adds more noise than value. Deal with the hard part (APIs, proof of concept) first. Try to be as much backwards compatible as possible. Then, if necessary, we can decide to refactor things.

[Aymeric Augustin]

#16960 is a duplicate that suggests introducing an XMB/XTB backend.

[Jonas Obrist]

I'm not sure point 3 in comment:7 should really be part of this ticket. Dynamic translations are a completely different thing than static translations. Wouldn't it make more sense to move the discussion etc about dynamic translations in Django to another ticket and keep this one purely for supporting additional backends for static translations?

[Jonas Obrist]

Ticket for dynamic translations (aka: model translations #6460)

[Ramiro Morales]

#19480 is a proposal related to this ticket and has a pull request at ​https://github.com/django/django/pull/590

[rtnpro]

Is any one working on this feature request?

[Tim Graham]

The PR above doesn't merge cleanly and lacks documentation.