UnicodeDecodeError in blocktrans

[Cedric Rossi <cedric@…>]

In [6675] the following:

{% blocktrans with title as the_title %}{{ the_title }}{% endblocktrans %}

will fail with the following traceback, if title is a string containing non ascii characters:

Traceback (most recent call last):
File "/Users/cedric/src/koffeeweb/django/template/__init__.py" in render_node
  814. result = node.render(context)
File "/Users/cedric/src/koffeeweb/django/templatetags/i18n.py" in render
  77. result = re.sub('%(?!\()', '%%', result) % context

  UnicodeDecodeError at /fail/
  'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

On the other hand, this:

{% blocktrans with title|escape as the_title %}{{ the_title }}{% endblocktrans %}

works as expected.

I couldn't reproduce the problem in [6636], so this has probably been introduced recently (auto-escaping stuff?)

Please see the attached project, which defines two urls: /fail/ and /pass/ showing the problem

[Malcolm Tredinnick]

What is the contents of "title" at this point when you trigger the error. The data involved is as important as the code for a case like this. Can you print type(title) and repr(title) in your view just before displaying the template please?

[Cedric Rossi <cedric@…>]

Sorry, I could have been more precise in the description, but I believed attaching a working minimal project would make it easier for you to look at.

Here's the content of the view:

# coding: utf-8

from django.shortcuts import render_to_response

def test_pass(request):
    return render_to_response('test-pass.django.html',
                              {'title': "é"})

def test_fail(request):
    return render_to_response('test-fail.django.html',
                              {'title': "é"})

In case Trac mangles it, that's a &eaccute; The file is saved as utf-8.

print type(title) 
> <type 'str'>
print repr(title)
> '\xc3\xa9'

replacing "é" by u"é" fixes the problem, as expected, so my example was badly choosen, sorry about that.

Now the problem is, in my real project, title is a models.CharField(); DEFAULT_CHARSET is 'utf-8'; my table (MySQL) is properly defined as utf-8, and still, I've got (from a real example):

print type(title) 
> <type 'str'>
print repr(title)
> '\xc3\x8ele de Batz'

Is that expected?

I've tested again in [6636], and print and repr give the same results; still, inside the blocktrans, the problem originally reported remains in [6675] and not in [6636].

[Malcolm Tredinnick]

(In [6682]) Fixed a few problems with variable resolving inside of blocktrans tags. A couple of these were exposed by the auto-escaping changes, but I suspect the other one has been hiding in plain sight for a while.

Fixed #5952, #5953