Javascript translations fail with non-BMP characters

[Ned Batchelder]

If a translated string includes a non-BMP character (above 0xFFFF), then javascript_catalog in views/i18n.py fails:

Traceback (most recent call last):
  File "/home/ned/.virtualenvs/edx-platform/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/home/ned/.virtualenvs/edx-platform/local/lib/python2.7/site-packages/django/views/i18n.py", line 264, in javascript_catalog
    csrc.append("catalog['%s'] = '%s';\n" % (javascript_quote(k), javascript_quote(v)))
  File "/home/ned/.virtualenvs/edx-platform/local/lib/python2.7/site-packages/django/utils/functional.py", line 176, in wrapper
    return func(*args, **kwargs)
  File "/home/ned/.virtualenvs/edx-platform/local/lib/python2.7/site-packages/django/utils/text.py", line 305, in javascript_quote
    return str(ustring_re.sub(fix, s))
UnicodeEncodeError: 'ascii' codec can't encode character u'\U0001d543' in position 31: ordinal not in range(128)

(this is running 1.4.8, but the javascript_quote code hasn't changed since then.)

It should be possible to fix javascript_quote to turn the character into a surrogate pair.

[Baptiste Mispelon <bmispelon@…>]

In 1c1dffca757b0b6acaf99d893d68847250ab4146:

Fixed #21725 -- Fixed JavaScript quoting encoding.

Thanks to nedbatchelder for the report.

[Ned Batchelder]

Is this really a full fix? Why does this function replace BMP characters with four-digit \uXXXX escapes, but allow non-BMP characters through unchanged? I would have thought that Javascript would need surrogate pairs.

[Honza Král]

I am getting consistent failures since this was merge in python 2 (not python 3) on mac:

======================================================================
FAIL: test_javascript_quote_unicode (utils_tests.test_text.TestUtilsText)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/honza/work/django/tests/utils_tests/test_text.py", line 162, in test_javascript_quote_unicode
    self.assertEqual(text.javascript_quote(input), output)
AssertionError: u"<script>alert(\\'Hello \\\\xff.\\n Wel\\ud835\\udd43come\\there\\r\\');<\\/scr [truncated]... != u"<script>alert(\\'Hello \\\\xff.\\n Wel\U0001d543come\\there\\r\\');<\\/script> [truncated]...
- <script>alert(\'Hello \\xff.\n Wel\ud835\udd43come\there\r\');<\/script>?                                   ^^^^^^^^^^^^
+ <script>alert(\'Hello \\xff.\n Wel\ud835\udd43come\there\r\');<\/script>?

[Baptiste Mispelon]

I did some digging and as it turns out, javascript_quote is not used anymore when doing javascript translation since a506b6981bc48caec30bca3de94d2ac3e6fc1660.

In fact, this function is undocumented, barely tested, and was only used internally for the javascript_catalogue view.

On top of that, it was also completely broken on Python2 if you ever passed it non-ascii input.

I think we should just delete it altogether.

As for the test failure, they pass on my machine both with Python 2 and 3 and our CI server is also in the green. There's a chance it might be related to a wide/narrow build of Python.

[Baptiste Mispelon]

Here's a pull request that adds a regression test to the javascript_catalog suite that makes sure that non-BMP characters in translation files are correctly handled.

It also deprecates javascript_quote altogether (it was undocumented and not used anymore after a506b6981bc48caec30bca3de94d2ac3e6fc1660).

Finally, it also skips the reported failing test on narrow python builds.

[Claude Paroz]

​https://github.com/django/django/pull/2339 I guess :-)

[Sasha Romijn]

PR 2339 looks fine to me, and tests run (although I tested on a non-wide python).

[Baptiste Mispelon <bmispelon@…>]

In 926e18d7d126fcf7f4b2d25ce4155423ac6e2f90:

Deprecated django.utils.text.javascript_quote.

Refs #21725.

[Baptiste Mispelon]

Thanks for the review.

[Claude Paroz <claude@…>]

In ac699cdc174a825e6b78c6f3c6e967bc961413c8:

Really hidden warnings in javascript_quote tests

Refs #21725.

[Tim Graham <timograham@…>]

In df3f3bbe2927b9bad80088c6adbf5e8c5ba778c9:

Removed utils.text.javascript_quote() per deprecation timeline; refs #21725.