UnicodeDecodeError when handling uncaught exception with UTF-8 in URL

[André Cruz]

When DEBUG==False, there is UTF-8 in URL, and an uncaught exception occurs:

Traceback (most recent call last):
  File "/Users/andre/work/penv/discosite/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 187, in __call__
    response = self.get_response(request)
  File "/Users/andre/work/penv/discosite/lib/python2.7/site-packages/django/core/handlers/base.py", line 199, in get_response
    response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
  File "/Users/andre/work/penv/discosite/lib/python2.7/site-packages/django/core/handlers/base.py", line 231, in handle_uncaught_exception
    'request': request
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 1175, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 1268, in _log
    self.handle(record)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 1278, in handle
    self.callHandlers(record)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 1318, in callHandlers
    hdlr.handle(record)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/logging/__init__.py", line 749, in handle
    self.emit(record)
  File "/Users/andre/work/penv/discosite/lib/python2.7/site-packages/django/utils/log.py", line 124, in emit
    message = "%s\n\nRequest repr(): %s" % (self.format(record), request_repr)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 30: ordinal not in range(128)

Example request:

$ curl "http://localhost:8000/%5BSC%5D%20TIC%207%C2%BAC"

This did not happen < 1.7.

[Tim Graham]

I couldn't reproduce this using the code from the tutorial:

$ curl "http://localhost:8000/%5BSC%5D%20TIC%207%C2%BAC"
<h1>Not Found</h1><p>The requested URL /[SC] TIC 7ºC was not found on this server.</p>

Could you provide more details? Ideally, a test case for Django's test suite if possible.

[André Cruz]

As I mentioned, this happens when DEBUG=False, an exception is thrown while servicing the request, and there is UTF-8 in the URL. It seems you only reproduced the "UTF-8 in the URL part"...

You can put the following url pattern:

urlpatterns = patterns('',
    url(r'^(*)$', 'utils.cross_domain.cross_domain_xml'),
)

and this will cause an exception. Then self.format(record) will end up: (notice it's unicode)

u'Internal Server Error: /[SC] TIC 7\xbaC\nTraceback (most recent call last):\n  File "/Users/andre/work/penv/discosite/lib/python2.7/site-packages/django/core/handlers/base.py", line 87, in get_response\n    response = middleware_method(request)\n  File "/Users/andre/work/penv/discosite/lib/python2.7/site-packages/django/middleware/common.py", line 72, in process_request\n    if (not urlresolvers.is_valid_path(request.path_info, urlconf) and\n  File "/Users/andre/work/penv/discosite/lib/python2.7/site-packages/django/core/urlresolvers.py", line 614, in is_valid_path\n    resolve(path, urlconf)\n  File "/Users/andre/work/penv/discosite/lib/python2.7/site-packages/django/core/urlresolvers.py", line 489, in resolve\n    return get_resolver(urlconf).resolve(path)\n  File "/Users/andre/work/penv/discosite/lib/python2.7/site-packages/django/core/urlresolvers.py", line 340, in resolve\n    sub_match = pattern.resolve(new_path)\n  File "/Users/andre/work/penv/discosite/lib/python2.7/site-packages/django/core/urlresolvers.py", line 211, in resolve\n    match = self.regex.search(path)\n  File "/Users/andre/work/penv/discosite/lib/python2.7/site-packages/django/core/urlresolvers.py", line 179, in regex\n    (regex, six.text_type(e)))\nImproperlyConfigured: "^(*)$" is not a valid regular expression: nothing to repeat'

[Tim Graham]

Adding from __future__ import unicode_literals to django/utils/log.py seems to fix the issue. We also need a regression test and a mention in the 1.7.1 release notes.

[Tim Graham]

Bisected to ad6fcdb8d24d84a130f847f45b51d7e892ccca08.

​PR

[Tim Graham <timograham@…>]

In 9dff5ce7c70780f4ee91567d98dba227093b49e2:

Fixed #23593 -- Fixed crash in AdminEmailHandler with non-ASCII characters in request.

Thanks edevil for the report and Simon Charette for review.

[Tim Graham <timograham@…>]

In 96f022203a552fef19403c01b12fa6d30da560a0:

[1.7.x] Fixed #23593 -- Fixed crash in AdminEmailHandler with non-ASCII characters in request.

Thanks edevil for the report and Simon Charette for review.

Backport of 9dff5ce7c7 from master