clean_html() bug

[Nikolay <djangoproject@…>]

The following is reproducible scenario for the issue:

[0:28 /home/nik]$ python
Python 2.5.2 (r252:60911, Apr 17 2008, 13:15:05) 
[GCC 4.2.3 (Debian 4.2.3-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import django
>>> django.utils.html.clean_html("<p>* * *</p><p>.</p>")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/niksite/lib/site-python/django/utils/functional.py", line 239, in wrapper
    return func(*args, **kwargs)
  File "/home/niksite/lib/site-python/django/utils/html.py", line 158, in clean_html
    text = hard_coded_bullets_re.sub(replace_p_tags, text)
  File "/home/niksite/lib/site-python/django/utils/html.py", line 156, in replace_p_tags
    s = s.replace('<p>%s' % d, '<li>')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 3: ordinal not in range(128)
>>> 

[George Vilches]

I've generated a patch for this ticket which prevents the crash error, although the output of your input looks strange to me.

>>> import django
>>> django.utils.html.clean_html(u"<p>* * *</p><p>.</p>")
u'<ul>\n<li> * *</li><p>.</li>\n</ul>'

However, I'm comfortable with saying that if it is invalid, that should be treated as a separate bug.

Also, the only thing that might be questionable with this patch is the change to utils/html.py, line 16:

DOTS = [u'·', u'*', u'\xe2\x80\xa2', u'•', u'•', u'•']

The 3rd entry might not be a safe Unicode conversion. However, all the unit tests pass. Can someone comment?

[George Vilches]

Patch against r7601 to fix crash error with clean_html.

[Adrian Holovaty]

Needs tests!

[Preston Timmons]

Thanks for updating the patch. The approach looks good but when I apply it I get the following test failure:

........................................................................F.............................
======================================================================
FAIL: test_clean_html (regressiontests.utils.html.TestUtilsHtml)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/pycon2011/djangoenv/src/django/tests/regressiontests/utils/html.py", line 135, in test_clean_html
    self.check_output(f, value, output)
  File "/root/pycon2011/djangoenv/src/django/tests/regressiontests/utils/html.py", line 14, in check_output
    self.assertEqual(function(value), output)
AssertionError: u'<p>I kill<br />whitespace</p>' != u'<p>I kill whitespace</p>'

[Aymeric Augustin]

Oops - new patch fixes the test failure.

[Preston Timmons]

The new patch looks good. I verified it addresses the bug and passes tests. It applies cleanly against r16094.

[Luke Plant]

The issue regarding '\xe2\x80\xa2' does need to be addressed. This is a UTF-8 bytestring sequence for ​Unicode character 2022. The correct way to convert it to a unicode object is:

    '\xe2\x80\xa2'.decode('utf-8')

which results in u'\u2022', not u'\xe2\x80\xa2'

[Aymeric Augustin]

This new patch addresses lukeplant's comment.

[Luke Plant]

In [16118]:

Fixed #7267 - UnicodeDecodeError in clean_html

Thanks to Nikolay for the report, and gav and aaugustin for the patch.