utils.html.urlize mishandles IDN style domain names

[Daniel Ryan]

The urlize function runs urlquote on the url it is processing which incorrectly handles domain names with unicode characters in them (IDN style).

To test, run urlize('​http://c✶.ws'). The text of the link is fine, the href of the anchor tag is not.

[Alex Gaynor]

a) Needs tests.

b) There's almost certainly no cause to catch EVERY single exception coming out of there, it should catch only the applicable errors.

[Daniel Ryan]

Better Exception handling

[Russell Keith-Magee]

Still needs tests; also needs PEP8 (i..e, strip all that extra space around the parentheses).

[Dougal Matthews]

I've reviewed the ticket when attempting to add tests. I'm not sure how to handle IDN domains. I got the following result from a quick doctest in the defaultfilters tests;

Expected:
    u'<a href="http://xn--c-lgq.ws" rel="nofollow">http://c✶.ws</a>'
Got:
    u'<a href="http://xn--c-lgq.ws" rel="nofollow">http://xn--c-lgq.ws</a>'

Should the URL be encoded the same way for both the href and the innerHTML? I wasn't sure if it needed 'html encoding' ? When I attempted adding '<a href="​http://xn--c-lgq.ws" rel="nofollow">​http://xn--c-lgq.ws</a>' to a HTML file and opened it in chrome it displayed the (punny code?) characters rather than the expected unicode result

[Adam Nelson]

#12988 might be relevant to this

[Claude Paroz]

Patch with tests

[Claude Paroz]

New patch handles IDN names in various recognized forms. The visible part is left unconverted, as suggested in comment:3.

[Aymeric Augustin]

In [17348]:

Fixed #13704 -- Handled IDN properly in the urlize template filter. Thanks Claude Paroz for the initial version of the patch.