uri_to_iri() perfoms percent decoding incorrectly

[Chronial]

The current implementation of uri_to_iri is incorrect.

In step two of ​the algorithm, it should:

Convert all percent-encodings ("%" followed by two hexadecimal digits) to the corresponding octets, except those corresponding to "%", characters in "reserved", and characters in US-ASCII not allowed in URIs.

But instead it just runs an unquote (​source)

This also makes this statement from ​the docs a lie:

Both iri_to_uri() and uri_to_iri() functions are idempotent, which means the following is always true:
uri_to_iri(uri_to_iri(some_string)) == uri_to_iri(some_string)

But at the moment

uri_to_iri(uri_to_iri("%2525")) == "%" != "%25" == uri_to_iri("%2525")

[Tim Graham]

If you could provide a patch soon, we might backport it to 1.8 given this function was introduced there.

[varun naganathan]

Hi,i wanted to work on this.Added a small tweak to avoid converting the percent encoding for '%' i.e. '%25'.basically I'm replacing all instances of '%25' in the uri string with '%'.I tried a few tests and they seem to work.Do let me know if that's fine.I'll organize the code and post a PR.
I'm pretty new to contributing to django so I'm sorry if I wasn't suppose to self assign this bug.

[varun naganathan]

Also I have a doubt regarding the RFC guidelines.When RFC 3987 states that the percent encoding corresponding to '%' i.e. '%25' must not be converted to its octet , does that mean that uri_to_iri("%2525") should return %2525 only as the result?

If %2525 only must be the output I suggest I override the call to the inbuilt unquote() method with a custom written method.Does that sound good?

[varun naganathan]

Created a trial Pull Request
​https://github.com/django/django/pull/6085
Please review

[Chronial]

@timgraham I created A PR against 1.8.x: ​https://github.com/django/django/pull/6426

This now correctly implements the algorithm from the RFC, except for step 4: The implementation is supposed to re-encode some unicode characters “that are not appropriate” for IRIs. I added a note about that to the docstring.

[Chronial]

New PR against master: ​https://github.com/django/django/pull/6428. I also didn't foresee what that this fix would affect the testing infrastructure so I didn't run the whole test suite last time. Now everything should work.

[Philip James]

Setting the 'patch_needs_improvement' bit as there are failing checks and open questions on the PR.

[Chronial]

Replying to phildini:

Setting the 'patch_needs_improvement' bit as there are failing checks and open questions on the PR.

What are the open questions?

[Tim Graham <timograham@…>]

In 03281d8:

Fixed #26005 -- Fixed some percent decoding cases in uri_to_iri().