Template Cache "make_template_fragment_key" function speed up + simplify (also discussing switch to alternate hashes)

[Daniel]

The make_template_fragment_key function in django.core.cache.utils has the following (minor) issues:

• Using urllib.quote for vary_on args, is not needed any more - it was originally added to make the unhashed strings safe to send to memcached and similar restricted systems. But since the value is hashed, this is now adding nothing. (See ​https://github.com/django/django/commit/ebc1325721e43808cef4334edaffc23a43f86614#diff-702b69be0100a594fd6fea1e4ab2feb1).

• Use of the MD5 hashing function is disallowed on certain (odd) systems, not being FIPS compliant. See (​https://github.com/django/django/pull/10605).

• Creates a string of all joined vary_on args to send to the hashing function, rather than using the hashlib .update() method.

Here is a version solving these, switching to SHA256, and speeding up the function quite a bit:

​https://github.com/danthedeckie/django/tree/simplified_make_template_fragment_key

And PR: ​https://github.com/django/django/pull/11772

And here's the repo showing performance improvement:

​https://github.com/danthedeckie/make_template_fragment_key_test

Which seems to be faster in every case.

The downside of this is that the cache key is now different from before. The tests have been updated to the new values.

There are other cache key generating functions used in other places which use MD5 still - if switching to SHA256 it would make sense to me to change those at the same time, meaning only one time invalidating keys on upgrade.

Thoughts?

[Claude Paroz]

I think I would separate the hashing algorithm issue, which seems to be the subject of #28401. In that perspective, using the hashlib.blake2s should also be evaluated, as it seems even quicker than MD5! (​http://atodorov.org/blog/2013/02/05/performance-test-md5-sha1-sha256-sha512/ notably the 2017 update of the post)

Could you make a version of your patch while keeping MD5?

[Daniel]

Replying to Claude Paroz:

Could you make a version of your patch while keeping MD5?

Certainly. The PR now does that, and the comparison test repo gives the following output:

    Testing: vary_on = None
    Original: 0.304896 (1.0)
    New (MD5): 0.180153 (1.692427 x)
    New (SHA256): 0.187187 (1.628828 x)
    New (Blake2s): 0.108020 (2.822575 x)

    Testing: vary_on = ['Hello World', 'This is some more text', 33]
    Original: 1.488445 (1.0)
    New (MD5): 0.434238 (3.427719 x)
    New (SHA256): 0.439837 (3.384082 x)
    New (Blake2s): 0.312340 (4.765460 x)

    Testing: vary_on = ['Single String']
    Original: 0.736225 (1.0)
    New (MD5): 0.281357 (2.616693 x)
    New (SHA256): 0.282072 (2.610059 x)
    New (Blake2s): 0.177312 (4.152141 x)

    Testing: vary_on = [100, 201740820, 10820804, 12048023864, 1]
    Original: 1.480063 (1.0)
    New (MD5): 0.632621 (2.339574 x)
    New (SHA256): 0.640986 (2.309041 x)
    New (Blake2s): 0.448273 (3.301699 x)

    Testing: vary_on = very long string
    Original: 47.067326 (1.0)
    New (MD5): 1.337903 (35.179913 x)
    New (SHA256): 1.854285 (25.383011 x)
    New (Blake2s): 1.374788 (34.236063 x)

[Mariusz Felisiak <felisiak.mariusz@…>]

In 5cb3ed18:

Fixed #30772 -- Optimized make_template_fragment_key().

Removed usage of urllib.quote(), unnecessary since cbbe60c7fc39fa8ff75554bd90104eaad6924bb1.
Used hasher's .update() on key fragments.