formtools.utils.form_hmac is not consistent with unicode input

[Claude Paroz]

This problem can only be reproduced with cPickle. When feeding the django.contrib.formtools.utils.form_hmac function with Unicode values, stripping the leading/ending spaces produces different pickled results than the unstripped identical values. Here is an output of the two form_hmac passes in the FormHmacTests.test_textfield_hash when you convert form values to Unicode:

(Pdb) p data, pickled
([('name', u'joe'), ('bio', u'Nothing notable.')], '\x80\x02]q\x01(U\x04nameq\x02X\x03\x00\x00\x00joeq\x03\x86q\x04U\x03bioq\x05X\x10\x00\x00\x00Nothing notable.q\x06\x86q\x07e.')

(Pdb) p data, pickled
([('name', u'joe'), ('bio', u'Nothing notable.')], '\x80\x02]q\x01(U\x04nameq\x02X\x03\x00\x00\x00joe\x86q\x03U\x03bioq\x04X\x10\x00\x00\x00Nothing notable.q\x05\x86q\x06e.')

Possible workarounds:

• use the (slower) pickle module only, as I don't think the performance penalty will be noticeable in this part of the code
• encode all unicode values to 'utf-8' (smart_str) before feeding them to pickle.dumps
• feed repr(data) to pickle.dumps

[Claude Paroz]

Do not use cPickle

[Claude Paroz <claude@…>]

In [1a66f53f9413583d1da266356bb35e5f82868641]:

Fixed #18340 -- Fixed formtools form_hmac with Unicode input

Using cPickle, two apparently identical Unicode strings could
generate different pickled results depending on previous operations
on those strings.

[Claude Paroz <claude@…>]

In [078ea51b1c7504e3f941bc1d96b0499ace29bdb0]:

Fixed test failures after commit 1a66f53. Refs #18340

[anonymous]

It's worth noting that while pickle currently seems to be idempotent, it's not guaranteed to anywhere in the spec. Relying on pickling python objects might not be the ideal solution here.

[Claude Paroz <claude@…>]

In [b109ff8062f4bb225181ec462d69c9dd79339567]:

Replaced pickle by json in form_hmac calculation

Refs #18340