DateTimeField, DateField fails covert unicode date string with non-ascii characters

[Gene]

it's old python bug - [python-Bugs-1280061]
in my case (ru_RU)

>>>from sys import platform
>>>from locale import setlocale, LC_ALL
>>>locale = 'russian' if platform == 'win32' else ''
>>>setlocale(LC_ALL, locale) #set locale for '%d %b %Y' convertion

>>>import time, datetime
>>>datetime.date(*time.strptime(u'31 мая 2011', '%d %b %Y')[:3]) # like in django/forms/fields.py, Line 362
ValueError: time data u'31 \u043c\u0430\u044f 2011' does not match format '%d %b %Y'

for strings contains non-ascii characters, you need

'string'.encode('utf8')

>>>datetime.date(*time.strptime(u'31 мая 2011'.encode('utf8'), '%d %b %Y')[:3])
datetime.date(2011, 5, 31)

[Jonas H.]

Seems like the right solution to me.

[Ramiro Morales]

Patch need to be generated from the root of the source code tree, needs tests.

[Wim Feijen]

Workon on this during DjangoCon Europe 2011 sprints

[Wim Feijen]

Bug is confirmed against trunk.

Suggested patch wirks, but has to be updated to trunk; and tests are needed. I'll try to write those in the upcoming hour.

# in settings.py:
LANGUAGE_CODE = 'ru'

# BEFORE PATCH:
>>> from django.forms.fields import DateField
>>> from locale import setlocale, LC_ALL
>>> setlocale(LC_ALL, 'ru_RU.utf-8')
'ru_RU.utf-8'
>>> DateField().strptime(u'31 мая 2011', '%d %b %Y')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/django/forms/fields.py", line 376, in strptime
    return datetime.date(*time.strptime(value, format)[:3])
  File "/usr/lib/python2.7/_strptime.py", line 454, in _strptime_time
    return _strptime(data_string, format)[0]
  File "/usr/lib/python2.7/_strptime.py", line 325, in _strptime
    (data_string, format))
ValueError: time data u'31 \u043c\u0430\u044f 2011' does not match format '%d %b %Y'

# AFTER PATCH:
>>> from django.forms.fields import DateField
>>> from locale import setlocale, LC_ALL
>>> setlocale(LC_ALL, 'ru_RU.utf-8')
'ru_RU.utf-8'
>>> DateField().strptime(u'31 мая 2011', '%d %b %Y')
datetime.date(2011, 5, 31)

[Wim Feijen]

Patch for ticket 16123

[Wim Feijen]

Ok, I included a new patch which fixes the problem.

Before committing, there are three things to look at:

1. You need to have the Russian language installed to run the test...
2. The tests import DateField and DateTimeField. Is this the proper way to do it? If so, the imports should be placed on top op the file. If not, the test should be rewritten.
3. In class BaseTemporalField, method to_python (django/forms/fields.py, line 347) a similar statement is used:

   datetime.datetime(*time.strptime(datetime_str, format[:-3])[:6]+(usecs,))
   

which might need to be converted to

datetime.datetime(*time.strptime(datetime_str.encode('utf-8'), format[:-3])[:6]+(usecs,))

but I cannot write a test case for this, because I do not know where this is being used.

[Claude Paroz <claude@…>]

In 4b3f7110ae1e9ef7862c67a38431780080f0062c:

Fixed #16123 -- Ensured strptime receive proper string type

strptime generates an UnicodeEncodeError when using a non-ascii
unicode string on Python 2.