Bug in admin with date_hierarchy and Daylight savings

[Hugo Tácito]

When I have a date_hierarchy in admin and I select a range with a record with the day equal the day of a daylight savings I got an error of NonExistentTimeError.

I discovered that indeed 2016-10-16 00:00:00 does not exist in my Timezone ('America/Sao_Paulo') because that was when Daylight Savings started.

So when I have a record with that date in my database, it will brake the date_hierarchy feature.

[Tim Graham]

I'm not sure what the expected behavior is. What could Django do if you have bad data?

[Hugo Tácito]

Hello Tim,

I'm sorry for my lack of information and clarity, but even if I do have an existing datetime the problem still occur.

For example I have a User with date_joined equals 2016-10-16 12:17:44.270221-03 and when I select a date_hierarchy filter, that still raise the same exception of:

Environment:


Request Method: GET
Request URL: http://localhost:8000/admin/auth/user/

Django Version: 1.9.9
Python Version: 2.7.12
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'daylight']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware']


Template error:
In template /Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/contrib/admin/templates/admin/change_list.html, error at line 73
   2016-10-16 00:00:00   63 :       {% endif %}
   64 :     {% endblock %}
   65 :     {% if cl.formset.errors %}
   66 :         <p class="errornote">
   67 :         {% if cl.formset.total_error_count == 1 %}{% trans "Please correct the error below." %}{% else %}{% trans "Please correct the errors below." %}{% endif %}
   68 :         </p>
   69 :         {{ cl.formset.non_form_errors }}
   70 :     {% endif %}
   71 :     <div class="module{% if cl.has_filters %} filtered{% endif %}" id="changelist">
   72 :       {% block search %}{% search_form cl %}{% endblock %}
   73 :       {% block date_hierarchy %} {% date_hierarchy cl %} {% endblock %}
   74 : 
   75 :       {% block filters %}
   76 :         {% if cl.has_filters %}
   77 :           <div id="changelist-filter">
   78 :             <h2>{% trans 'Filter' %}</h2>
   79 :             {% for spec in cl.filter_specs %}{% admin_list_filter cl spec %}{% endfor %}
   80 :           </div>
   81 :         {% endif %}
   82 :       {% endblock %}
   83 : 


Traceback:

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  174.                     response = self.process_exception_by_middleware(e, request)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  172.                     response = response.render()

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/response.py" in render
  160.             self.content = self.rendered_content

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/response.py" in rendered_content
  137.         content = template.render(context, self._request)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/backends/django.py" in render
  95.             return self.template.render(context)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py" in render
  206.                     return self._render(context)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py" in _render
  197.         return self.nodelist.render(context)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py" in render
  992.                 bit = node.render_annotated(context)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py" in render_annotated
  959.             return self.render(context)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/loader_tags.py" in render
  173.         return compiled_parent._render(context)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py" in _render
  197.         return self.nodelist.render(context)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py" in render
  992.                 bit = node.render_annotated(context)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py" in render_annotated
  959.             return self.render(context)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/loader_tags.py" in render
  173.         return compiled_parent._render(context)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py" in _render
  197.         return self.nodelist.render(context)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py" in render
  992.                 bit = node.render_annotated(context)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py" in render_annotated
  959.             return self.render(context)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/loader_tags.py" in render
  69.                 result = block.nodelist.render(context)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py" in render
  992.                 bit = node.render_annotated(context)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py" in render_annotated
  959.             return self.render(context)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/loader_tags.py" in render
  69.                 result = block.nodelist.render(context)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py" in render
  992.                 bit = node.render_annotated(context)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/base.py" in render_annotated
  959.             return self.render(context)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/template/library.py" in render
  223.         _dict = self.func(*resolved_args, **resolved_kwargs)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/contrib/admin/templatetags/admin_list.py" in date_hierarchy
  380.                 } for day in days]

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/db/models/query.py" in __iter__
  258.         self._fetch_all()

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/db/models/query.py" in _fetch_all
  1074.             self._result_cache = list(self.iterator())

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/db/models/query.py" in __iter__
  158.         for row in compiler.results_iter():

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in results_iter
  808.                     row = self.apply_converters(row, converters)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in apply_converters
  792.                 value = converter(value, expression, self.connection, self.query.context)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/db/models/expressions.py" in convert_value
  950.             value = timezone.make_aware(value, self.tzinfo)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/django/utils/timezone.py" in make_aware
  364.         return timezone.localize(value, is_dst=is_dst)

File "/Users/hugo/.virtualenvs/suap/lib/python2.7/site-packages/pytz/tzinfo.py" in localize
  327.                 raise NonExistentTimeError(dt)

Exception Type: NonExistentTimeError at /admin/auth/user/
Exception Value: 2016-10-16 00:00:00

Even with an existing datetime.

The admin I did to test:

class UserAdmin(admin.ModelAdmin):
    date_hierarchy = 'date_joined'
    list_display = ('email', 'first_name', 'last_name', 'date_joined')
    list_filter = ('is_staff', 'is_superuser')


admin.site.unregister(User)
admin.site.register(User, UserAdmin)

The internationalization settings:

LANGUAGE_CODE = 'pt-br'

TIME_ZONE = 'America/Sao_Paulo'

USE_I18N = True

USE_L10N = True

USE_TZ = True

With TIME_ZONE = 'UTC' the problem doesn't occur.

Edit:
I've did this in a new clean project, with just this object at the User model.

[Hugo Tácito]

This also happens if we set a datetime to 2015-10-18. For example: 2015-10-18 10:45:29.270221-03
In 2015 at 10-18 was started the daylight savings at Brazil too.

​https://www.timeanddate.com/time/change/brazil/sao-paulo?year=2015

[Tim Graham]

Any idea what Django should do to resolve this?

[Hugo Tácito]

Hello Tim,

It seems that the datetime truncation is too hard in the date_hierarchy templatetag.

Changing this line:
​https://github.com/django/django/blob/master/django/contrib/admin/templatetags/admin_list.py#L385

to:
days = getattr(days, dates_or_datetimes)(field_name, 'second')

Probably will fix it.
I can also create a patch if you want.

[Aymeric Augustin]

The problem seems to stem from the fact that Django wants dates, but fetches them as datetimes, and applies the converter for datetimes, which causes an exception in that case.

It isn't possible to represent "2016-10-16 in local time in São Paulo" as datetime.datetime(2016, 10, 16, tzinfo=pytz.timezone('America/Sao_Paulo')) because that value doesn't exist. The only proper way to represent it is datetime.date(2016, 10, 16).

I believe the resolution should be along the lines of making the year / month / day truncation expressions cast their result to a DateField instead of a DateTimeField. Reducing the problem to a lower-level test case may help.

It seems possible that the same problem could occur with the hour truncation expression if some time zones have their DST between 1:30 and 2:30. I'm not sure what we could do about that; it seems much harder to fix.

[Anton Samarchyan]

I've attached a django project to reproduce the problem. Admin credentials - admin/AiNeuw3uOhvei7da
To reproduce, open /admin/app/test/?date__month=10&date__year=2016

[Anton Samarchyan]

Added ​PR

[Tim Graham <timograham@…>]

In e88d2df:

Fixed #27475 -- Fixed NonExistentTimeError crash in ModelAdmin.date_hierarchy.