`date` filter raises an exception for naive datetimes during DST change

[Markus Bertheau]

When an exception, that would normally - with DEBUG=True - be displayed in the detailed 500 page, occurs during the DST/no-DST ambiguous hour of the geographical area that is set as TIMEZONE, the handler crashes with AmbiguousTimeError. This seriously limits the ability of many developers to develop and Django applications for a time span of 1/8760 of the year. This is unacceptable! ;)

2014-10-26 02:40:14,104    ERROR django.request                     :handle_uncaught_exception  231 Internal Server Error: /dashboard/suppliers/
Traceback (most recent call last):
  File "/home/markus/src/django/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/markus/src/django/django/db/transaction.py", line 394, in inner
    return func(*args, **kwargs)
  File "/home/markus/src/django/django/contrib/auth/decorators.py", line 22, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/home/markus/src/django/django/contrib/auth/decorators.py", line 22, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/home/markus/src/django/django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/markus/src/django/django/views/generic/base.py", line 87, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/markus/src/django/django/views/generic/list.py", line 145, in get
    self.object_list = self.get_queryset()
  File "/home/markus/src/machtfit/apps/dashboard/supplier/views.py", line 44, in get_queryset
    .annotate(num_orders=Count('partner__order_line')))
  File "/home/markus/src/django/django/db/models/query.py", line 802, in annotate
    is_summary=False)
  File "/home/markus/src/django/django/db/models/sql/query.py", line 1012, in add_aggregate
    field_list, opts, self.get_initial_alias())
  File "/home/markus/src/django/django/db/models/sql/query.py", line 1419, in setup_joins
    names, opts, allow_many, fail_on_missing=True)
  File "/home/markus/src/django/django/db/models/sql/query.py", line 1383, in names_to_path
    self.raise_field_error(opts, name)
  File "/home/markus/src/django/django/db/models/sql/query.py", line 1389, in raise_field_error
    "Choices are: %s" % (name, ", ".join(available)))
FieldError: Cannot resolve keyword u'order_line' into field. Choices are: <irrelevant>
Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 85, in run
    self.result = application(self.environ, self.start_response)
  File "/home/markus/src/django/django/contrib/staticfiles/handlers.py", line 64, in __call__
    return self.application(environ, start_response)
  File "/home/markus/src/django/django/core/handlers/wsgi.py", line 187, in __call__
    response = self.get_response(request)
  File "/home/markus/src/django/django/core/handlers/base.py", line 199, in get_response
    response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
  File "/home/markus/src/django/django/core/handlers/base.py", line 236, in handle_uncaught_exception
    return debug.technical_500_response(request, *exc_info)
  File "/home/markus/src/django/django/views/debug.py", line 91, in technical_500_response
    html = reporter.get_traceback_html()
  File "/home/markus/src/django/django/views/debug.py", line 350, in get_traceback_html
    return t.render(c)
  File "/home/markus/src/django/django/template/base.py", line 148, in render
    return self._render(context)
  File "/home/markus/src/django/django/test/utils.py", line 88, in instrumented_test_render
    return self.nodelist.render(context)
  File "/home/markus/src/django/django/template/base.py", line 844, in render
    bit = self.render_node(node, context)
  File "/home/markus/src/django/django/template/debug.py", line 80, in render_node
    return node.render(context)
  File "/home/markus/src/django/django/template/debug.py", line 90, in render
    output = self.filter_expression.resolve(context)
  File "/home/markus/src/django/django/template/base.py", line 624, in resolve
    new_obj = func(obj, *arg_vals)
  File "/home/markus/src/django/django/template/defaultfilters.py", line 769, in date
    return format(value, arg)
  File "/home/markus/src/django/django/utils/dateformat.py", line 343, in format
    return df.format(format_string)
  File "/home/markus/src/django/django/utils/dateformat.py", line 35, in format
    pieces.append(force_text(getattr(self, piece)()))
  File "/home/markus/src/django/django/utils/dateformat.py", line 268, in r
    return self.format('D, j M Y H:i:s O')
  File "/home/markus/src/django/django/utils/dateformat.py", line 35, in format
    pieces.append(force_text(getattr(self, piece)()))
  File "/home/markus/src/django/django/utils/dateformat.py", line 136, in O
    seconds = self.Z()
  File "/home/markus/src/django/django/utils/dateformat.py", line 189, in Z
    offset = self.timezone.utcoffset(self.data)
  File "/home/markus/.virtualenvs/machtfit/local/lib/python2.7/site-packages/pytz/tzinfo.py", line 406, in utcoffset
    dt = self.localize(dt, is_dst)
  File "/home/markus/.virtualenvs/machtfit/local/lib/python2.7/site-packages/pytz/tzinfo.py", line 349, in localize
    raise AmbiguousTimeError(dt)
AmbiguousTimeError: 2014-10-26 02:40:14.190834

[Aymeric Augustin]

Good catch.

Either django.utils.dateformat.TimeFormat.Z or django.template.defaultfilters.date should deal with that exception — probably the latter, as failing silently on errors is a behavior of the Django Template Language.

[Tomáš Ehrlich]

I can't reproduce this bug at all.

The debug page formats datetime.datetime.now() with {{ server_time|date:'r' }}. Date format r is equivalent to D, j M Y H:i:s O as seen in traceback and O call Z whitch causes the exception. I tried to simply format datetime from traceback, but I didn't get any exception:

from django.conf import settings
settings.configure(USE_TZ=True, TIME_ZONE='Europe/Amsterdam')  # required for DateFormat, doesn't raise exception for USE_TZ=False neither

import datetime
from django.utils.dateformat import DateFormat

try:
    DateFormat(datetime.datetime(2014, 10, 26, 2, 30)).format('Z')
    DateFormat(datetime.datetime(2014, 10, 26, 2, 30, tzinfo=None)).format('Z')
except AmiguousTimeError:
    pass
else:
    print('Exception wasn't raised')

# Exception wasn't raised...

There seems to be missing information about timezone, which makes date&time 2014-10-26 2:30 really ambiguous, but even when I try to format timezone-naive datetime, it never raises exception.

This bug can actually affects debug page only once a year, when the time is switched back by one hour. So it's after all just 1/4380 of the year ;)

Could you please provide more information? What TIME_ZONE, USE_TZ settings do you use and etc.

[Markus Bertheau]

You have to install pytz to see the error.

[Tomáš Ehrlich]

That's right, with installed pytz I can see the exception.

Minimal code to reproduce the problem:

from django.conf import settings
settings.configure(USE_TZ=True, TIME_ZONE='Europe/Amsterdam')

import datetime
from django.utils.dateformat import DateFormat

right_now = datetime.datetime.now().replace(2014, 10, 26, 2, 30)
print(DateFormat(right_now).format('Z'))

In my opinion it's more serious because it doesn't affect just 500 debug page, but all templates that uses date filter with r, O or Z.

[Tomáš Ehrlich]

Tests are still missings. Should be simple though.

I don't like the fix, but can't figure out anything better. As mentioned in PR: The except block catching all exceptions is ugly, but I can't rely on pytz library, which is optional, and therefore I can't catch pytz.exceptions.AmbiguousTimeError exception.

[Markus Bertheau]

I think handling AmbiguousTimeError in dateformat essentially means we're making a business logic decision on behalf of the implementor that we shouldn't make for him - his business logic might dictate a different decision.

Instead the template code should handle AmbiguousTimeError and silently return an empty string.

Kudos for fixing the root issue where a naive datetime was put in the context :) This part can surely be commit separately, because it solves the first problem that the technical 500 page doesn't work during DST change.

[Aymeric Augustin]

Replying to mbertheau:

Instead the template code should handle AmbiguousTimeError and silently return an empty string.

Yes, that is the correct fix (or at least the fix consistent with how Django handles ambiguous datetimes in general).

[Tomáš Ehrlich]

I understand that usually templates handle exceptions silently, printing nothing on error. I just don't like the idea that for one hour a year, the template won't display datetime at all (for naive datetimes). Yes, we're making a decision instead of users, but they won't know about this issue, since its occurrence is so rare. This isn't a common scenario which is easily tested.

Maybe at least show warning so user can see in logs what happened and why?

[Tomáš Ehrlich]

Alright, I simplified the solution a little bit. Added explanation into docs.

[Markus Bertheau]

Reading the code more closely, I'm convinced, that the correct fix is to change TimeFormat.Z to return an empty string if given a naive datetime. This is in line with python's strftime, which exhibits the same behaviour.

If you agree, then the scope of this is actually bigger: TimeFormat shouldn't make up a timezone for anything, if given a naive datetime. This then concerns TimeFormat.e, TimeFormat.O, TimeFormat.T, TimeFormat.Z and DateFormat.I.

[Markus Bertheau]

I'm not aware of all the circumstances that TimeFormat is given a naive datetime - with that caveat I think showing a warning, similar to what tricoder42 suggested, whenever TimeFormat gets a naive datetime. The warning could be similar to the datetime model field warning.

[Tomáš Ehrlich]

Just to be precise: The problem occurs only for naive datetime during DST change. For other cases the timezone functions/formats works fine even for naive datetimes.

And yes, we should fix all other formats as well. I'll do it later.

[Tomáš Ehrlich]

I've updated the pull request, fixed also other TimeFormat formats.

[Carl Meyer]

I'm not seeing a link to a pull request anywhere in this thread.

[Tomáš Ehrlich]

Replying to carljm:

I'm not seeing a link to a pull request anywhere in this thread.

It's in the ticket's header (​https://github.com/django/django/pull/3576)

[Tomáš Ehrlich]

After discussion on PR, we're waiting for approval from @aaugustin:
Naive datetimes returns empty string for any timezone-related format specifiers (for the whole year). This is backward-incompatible change which will be mentioned in docs.

[Aymeric Augustin <aymeric.augustin@…>]

In c7a6996df7e77bc3b9c5e581e67d766627ebabec:

Fixed a crash of the debug page.

During the autumn DST change, the template engine would fail to convert
the naive representation of now when USE_TZ = True. Passing now in UTC
eliminates the issue.

Thanks mbertheau and tricoder42.

Refs #23714.

[Tim Graham <timograham@…>]

In c6a49d4f17db4579389e3f0477436f2444dead10:

[1.8.x] Fixed a crash of the debug page.

During the autumn DST change, the template engine would fail to convert
the naive representation of now when USE_TZ = True. Passing now in UTC
eliminates the issue.

Thanks mbertheau and tricoder42.

Refs #23714.

Backport of c7a6996df7e77bc3b9c5e581e67d766627ebabec from master

[Aymeric Augustin <aymeric.augustin@…>]

In 1014ba02:

Fixed debug view crash during autumn DST change.

This only happens if USE_TZ = False and pytz is installed (perhaps not
the most logical combination, but who am I to jugde?)

Refs #23714 which essentially fixed the same problem when USE_TZ = True.

Thanks Florian and Carl for insisting until I wrote a complete patch.

[Aymeric Augustin <aymeric.augustin@…>]

In 94d1341:

[1.8.x] Fixed debug view crash during autumn DST change.

This only happens if USE_TZ = False and pytz is installed (perhaps not
the most logical combination, but who am I to jugde?)

Refs #23714 which essentially fixed the same problem when USE_TZ = True.

Thanks Florian and Carl for insisting until I wrote a complete patch.

Backport of 1014ba026e from master

[Tim Graham <timograham@…>]

In ca0278f4:

[1.9.x] Fixed debug view crash during autumn DST change.

This only happens if USE_TZ = False and pytz is installed (perhaps not
the most logical combination, but who am I to jugde?)

Refs #23714 which essentially fixed the same problem when USE_TZ = True.

Thanks Florian and Carl for insisting until I wrote a complete patch.

Backport of 1014ba026e879e56e0f265a8d9f54e6f39843348 from master

[Mariusz Felisiak]

It was fixed by 1014ba026e879e56e0f265a8d9f54e6f39843348 and c7a6996df7e77bc3b9c5e581e67d766627ebabec.