Test suite side-effects cause dateformat tests to fail

[Malcolm Tredinnick]

Running the full tests (as of r10008) shows a failure in the dateformat tests. Running various tests in isolation plus the dateformat test does not generate the failure. So this is some kind of test suite unintended side-effect. Seems to happen under every backend.

The failing test reports this

======================================================================
FAIL: Doctest: regressiontests.dateformat.tests
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/buildbot/slave/parts/ubuntu-8.04-python2.4-mysql5.0/django-trunk_ubuntu-8.04-python2.4-mysql5.0/
build/django/test/_doctest.py", line 2180, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for regressiontests.dateformat.tests
  File "/var/buildbot/slave/parts/ubuntu-8.04-python2.4-mysql5.0/django-trunk_ubuntu-8.04-python2.4-mysql5.0/
build/tests/regressiontests/dateformat/tests.py", line 0, in tests

----------------------------------------------------------------------
File "/var/buildbot/slave/parts/ubuntu-8.04-python2.4-mysql5.0/django-trunk_ubuntu-8.04-python2.4-mysql5.0/
build/tests/regressiontests/dateformat/tests.py", line 26, in regressiontests.dateformat.tests
Failed example:
    no_tz or format(my_birthday, 'O') == '+0100'
Expected:
    True
Got:
    False
----------------------------------------------------------------------
File "/var/buildbot/slave/parts/ubuntu-8.04-python2.4-mysql5.0/django-trunk_ubuntu-8.04-python2.4-mysql5.0/
build/tests/regressiontests/dateformat/tests.py", line 30, in regressiontests.dateformat.tests
Failed example:
    no_tz or format(my_birthday, 'r') == 'Sun, 8 Jul 1979 22:00:00 +0100'
Expected:
    True
Got:
    False
----------------------------------------------------------------------
File "/var/buildbot/slave/parts/ubuntu-8.04-python2.4-mysql5.0/django-trunk_ubuntu-8.04-python2.4-mysql5.0/
build/tests/regressiontests/dateformat/tests.py", line 38, in regressiontests.dateformat.tests
Failed example:
    no_tz or format(my_birthday, 'T') == 'CET'
Expected:
    True
Got:
    False
----------------------------------------------------------------------
File "/var/buildbot/slave/parts/ubuntu-8.04-python2.4-mysql5.0/django-trunk_ubuntu-8.04-python2.4-mysql5.0/
build/tests/regressiontests/dateformat/tests.py", line 40, in regressiontests.dateformat.tests
Failed example:
    no_tz or format(my_birthday, 'U') == '300531600'
Expected:
    True
Got:
    False
----------------------------------------------------------------------
File "/var/buildbot/slave/parts/ubuntu-8.04-python2.4-mysql5.0/django-trunk_ubuntu-8.04-python2.4-mysql5.0/
build/tests/regressiontests/dateformat/tests.py", line 52, in regressiontests.dateformat.tests
Failed example:
    no_tz or format(my_birthday, 'Z') == '3600'
Expected:
    True
Got:
    False
----------------------------------------------------------------------
File "/var/buildbot/slave/parts/ubuntu-8.04-python2.4-mysql5.0/django-trunk_ubuntu-8.04-python2.4-mysql5.0/
build/tests/regressiontests/dateformat/tests.py", line 57, in regressiontests.dateformat.tests
Failed example:
    no_tz or format(summertime, 'O') == '+0200'
Expected:
    True
Got:
    False
----------------------------------------------------------------------
File "/var/buildbot/slave/parts/ubuntu-8.04-python2.4-mysql5.0/django-trunk_ubuntu-8.04-python2.4-mysql5.0/
build/tests/regressiontests/dateformat/tests.py", line 61, in regressiontests.dateformat.tests
Failed example:
    no_tz or format(wintertime, 'O') == '+0100'
Expected:
    True
Got:
    False

I snarfed that output from ​the buildbot, but I'm seeing the same results locally.

For anybody investigating this: the wintertime/summertime times are set up to cross a daylight savings boundary from 2005 in Copenhagen (we're very tricky when we write our tests for things like that). The results should be (and have been, until recently) independent of the timezone of the machine doing the testing.

[Malcolm Tredinnick]

Should mention that I haven't run every combination of tests to work out the side-effect. My laptop is slow enough and this is low-priority enough that I've decided to punt for now.

One way to debug this is to run the tests with -v2, capture the output and notice the order. Then run half of the test preceding the dateformat one, plus dateformat and see if the problem appears, etc. Alternatively, get lucky and work out by eye-inspection which test is causing the problem (or work out which revision introduced the failure and work from there).

[Alex Gaynor]

I get a full test suite passage so this is probably somewhat platform dependent. I'm in the default Chicago timezone, so that may be it.

[Karen Tracey]

I'm not seeing this either (using Ubuntu & MySQL/InnoDB, US Eastern Daylight Time). Malcolm you say it happens under every backend but buildbot shows green for sqlite so it doesn't seem to happen there? Currently running the test suite using MyISAM since that's what the failing buildbot is running but that'll take a while to finish...

[Malcolm Tredinnick]

I saw it locally (and repeatedly) under SQLite, initially. Then I verified it happened under PostgreSQL and say the buildbot failing similarly for MySQL.

Most odd. I'll look a bit further when I have time. Does anything change if you set Django's timezone setting to something east of UTC (UTC + some hours, instead of minus some hours, as you'd get in the US)?

[Ramiro Morales]

More information: I'm seeing this when running the test suite agains PostgreSQL 8.1 (that's the version I have access to now, Debian etch). Running bisection shows the problem was introduced in r9994, and this sounds logical and consistent with what Malcolm says above because the regressiontests/app_loading test added in that commit is run before regressiontests/dateformat. It seems the former isn't leaving things in a state expected by the latter (maybe the way it is installing new settings has something to do with this?).

This can be reproduced (by people already seeing it) with ./runtests.py app_loading dateformat

[Karen Tracey]

Replying to ramiro:

This can be reproduced (by people already seeing it) with ./runtests.py app_loading dateformat

Huh. If I run just those two tests, I see the error (using both sqlite, and mysql with either storage engine). If I run the full suite I can't recreate it. Time zone setting doesn't seem to matter as I can see the error with settings both east and west of GMT if I run just those two tests, and cannot see the error even with a time zone set east of GMT when I run the full suite. Odd.

[Ramiro Morales]

Replying to ramiro:

regressiontests/app_loading test added in that commit is run before regressiontests/dateformat. It seems the former isn't leaving things in a state expected by the latter (maybe the way it is installing new settings has something to do with this?).

Problem seems to be related to the fact that django.conf.Settings' __init__ ​touches global state and it's being ​called to set up the regressiontests/app_loading tests

[Ramiro Morales]

One attempt at fixing this by modifying django.conf.Settings.init

[Malcolm Tredinnick]

Awesome, Ramiro! Everything makes sense and it was a recent change that caused it, not something else that changed on my system. Patch looks like roughly the right thing, too.

[Malcolm Tredinnick]

(In [10016]) [1.0.X] Fixed #10439 -- Fixed a subtle test failure caused by r9995.

Thanks to Ramiro Morales for debugging what was going on here.

Backport of r10015 from trunk.