regression in test discovery

[Brian May]

I looked through ​https://docs.djangoproject.com/en/1.8/releases/1.8/ and didn't find anything - apologies if I missed something.

Looking at django-guardian project from ​https://github.com/django-guardian/django-guardian/tree/v1.3 - the latest release

If I run the tests using Django 1.7, all the tests run.

If I run the tests using Django 1.8 and Python 2.7 or Python 3.4, it can't find any of the tests.

If I run the tests using Django 1.8 and Python 3.5 it can find the tests.

The tests, fail badly for Django 1.8 however that is not the point of this ticket; rather the point is that the tests are not discovered with Django 1.8 unless Python 3.5 is used. This would appear to be a regression.

Or if this is not a regression, please tell me what I missed.

The tests are in guardian/testapp/tests/init.py which selectively imports from *_test.py files.

I also filled a bug report at django-guardian, see ​https://github.com/django-guardian/django-guardian/issues/356

[Tim Graham]

Could you please use ​git bisect to find the Django commit that introduced the behavior change?

[Brian May]

Ok, will look into doing a bisect. Just as soon as I get some time :-)

[Brian May]

Having problems doing a reliable bisect.

bbe7964185fb786599b9f4bcaa07fc54156c9e0d - migrations don't seem to work properly, get "django.db.utils.OperationalError: no such table: auth_email" - this appears to happen before test discover although I am not 100% certain on this.

next commit:

e55d888b6680a45ca957434599c37f0286034cab - no errors, no tests being run.

So it is possible that I found the commit that causes the problem. Or I found the commit that fixes the db migration issue - which I don't particularly care about. In actual fact I don't see anything in that commit at quick glance should do any of the above.

Is there some work around for the no such table error so I can get a more reliable bisect?

[Brian May]

Still trying to understand this, however beginning to think this isn't a Django regression after all. The file guardian/testsettings.py contains:

if django.VERSION < (1, 8):
    TEST_RUNNER = 'django.test.simple.DjangoTestSuiteRunner'
else:
    TEST_RUNNER = 'django.test.runner.DiscoverRunner'

So it might be expected behavior for the different test runner.

Still don't understand why Python 3.5 is different, suspect there might be a good explanation for this too.

[Brian May]

From a Debian mailing list discussion:

"There was a change in unittest autoloader in Python 3.5. It now tries to import the package even if it has no tests. ... ​http://bugs.python.org/issue22457"

So I suspect Python 3.5 loads guardian/testapp/tests/init.py even though it didn't find any tests, and suddenly discovers the tests.

Think we can close this bug report.

[Tim Graham]

Looks correct to me, thanks for the analysis.