Make it possible to specify tests to skip when running runtests.py

[Ramiro Morales]

Not a bug but a small enhancement to the test runner. It allows one to run, for example:

tests $ runtests.py --verbosity=2 -- -basic

to run all the Django test suite minus the modeltests/basic ones

or

tests $ runtests.py -- -model_forms -dispatch

to run all the Django test suite minus modeltests/model_forms and regressiontests/dispatch ones.

[Malcolm Tredinnick]

That's a very bad syntax. Things start with a single dash are short-form options. Don't overload that (for example "-validate" is ambiguous, since -v is already used as an option).

An "exclude certain tests" is probably reasonable, but you probably need a proper option for it.

[Russell Keith-Magee]

The best option for syntax is to take the lead from #7254, and use an --exclude option.

[Ramiro Morales]

Patch modified to use --exclude option and updated to r9569

[(none)]

Milestone post-1.0 deleted

[Ramiro Morales]

Eric Hoslscher correctly notes in a ​comment to #4501 that the Django simple test runner run_tests() function is an ​exposed API so adding a parameter like the patch does with the exclude_labels parameter would introduce a backwards-incompatibility for third party test runners.

[Ramiro Morales]

Patch updated to r12262

[Ramiro Morales]

I don't know is this is worth adding to the Django test infrastructure or belongs to a custom runner but has been very helpful when using the Django test suite to so I thought it could also be useful to others.

[Eric Holscher]

Responded to ramiro on IRC, but putting my thoughts here too to have a better record.

Seems that this should be exposed to the user-facing test management command as well. I think that having as little special in runtests, and keeping feature parity there is a good goal. This also means that appropriate documentation should be added.

Seems that there should also be some tests, at least running through the code-path where the suite is built, and making sure that excluded things aren't in that suite.

Otherwise this looks like a big win.

[Ramiro Morales]

Replying to ericholscher:

Responded to ramiro on IRC, but putting my thoughts here too to have a better record.

Seems that this should be exposed to the user-facing test management command as well. I think that having as little special in runtests, and keeping feature parity there is a good goal. This also means that appropriate documentation should be added.

Problem is that currently building of the test suite is delegated to a tightly cooperation of the Django app cache and unittest2 (​http://code.djangoproject.com/browser/django/trunk/django/test/simple.py?rev=14696#L209). *

In the test management command case, the app cache is populated from INSTALLED_APPS and we don't have (yet) an API to get app names (the ones we would get form the command line) from app objects (the ones we are handling in django.db.models.get_apps()), we need this to hook there and skip addition of an app to the suite.

Also, unittest[2] doesn't have an API to remove tests from a suite once it is built and before running it.

In runtests.py we have more control over what apps from tests/regressiontests and tests/modeltests are loaded and can control the suite contents by simply avoiding the loading of the apps we want excluded.

AFAICT this mean it would be practical to only add this feature to the runtests.py script.

• This lack of granularity is the same root cause that keep us from excluding test cases or test methods; so the syntax supported by the --exclude switch will only allow the user to exclude entire test apps from the test run.

[patchhammer]

8363-r14297.2.diff fails to apply cleanly on to trunk

[Julien Phalip]

So I've spent a bit of time working on this. The attached patch implements full support for the --exclude option both for the test and runtests commands. To do so I've had to refactor the DjangoTestSuiteRunner quite heavily following an approach that one may not find appropriate, so I'd welcome some feedback.

In a nutshell, the suggested approach is to generate a one-level-deep dictionary: label->test, where label is a full string reference to a test (of the form app.TestClass.test_method) and test is either a test case or a doctest instance. For example:

{
    'auth.AnonymousUserBackendTest.test_get_all_permissions':
        <django.contrib.auth.tests.auth_backends.AnonymousUserBackendTest testMethod=test_get_all_permissions>,
    'auth.AnonymousUserBackendTest.test_has_module_perms':
        <django.contrib.auth.tests.auth_backends.AnonymousUserBackendTest testMethod=test_has_module_perms>,
    ...
}

This then makes it trivial to exclude some given tests before running the test suite by removing the dictionary items corresponding to the excluded tests' labels. However, there is a concern: using this flat structure means that some information may get lost in the process (e.g. the nesting of the various sub-suite instances) which may potentially introduce some annoying limitations in the future.

By the way, about 13 tests out of 4065 don't seem to be run after this patch. I need to narrow down the cause of this. Another note: it seems bizarre that, even in trunk, both unittest.TestLoader and unittest.defaultTestLoader are used in simple.py.

[Jacob]

Milestone 1.3 deleted

[Mitar]

I would really like to see this also a settings options, like ​AdvancedTestSuiteRunner has.

[Preston Holmes]

julien,

this would be pretty handy - but the patch is pretty stale an no longer applies cleanly.

I'm trying to debug some testsuite interactions - and there are a couple tests I'd like to take off the table for a bit and come back to.

I found the best workaround is just to remove the module directories from modeltests/regressiontests temporarily - yuck.

[Julien Phalip]

I think this would actually need some deeper refactoring of the test discovery system to make it work more like other frameworks (e.g. pytest, nose). I think carljm and alex have got some initial thoughts on that. It would be a pretty large effort, but one that would solve many problems at once by doing things right.

[Aymeric Augustin]

This will most likely be closed as a duplicate of #17365 at some point.

[Carl Meyer]

The plan for #17365 is basically to make "manage.py test" work just like "python -m unittest discover", and avoid custom Django test-discovery code. As far as I know unittest does not provide the feature requested here, so closing this as a duplicate of #17365 would be essentially the same as closing wontfix. This is personally fine with me, but a number of other core devs here have commented that they find this useful.

py.test does provide this feature (via the -k option with a - prefixed keyword). I think switching to py.test for testing Django itself would be a legitimate option; I'd much rather do that than add more custom Django test-runner code. This would be a separate thing from #17365.

[Aymeric Augustin]

I understand.

Like you, I believe the way forwards is to make it possible to run Django's tests with third-party test runners, not to extend our custom test runner. Although some core devs have said they'd find this useful, none of them have found it sufficiently useful to implement it.

I'm going to close the ticket because of the new direction our test discovery is taking — not to prevent anyone from implementing it, just to stop tracking an increasingly irrelevant feature request.