Provide a way to run only the selenium tests

[Tim Graham]

It would be nice if we had an option to run only the selenium tests in the Django test suite. These tests are slow and don’t run by default (you must specify ./runtests.py --selenium). A common use case is to want to check the selenium tests after a normal run of the test suite which skips them. Currently you must rerun the entire test suite with selenium enabled instead of running only the selenium tests that were skipped in the first test run.

I think it would be okay if we changed ./runtests.py --selenium to do that. My argument is that there isn't a common need to run both the non-selenium and selenium tests in one command which that option provides now. Feel free to voice disagreement on this; for example, if you think we should add a separate option for running only the selenium tests.

I'm not sure about the best implementation, so to whoever picks this up, please share what you have in mind before coding.

[Asif Saifuddin Auvi]

+1

any guide for workaround?

[Anssi Kääriäinen]

I really, really think we want to add "test tags". Users could define tags manually. Selenium tests could have tag "selenium" by default.

So, to run tests without selenium tests: python manage.py test --exclude-tags=selenium. To run only selenium tests: python manage.py test --tags=selenium.

Tags would have a lot of other uses, too. Say, you might want to tag a subset of your tests as smoke tests, or you might want to tag those tests that are slow (not only selenium tests are slow).

If you want to do an incremental test run, and you have tags fast, slow and selenium.

python manage.py test --tags=fast  # run fast tests first
python manage.py test --tags=slow --exclude-tags=selenium  # run those slow tests that aren't selenium
python manage.py test --tags=selenium # run selenium tests

[Carl Meyer]

I agree that tagging or marking tests is a really useful feature.

It seems worth noting that this would be another case where we would be re-inventing an existing py.test (and probably nose?) feature in our test runner.

[Jakub Paczkowski]

I've prepared a proof of concept(​https://github.com/django/django/pull/5585) to this feature- what do you think about it?

About re-inventing py.test or nose- well, in my opinion we're doing this this partially this way or another and this is feature small enough to fit into our codebase. It's also quite useful so shouldn't be that much of a problem.

[Carl Meyer]

Yes, I think you're right. The patch is small and useful, and I don't think switching to nose or py.test is a good or feasible plan in the medium-term anyway (posted more rationale for that just now in #25707).

PR just needs some cleanup, and there's a question about whether we want an --exclude-tags flag as well. I think probably we'll end up wanting it, but it could also always be added later if we do.

[Carl Meyer]

I've created a new ticket (#25735) specifically for the test-tagging feature, since currently that's all the PR implements, and I think it's best to separate the new feature from its use in Django's own test suite.

Even once we merge that feature, before this ticket can be closed we still need to figure out how the use of the new tagging feature for Selenium tests interacts with the existing --selenium option. Some options (all of these options assume that we tag all Selenium tests with a "selenium" tag):

1) Deprecate/remove --selenium and just use the tagging feature alone. This means that you would need to specify --exclude-tags selenium to get the equivalent of the current behavior. I don't like this option, because I still don't think the Selenium tests should run by default in local development: they are too slow and the constant windows-popping-up is too invasive.

2) Leave --selenium as-is. This means that to run all tests you'd provide --selenium, to run Selenium only you'd provide --selenium --tags selenium. The latter seems a bit redundant, but perhaps OK?

3) Leave --selenium as-is, and make --tags selenium automatically imply --selenium. So to run all tests you'd run --selenium, to run only the Selenium tests you'd run --tags selenium.

There are probably other reasonable permutations I haven't considered. Thoughts?

[Tim Graham]

Unless I missed something, the --selenium option is specific to runtests.py so I don't think we need to worry about backwards compatibility and can just remove it.

[Carl Meyer]

I'm not concerned about backwards-compatibility with the --selenium option, I'm concerned about arriving at a good usable situation for Django's internal tests, and (as I described above) I'm not sure that "just remove --selenium" achieves that. I mentioned above why I don't like the option of just removing --selenium -- because that would imply that Selenium tests would be run by default, which I don't think is good.

Another option might be to get rid of --selenium and then have --exclude-tag=selenium be the default, but it's not obvious how that would work either. If you used --exclude-tag=somethingelse would the Selenium tests suddenly start running as a side effect? That's not good. So if you can't override the default that way, how can you ever run the Selenium tests at all, since --exclude-tag overrides --tag?

[Tim Graham]

I think I would remove --selenium and add some logic in runtests.py that appends selenium to --exclude-tag except if --tag=selenium is specified.

(Sorry, I didn't read your first comment closely.)

[Carl Meyer]

No problem. That's an option, but then how do you run "all the tests, including Selenium" (which I presume is what CI wants to do)? --tag=selenium means "run only the Selenium tests."

[Tim Graham]

On CI we have a separate build for selenium and I'd like to run only those tests there. If we ever moved to a testing service like Sauce Labs, I think we'd want the same functionality.

[Carl Meyer]

Ah right! I forgot that CI does run Selenium separately. In that case I think your proposal is fine. Personally for local dev I think I'd be quite happy to either run the full test suite without Selenium, or just the Selenium tests, but not have a simple way to run both together.

[Tim Graham <timograham@…>]

In 6670da75:

Fixed #25653 -- Made --selenium run only the selenium tests.

[Tim Graham <timograham@…>]

In 87994b40:

Refs #25653 -- Corrected help text for runtests.py --selenium option.