Add tox support

[Jeremy Bowman]

tox (​https://testrun.org/tox/) makes it easier to run tests in multiple versions of Python, either all in the same test run or one at a time (among other features). Given that Django currently supports multiple Python versions straddling 2.x and 3.x, making this easier seems like a significant win. Key points:

• A tox.ini file would be added which defines all the tox environments
• The default environment could be run just withtox
• Specific versions could be run via tox -e py27, tox -e py33, tox -e pypy, etc.
• Multiple versions could be tested in a single run via commands like tox -e py27,py32,py33,pypy or tox -e ALL (especially handy if you'll be stepping away from your computer for a while)
• The desired python interpreter(s) would need to be installed beforehand, but tox itself would create the virtualenv(s) and install any necessary dependencies into it
• Documentation could be built with tox -e docs
• In comparison to the current documented commands for running tests, this would be faster to type and easier to remember
• The documentation would be updated to mention this as one way (the recommended way?) to run tests and build documentation
• No other changes would be needed beyond the creation of tox.ini, so all the old ways of running tests and such would still work unchanged (tox itself would still run those commands)

One attempt at adding tox support was made 2 years ago (​https://github.com/django/django/pull/110), but had been lumped together with TravisCI support and was rejected on those grounds. It seems more valuable now with the desire to be able to test Python 2, Python 3, and PyPy (and others?) easily.

[Simon Charette]

As discussed with Jannis, who closed the ​referenced PR, at PyCon2014 sprints we might want to move this discussion the django-developers mailing list.

While it could be useful to provide a simple way for contributors to run the test suite against multiple versions of Python we'd need to add a big warning this is by no mean a way of achieve (given all the supported environments {Unix, Win} X {SQLite3, MySQL, PostGres, Oracle, ...} X {Py2.7, ..., Py3.4})

In my case I would definitely leverage this when reviewing patches, I actually have a tox.ini file I keep of VCS tracking using .git/info/exclude.

[Tim Graham]

I'm in favor of recommending the ​djangocore-box, but tox could probably be leveraged there as well.

[Tim Graham]

Accepting the idea of exploration in this area.

[Collin Anderson]

Is this still needed now that we have ​http://djangoci.com/?

Not sure what the approach would be now. Maybe a docker container with all of the databases (kind of outside the scope of django itself)? Or an ansible script to set it all up?

[Berker Peksag]

+1 for tox support. tox is really helpful when you want to run specific tests on multiple Python versions.

Here is my tox.ini as a reference point: ​https://gist.github.com/berkerpeksag/1f644aa3454f99bb4c51 (It doesn't support different backends such as MySQL.)

[Tobias McNulty]

(Also made a similar post to #19891)

I created a "lightweight" Travis/tox setup for my own purposes so I can check pull requests before submitting them to Django proper. The goal is not to replace djangoci.com but to provide a lightweight sanity check and help avoid clogging up Travis when making frequent commits.

If there is a desire to include a tox.ini with Django itself, this file might work as a foundation: ​https://github.com/tobiasmcnulty/django/blob/travis/tox.ini

As I type this, I'm wondering if we could supply a generic tox.ini that includes all the builds Jenkins runs. Then Jenkins could run tox -e for each applicable environment, and anyone wanting to run their own Travis build server could do the same for whatever environments they care about.

In the meantime, anyone who stumbles across this ticket is welcome to copy what I did to your own GitHub/Travis account or submit a PR to my fork to see if it builds, before submitting to Django proper (Jenkins will only fully build your PRs if your username is on the whitelist). I will not, of course, review your PR or merge it into this fork. There are no guarantees that it'll be completely up to date, etc. (create a fork for yourself if needed).

The repo is here, on the default "travis" branch: ​https://github.com/tobiasmcnulty/django

[Tobias McNulty]

PR is here: ​https://github.com/django/django/pull/6807

[Tim Graham]

Comments for improvement on the PR.

[Tim Graham <timograham@…>]

In 09d3874:

Fixed #22446 -- Added tox.ini to automate pull request checks.