Add test command option to run tests in a random order

[Andreas Savvides]

Similar in spirit to ​--reverse introduced in 1.8, the idea is to add a --random option that will randomise the test run order. Grouping by test class would be preserved when using this option.

I think the idea behind --reverse is great, so I see something like --random a way of going a step further to help with debugging test cases that are not properly isolated.

Going a step further - which might be unnecessary, so this is just a minor aside - we could even add an option that generates and runs X randomised test runs in all possible orders given a set of tests. That way, people who are really paranoid can confirm that their tests really pass and order genuinely does not matter in any case.

[Tim Graham]

I think we would need some way of writing the order to disk so a particular random order could be reproduced. I'm not sure it's worth it.

[Tim Graham]

If you feel strongly about this, please raise it on the DevelopersMailingList so we can get some more opinions.

[Dan Palmer]

For what it's worth, I don't think this needs writing to disk.

I propose that with --random Django generates a random seed and prints it out at the beginning of the test run. This will likely be stored in shell scrollback, or on disk/in a database if running tests in a CI environment. From there, tests could be re-run with a given seed with --random-seed=XXXXX.

[Ahmad A. Hussein]

I agree with Dan, we could use a random seed to accomplish this. This is worth looking into to expand our test runner's functionality.

It is something already supported in pytest though.

[Chris Jerdonek]

I just posted a message to the developers' list asking if this ticket could be reopened: ​https://groups.google.com/g/django-developers/c/68LZwTr2vpI/m/m3wlER5rAAAJ

[Chris Jerdonek]

I'm reopening the ticket based on the mailing list discussion (a number of people replied in support).

I also posted a PR here: ​https://github.com/django/django/pull/14137

[Chris Jerdonek]

By the way, while working on the PR, I realized that we could name the option --shuffle instead of --random. I felt --shuffle was more specific and self-descriptive, whereas --random can be used in broader contexts (e.g. altering random data in tests themselves). So if we have the opportunity to be more specific in naming, I thought we should take it.

[Adam Johnson]

--shuffle sounds good to me.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 90ba716b:

Fixed #24522 -- Added a --shuffle option to DiscoverRunner.

[GitHub <noreply@…>]

In f5dccbaf:

Refs #24522 -- Fixed code comment about seeds in Shuffler.init().