Add a --failfast argument to the testrunner

[Alex Gaynor]

This would basically signal the testrunner to stop and report the failure as soon as the failure occured, rather than obligating the user to wait for the test suite to finis running to see the failure. This is useful while testing for regressions.

[Andriy Drozdyuk]

I think simple argument of "--verbosity=2" will let you know which tests failed.
Failing the test suite makes no sense as it can be run on an automated (nightly) schedule, which makes not sense stopping it just because of one test.

[Karen Tracey]

Replying to drozzy:

I think simple argument of "--verbosity=2" will let you know which tests failed.

Yes, that will show you what failed (if you happen to notice it before it scrolls off the screen), but you don't get the failure details until the end of the run. If you are running the whole suite that may take a while. If you Ctrl-C the run after you see the first failure you still don't get the failure information so you then have to run just that test again to see if it fails in isolation...plus with this approach you have to be watching the test run like a hawk to catch the first failure. It would be more convenient if there was an option for the run to stop and print the details of the failure once the first failure is encountered. Then you can start the run and periodically check on it to see how it is going -- you don't have to watch it in detail and so long as it keeps running you know you are in good shape.

Failing the test suite makes no sense as it can be run on an automated (nightly) schedule, which makes not sense stopping it just because of one test.

That's why this is proposed as an option. It does certainly make sense when manually running the suite to check for regressions due to a code change you just made.

[Jukka Välimaa]

I've come across situations where something like this would have been useful. When there are 1000+ tests to run to check if anything is broken, getting some info about test failure as soon as it happens can save a lot of time.

[robmadole]

​Related post on django-developers group (there is a mention about the --failfast option).

[Randy Barlow]

w00t. This is very cool.

[Randy Barlow]

Also created #12364, which is similar to this.

[Randy Barlow]

I checked out that patch. It looks good. The unit test didn't include an init.py or a models.py, so it wouldn't run. I created a new patch that includes these files. Thanks for the patch!

[Randy Barlow]

Hey Karen, this one's ready for commit. Go failfast!

[Karen Tracey]

There's a problem in adding a new keyword argument to the test runner: existing alternate test runners won't be expecting the new arg. It would be good to figure out how to add this in a way that won't cause older runners to fail.

[Randy Barlow]

if you have a function foo(arg1, arg2, ...), you can get it's args by foo.func_code.co_varnames. We can use this to find out if our arguments are accepted by the function we are about to pass them to to get around this problem.

[Randy Barlow]

Alrighty, this patch will do the trick!

[Randy Barlow]

Hey Karen, now the patch only will use the failfast flag if the test runner will take it. Thanks!

[Karen Tracey]

(In [11843]) Fixed #11613: Added a failfast option for test running. Thanks jukvalim and Randy Barlow.

[Karen Tracey]

(In [11846]) Added tet runner tests mistakenly left out of r11843. Refs #11613.

[Jacob]

Milestone 1.2 deleted