Opened 15 years ago

Closed 15 years ago

Last modified 13 years ago

#11613 closed (fixed)

Add a --failfast argument to the testrunner

Reported by: Alex Gaynor Owned by: Karen Tracey
Component: Testing framework Version: 1.0
Severity: Keywords:
Cc: kmike84@…, Jukka Välimaa, randy@…, Randy Barlow Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description

This would basically signal the testrunner to stop and report the failure as soon as the failure occured, rather than obligating the user to wait for the test suite to finis running to see the failure. This is useful while testing for regressions.

Attachments (2)

failfast_1.diff (7.3 KB ) - added by Jukka Välimaa 15 years ago.
ticket_11613.patch (7.5 KB ) - added by Randy Barlow 15 years ago.
Alrighty, this patch will do the trick!

Download all attachments as: .zip

Change History (18)

comment:1 by anonymous, 15 years ago

Cc: kmike84@… added

comment:2 by Alex Gaynor, 15 years ago

Triage Stage: UnreviewedAccepted

comment:3 by Andriy Drozdyuk, 15 years ago

I think simple argument of "--verbosity=2" will let you know which tests failed.
Failing the test suite makes no sense as it can be run on an automated (nightly) schedule, which makes not sense stopping it just because of one test.

in reply to:  3 comment:4 by Karen Tracey, 15 years ago

Replying to drozzy:

I think simple argument of "--verbosity=2" will let you know which tests failed.

Yes, that will show you what failed (if you happen to notice it before it scrolls off the screen), but you don't get the failure details until the end of the run. If you are running the whole suite that may take a while. If you Ctrl-C the run after you see the first failure you still don't get the failure information so you then have to run just that test again to see if it fails in isolation...plus with this approach you have to be watching the test run like a hawk to catch the first failure. It would be more convenient if there was an option for the run to stop and print the details of the failure once the first failure is encountered. Then you can start the run and periodically check on it to see how it is going -- you don't have to watch it in detail and so long as it keeps running you know you are in good shape.

Failing the test suite makes no sense as it can be run on an automated (nightly) schedule, which makes not sense stopping it just because of one test.

That's why this is proposed as an option. It does certainly make sense when manually running the suite to check for regressions due to a code change you just made.

by Jukka Välimaa, 15 years ago

Attachment: failfast_1.diff added

comment:5 by Jukka Välimaa, 15 years ago

Cc: Jukka Välimaa added
Has patch: set

I've come across situations where something like this would have been useful. When there are 1000+ tests to run to check if anything is broken, getting some info about test failure as soon as it happens can save a lot of time.

comment:6 by robmadole, 15 years ago

Related post on django-developers group (there is a mention about the --failfast option).

comment:7 by Randy Barlow, 15 years ago

Owner: changed from nobody to Randy Barlow
Status: newassigned

w00t. This is very cool.

comment:8 by Randy Barlow, 15 years ago

Cc: randy@… added

Also created #12364, which is similar to this.

comment:9 by Randy Barlow, 15 years ago

I checked out that patch. It looks good. The unit test didn't include an init.py or a models.py, so it wouldn't run. I created a new patch that includes these files. Thanks for the patch!

comment:10 by Randy Barlow, 15 years ago

Cc: Randy Barlow added
Owner: changed from Randy Barlow to Karen Tracey
Status: assignednew
Triage Stage: AcceptedReady for checkin

Hey Karen, this one's ready for commit. Go failfast!

comment:11 by Karen Tracey, 15 years ago

Owner: changed from Karen Tracey to Randy Barlow
Triage Stage: Ready for checkinAccepted

There's a problem in adding a new keyword argument to the test runner: existing alternate test runners won't be expecting the new arg. It would be good to figure out how to add this in a way that won't cause older runners to fail.

comment:12 by Randy Barlow, 15 years ago

Patch needs improvement: set
Status: newassigned

if you have a function foo(arg1, arg2, ...), you can get it's args by foo.func_code.co_varnames. We can use this to find out if our arguments are accepted by the function we are about to pass them to to get around this problem.

by Randy Barlow, 15 years ago

Attachment: ticket_11613.patch added

Alrighty, this patch will do the trick!

comment:13 by Randy Barlow, 15 years ago

Owner: changed from Randy Barlow to Karen Tracey
Patch needs improvement: unset
Status: assignednew
Triage Stage: AcceptedReady for checkin

Hey Karen, now the patch only will use the failfast flag if the test runner will take it. Thanks!

comment:14 by Karen Tracey, 15 years ago

Resolution: fixed
Status: newclosed

(In [11843]) Fixed #11613: Added a failfast option for test running. Thanks jukvalim and Randy Barlow.

comment:15 by Karen Tracey, 15 years ago

(In [11846]) Added tet runner tests mistakenly left out of r11843. Refs #11613.

comment:16 by Jacob, 13 years ago

milestone: 1.2

Milestone 1.2 deleted

Note: See TracTickets for help on using tickets.
Back to Top