Support unittest.TestCase.debug() method.

[Pavel Savchenko]

Introduction

The documentation for ​unittest.TestCase.debug states that invoking it would:

Run the test without collecting the result. This allows exceptions raised by the test to be propagated to the caller, and can be used to support running tests under a debugger.

In essence, debug() invokes the test method same as ​TestCase.run() does, without the added functionality of catching exceptions and related cruft.

Side effects

A recent change in pytest (see references below) brought to our attention the fact that Django does not implement a debug method, and performs important stuff in TestCase.__call__ (besides calling run()), such as calling _pre_startup and _post_teardown methods.

Thus, since debug() is unimplemented, running debug on a django TestCase (for example for debugging purposes), results unintended behavior, such as certain methods not being called. Most importantly _fixture_callback and _fixture_teardown which also perform database rollback (or call flush).

References:
​https://github.com/pytest-dev/pytest/pull/1890
​https://github.com/pytest-dev/pytest/issues/1932
​https://github.com/pytest-dev/pytest-django/pull/406
​https://github.com/python/cpython/blob/master/Lib/unittest/case.py#L651-L658

[Tim Graham]

Would this have benefit outside of the usage of third-party test runners such as pytest? If so, what does it look like?

[Pavel Savchenko]

Well, not that I know of yet, but basically, any test runner that allows unittests may assume "debug" can be used for exception debugging; and right now this cannot be done without issues when using Django's SimpleTestCase derivatives.

Regardless, the patch should be fairly simple 10 line method and I can submit that. It's invoking the test that I'm having trouble with, how to invoke "debug" as part of the regular test suite, maybe we can use call_command...

[Pavel Savchenko]

As for other test runners, here's now nose2 invokes a debugger (IIRC pytest does something similar):
​https://github.com/nose-devs/nose2/blob/master/nose2/plugins/debugger.py#L41-L60

This means that the post_mortem is invoked only after the exception was already caught in SimpleTestCase.__call__ and, for example, the database was already rolled back. The test frameworks simple have no real way to examine the state of the application at the time of the error (but only "after", when it may be too late).

[Pavel Savchenko]

Added test case (demonstrating usage expectations) and resolution in ​PR 7436

[Tim Graham]

I think the test could be a bit cleaner. I sent some edits to the PR, though I broke some things too.

[Pavel Savchenko]

Replying to Asif Saifuddin Auvi:

Patch was improved as per Tim Martin's notes ​in commit 7ec5fae. I added a sketch for release note mention in ​this comment.

Is there other documentation you had in mind?

[Mariusz Felisiak <felisiak.mariusz@…>]

In dc8cd2fe:

Refs #27391 -- Added more tests for SimpleTestCase.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 1711c509:

Fixed #27391 -- Implemented SimpleTestCase.debug().

debug() should bubbled up exceptions if occurring in test, but behave
the same as run() when no exceptions occurred.