Clarify wording re: parallel testing and test case vs. test case class

[Chris Jerdonek]

Currently, there are some places in the docs, code, and code comments that aren't quite clear or right in the way they talk about parallel testing as it relates to test cases versus test case classes.

For example, here in the docs:
​https://docs.djangoproject.com/en/3.1/topics/testing/advanced/#django.test.runner.DiscoverRunner
it says:

If there are fewer test cases than configured processes, Django will reduce the number of processes accordingly.

But it should say, "If there are fewer test case classes than configured processes."

Here is similar language elsewhere in the docs:
​https://docs.djangoproject.com/en/3.1/ref/django-admin/#envvar-DJANGO_TEST_PROCESSES

Similarly, this code comment:
​https://github.com/django/django/blob/41850eec99366a51f98123f7c51e5bc5a8b2798c/django/test/runner.py#L653-L654
should say:

# Since tests are distributed across processes on a per-TestCase
# *class* basis, there's no need for more processes than TestCase
# *classes*.

And also, partition_suite_by_case():
​https://github.com/django/django/blob/41850eec99366a51f98123f7c51e5bc5a8b2798c/django/test/runner.py#L845
should really be called something like partition_suite_by_class() or partition_suite_by_test_class() since it groups by type, and the docstring updated accordingly.

I think this is subtle but important because, without being clear, it's easy for people to mistakenly think that the parallel test runner is parallelizing individual test cases, when it's really distributing out the test cases grouped by class.

[Tim Graham]

I disagree with your definitions. In my interpretation of ​unittest's docs, what you call is "test case class" is a "test case" and what you call a "test case" is a "test method."

[Chris Jerdonek]

It says right at the top of the page you linked to:

test case. A test case is the individual unit of testing. It checks for a specific response to a particular set of inputs. unittest provides a base class, TestCase, which may be used to create new test cases.

Its definition of test suite is consistent with that, too:

test suite. A test suite is a collection of test cases, test suites, or both. It is used to aggregate tests that should be executed together.

A test suite is made up of TestCase instances. It's not a collection of test classes.

[Chris Jerdonek]

Also, from the "Organizing test code" section:
​https://docs.python.org/3/library/unittest.html#organizing-test-code

The basic building blocks of unit testing are test cases — single scenarios that must be set up and checked for correctness. In unittest, test cases are represented by unittest.TestCase instances. To make your own test cases you must write subclasses of TestCase or use FunctionTestCase.

... If the test fails, an exception will be raised with an explanatory message, and unittest will identify the test case as a failure.

Another place is if you look under TestCase.id():
​https://docs.python.org/3/library/unittest.html#unittest.TestCase.id

Return a string identifying the specific test case. This is usually the full name of the test method, including the module and class name.

[Tim Graham]

Maybe I have a faulty understanding of what you're trying to say, but for example:

class TestStringMethods(unittest.TestCase):
    def test_upper(self):
        self.assertEqual('foo'.upper(), 'FOO')

I would call TestStringMethods a test case and test_upper a test method. My impression is that you would call test_upper a test case? (If so, what's a test method?)

Under my definitions, I do think the unittest docs are confusing because a test method seems more like an "individual unit of testing" than a test case.

[Chris Jerdonek]

In your example, TestStringMethods is a test case class. unittest creates test cases from test methods, so test cases are in one-to-one correspondence with test methods. (That's why the id of a test case includes the method name.)

For example, if you read the docs for loadTestsFromModule():
​https://docs.python.org/3/library/unittest.html#unittest.TestLoader.loadTestsFromModule
it says:

Return a suite of all test cases (my emphasis) contained in the given module. This method searches module for classes derived from TestCase and creates an instance of the class for each test method defined for the class.

You can view test methods as the way to define the test cases that will run.

[Tim Graham]

OK. I think there's confusion about this distinction across the ecosystem. For example, ​The Hitchhiker's Guide to Python says, "Learn your tools and learn how to run a single test or a test case" (where test case seems to mean all the tests in a class).

In Django's comment, I think per-TestCase isn't confusing because the casing implies a class. My guess is that most people read "test case" as pertaining to classes, but it seems that being more precise won't hurt.

[Natalia Bidart]

When I read the ticket description, my first reaction was the same as Tim's: a test case, to me, is unequivocally the class inheriting from TestCase.

But, after reading the follow up comments and links, I also agree with Tim that being more explicit (which is better than implicit :-)) can benefit future readers.

[Akbar Ahmad Shah]

Not new to django but new to open source, this seems to be a good issue to get started with. Will start looking in developer docs from today.

[MandadiMahendharReddy]

Dear Team, I have commenced working on this First ticket and have made changes to three files respectively.

1. django/_build/html/topics/testing/advanced.html
2. django/_build/html/ref/django-admin.html
3. django/test/runner.py

I am able to view commit changes and commit third file in local i.e. django/test/runner.py. However, I am unable to see files and changes in git status for the first and second file in local.
I am unable to view the first and second file and their changes in git status. I noticed that the docs/_build/ folder is included in the ".gitignore" file. Can you please suggest how to proceed with this?

[Tim Graham]

You should edit the documentation in the docs folder, not in _build. The _build` directory contains the html files generated from the source txt files.

[MandadiMahendharReddy]

Thank you Replying to Tim Graham:

You should edit the documentation in the docs folder, not in _build. The _build` directory contains the html files generated from the source txt files.

[Mariusz Felisiak]

​PR

[Mariusz Felisiak]

​New PR

[Mariusz Felisiak <felisiak.mariusz@…>]

In f4e72e6:

Fixed #32602 -- Clarified wording of TestCase class.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 25a61463:

[5.0.x] Fixed #32602 -- Clarified wording of TestCase class.

Backport of f4e72e6523e6968d9628dfbff914ab57dbf19e6b from main