#32602 closed Cleanup/optimization (fixed)
Clarify wording re: parallel testing and test case vs. test case class
Reported by: | Chris Jerdonek | Owned by: | Faishal Manzar |
---|---|---|---|
Component: | Documentation | Version: | 3.1 |
Severity: | Normal | Keywords: | |
Cc: | Natalia Bidart | Triage Stage: | Ready for checkin |
Has patch: | yes | Needs documentation: | no |
Needs tests: | no | Patch needs improvement: | no |
Easy pickings: | yes | UI/UX: | no |
Description (last modified by )
Currently, there are some places in the docs, code, and code comments that aren't quite clear or right in the way they talk about parallel testing as it relates to test cases versus test case classes.
For example, here in the docs:
https://docs.djangoproject.com/en/3.1/topics/testing/advanced/#django.test.runner.DiscoverRunner
it says:
If there are fewer test cases than configured processes, Django will reduce the number of processes accordingly.
But it should say, "If there are fewer test case classes than configured processes."
Here is similar language elsewhere in the docs:
https://docs.djangoproject.com/en/3.1/ref/django-admin/#envvar-DJANGO_TEST_PROCESSES
Similarly, this code comment:
https://github.com/django/django/blob/41850eec99366a51f98123f7c51e5bc5a8b2798c/django/test/runner.py#L653-L654
should say:
# Since tests are distributed across processes on a per-TestCase # *class* basis, there's no need for more processes than TestCase # *classes*.
And also, partition_suite_by_case()
:
https://github.com/django/django/blob/41850eec99366a51f98123f7c51e5bc5a8b2798c/django/test/runner.py#L845
should really be called something like partition_suite_by_class()
or partition_suite_by_test_class()
since it groups by type
, and the docstring updated accordingly.
I think this is subtle but important because, without being clear, it's easy for people to mistakenly think that the parallel test runner is parallelizing individual test cases, when it's really distributing out the test cases grouped by class.
Change History (22)
comment:1 by , 4 years ago
Description: | modified (diff) |
---|
comment:2 by , 4 years ago
comment:3 by , 4 years ago
It says right at the top of the page you linked to:
test case. A test case is the individual unit of testing. It checks for a specific response to a particular set of inputs.
unittest
provides a base class,TestCase
, which may be used to create new test cases.
Its definition of test suite is consistent with that, too:
test suite. A test suite is a collection of test cases, test suites, or both. It is used to aggregate tests that should be executed together.
A test suite is made up of TestCase
instances. It's not a collection of test classes.
comment:4 by , 4 years ago
Also, from the "Organizing test code" section:
https://docs.python.org/3/library/unittest.html#organizing-test-code
The basic building blocks of unit testing are test cases — single scenarios that must be set up and checked for correctness. In
unittest
, test cases are represented byunittest.TestCase
instances. To make your own test cases you must write subclasses of TestCase or use FunctionTestCase.
... If the test fails, an exception will be raised with an explanatory message, and
unittest
will identify the test case as a failure.
comment:5 by , 4 years ago
Maybe I have a faulty understanding of what you're trying to say, but for example:
class TestStringMethods(unittest.TestCase): def test_upper(self): self.assertEqual('foo'.upper(), 'FOO')
I would call TestStringMethods
a test case and test_upper
a test method. My impression is that you would call test_upper
a test case? (If so, what's a test method?)
Under my definitions, I do think the unittest docs are confusing because a test method seems more like an "individual unit of testing" than a test case.
comment:6 by , 4 years ago
In your example, TestStringMethods
is a test case class. unittest
creates test cases from test methods, so test cases are in one-to-one correspondence with test methods. (That's why the id of a test case includes the method name.)
For example, if you read the docs for loadTestsFromModule()
:
https://docs.python.org/3/library/unittest.html#unittest.TestLoader.loadTestsFromModule
it says:
Return a suite of all test cases (my emphasis) contained in the given module. This method searches module for classes derived from
TestCase
and creates an instance of the class for each test method defined for the class.
You can view test methods as the way to define the test cases that will run.
comment:7 by , 4 years ago
Triage Stage: | Unreviewed → Accepted |
---|
OK. I think there's confusion about this distinction across the ecosystem. For example, The Hitchhiker's Guide to Python says, "Learn your tools and learn how to run a single test or a test case" (where test case seems to mean all the tests in a class).
In Django's comment, I think per-TestCase
isn't confusing because the casing implies a class. My guess is that most people read "test case" as pertaining to classes, but it seems that being more precise won't hurt.
comment:8 by , 21 months ago
Cc: | added |
---|
comment:9 by , 20 months ago
When I read the ticket description, my first reaction was the same as Tim's: a test case, to me, is unequivocally the class inheriting from TestCase
.
But, after reading the follow up comments and links, I also agree with Tim that being more explicit (which is better than implicit :-)) can benefit future readers.
comment:10 by , 20 months ago
Component: | Testing framework → Documentation |
---|---|
Easy pickings: | set |
comment:11 by , 20 months ago
Owner: | changed from | to
---|---|
Status: | new → assigned |
Not new to django but new to open source, this seems to be a good issue to get started with. Will start looking in developer docs from today.
comment:12 by , 17 months ago
Owner: | removed |
---|---|
Status: | assigned → new |
comment:13 by , 17 months ago
Owner: | set to |
---|---|
Status: | new → assigned |
comment:14 by , 17 months ago
Dear Team, I have commenced working on this First ticket and have made changes to three files respectively.
- django/_build/html/topics/testing/advanced.html
- django/_build/html/ref/django-admin.html
- django/test/runner.py
I am able to view commit changes and commit third file in local i.e. django/test/runner.py. However, I am unable to see files and changes in git status for the first and second file in local.
I am unable to view the first and second file and their changes in git status. I noticed that the docs/_build/ folder is included in the ".gitignore" file. Can you please suggest how to proceed with this?
follow-up: 16 comment:15 by , 17 months ago
You should edit the documentation in the docs
folder, not in _build. The
_build` directory contains the html files generated from the source txt files.
comment:16 by , 17 months ago
Thank you Replying to Tim Graham:
You should edit the documentation in the
docs
folder, not in_build. The
_build` directory contains the html files generated from the source txt files.
comment:18 by , 15 months ago
Owner: | changed from | to
---|
comment:20 by , 15 months ago
Patch needs improvement: | unset |
---|---|
Triage Stage: | Accepted → Ready for checkin |
I disagree with your definitions. In my interpretation of unittest's docs, what you call is "test case class" is a "test case" and what you call a "test case" is a "test method."