Opened 4 years ago

Closed 15 months ago

Last modified 15 months ago

#32602 closed Cleanup/optimization (fixed)

Clarify wording re: parallel testing and test case vs. test case class

Reported by: Chris Jerdonek Owned by: Faishal Manzar
Component: Documentation Version: 3.1
Severity: Normal Keywords:
Cc: Natalia Bidart Triage Stage: Ready for checkin
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: yes UI/UX: no

Description (last modified by Chris Jerdonek)

Currently, there are some places in the docs, code, and code comments that aren't quite clear or right in the way they talk about parallel testing as it relates to test cases versus test case classes.

For example, here in the docs:
https://docs.djangoproject.com/en/3.1/topics/testing/advanced/#django.test.runner.DiscoverRunner
it says:

If there are fewer test cases than configured processes, Django will reduce the number of processes accordingly.

But it should say, "If there are fewer test case classes than configured processes."

Here is similar language elsewhere in the docs:
https://docs.djangoproject.com/en/3.1/ref/django-admin/#envvar-DJANGO_TEST_PROCESSES

Similarly, this code comment:
https://github.com/django/django/blob/41850eec99366a51f98123f7c51e5bc5a8b2798c/django/test/runner.py#L653-L654
should say:

# Since tests are distributed across processes on a per-TestCase
# *class* basis, there's no need for more processes than TestCase
# *classes*.

And also, partition_suite_by_case():
https://github.com/django/django/blob/41850eec99366a51f98123f7c51e5bc5a8b2798c/django/test/runner.py#L845
should really be called something like partition_suite_by_class() or partition_suite_by_test_class() since it groups by type, and the docstring updated accordingly.

I think this is subtle but important because, without being clear, it's easy for people to mistakenly think that the parallel test runner is parallelizing individual test cases, when it's really distributing out the test cases grouped by class.

Change History (22)

comment:1 by Chris Jerdonek, 4 years ago

Description: modified (diff)

comment:2 by Tim Graham, 4 years ago

I disagree with your definitions. In my interpretation of unittest's docs, what you call is "test case class" is a "test case" and what you call a "test case" is a "test method."

comment:3 by Chris Jerdonek, 4 years ago

It says right at the top of the page you linked to:

test case. A test case is the individual unit of testing. It checks for a specific response to a particular set of inputs. unittest provides a base class, TestCase, which may be used to create new test cases.

Its definition of test suite is consistent with that, too:

test suite. A test suite is a collection of test cases, test suites, or both. It is used to aggregate tests that should be executed together.

A test suite is made up of TestCase instances. It's not a collection of test classes.

comment:4 by Chris Jerdonek, 4 years ago

Also, from the "Organizing test code" section:
https://docs.python.org/3/library/unittest.html#organizing-test-code

The basic building blocks of unit testing are test cases — single scenarios that must be set up and checked for correctness. In unittest, test cases are represented by unittest.TestCase instances. To make your own test cases you must write subclasses of TestCase or use FunctionTestCase.

... If the test fails, an exception will be raised with an explanatory message, and unittest will identify the test case as a failure.

Version 0, edited 4 years ago by Chris Jerdonek (next)

comment:5 by Tim Graham, 4 years ago

Maybe I have a faulty understanding of what you're trying to say, but for example:

class TestStringMethods(unittest.TestCase):
    def test_upper(self):
        self.assertEqual('foo'.upper(), 'FOO')

I would call TestStringMethods a test case and test_upper a test method. My impression is that you would call test_upper a test case? (If so, what's a test method?)

Under my definitions, I do think the unittest docs are confusing because a test method seems more like an "individual unit of testing" than a test case.

comment:6 by Chris Jerdonek, 4 years ago

In your example, TestStringMethods is a test case class. unittest creates test cases from test methods, so test cases are in one-to-one correspondence with test methods. (That's why the id of a test case includes the method name.)

For example, if you read the docs for loadTestsFromModule():
https://docs.python.org/3/library/unittest.html#unittest.TestLoader.loadTestsFromModule
it says:

Return a suite of all test cases (my emphasis) contained in the given module. This method searches module for classes derived from TestCase and creates an instance of the class for each test method defined for the class.

You can view test methods as the way to define the test cases that will run.

comment:7 by Tim Graham, 4 years ago

Triage Stage: UnreviewedAccepted

OK. I think there's confusion about this distinction across the ecosystem. For example, The Hitchhiker's Guide to Python says, "Learn your tools and learn how to run a single test or a test case" (where test case seems to mean all the tests in a class).

In Django's comment, I think per-TestCase isn't confusing because the casing implies a class. My guess is that most people read "test case" as pertaining to classes, but it seems that being more precise won't hurt.

comment:8 by Mariusz Felisiak, 21 months ago

Cc: Natalia Bidart added

comment:9 by Natalia Bidart, 20 months ago

When I read the ticket description, my first reaction was the same as Tim's: a test case, to me, is unequivocally the class inheriting from TestCase.

But, after reading the follow up comments and links, I also agree with Tim that being more explicit (which is better than implicit :-)) can benefit future readers.

comment:10 by Natalia Bidart, 20 months ago

Component: Testing frameworkDocumentation
Easy pickings: set

comment:11 by Akbar Ahmad Shah, 20 months ago

Owner: changed from nobody to Akbar Ahmad Shah
Status: newassigned

Not new to django but new to open source, this seems to be a good issue to get started with. Will start looking in developer docs from today.

comment:12 by Mariusz Felisiak, 17 months ago

Owner: Akbar Ahmad Shah removed
Status: assignednew

comment:13 by MandadiMahendharReddy, 17 months ago

Owner: set to Mandadi Mahendhar Reddy
Status: newassigned

comment:14 by MandadiMahendharReddy, 17 months ago

Dear Team, I have commenced working on this First ticket and have made changes to three files respectively.

  1. django/_build/html/topics/testing/advanced.html
  2. django/_build/html/ref/django-admin.html
  3. django/test/runner.py

I am able to view commit changes and commit third file in local i.e. django/test/runner.py. However, I am unable to see files and changes in git status for the first and second file in local.
I am unable to view the first and second file and their changes in git status. I noticed that the docs/_build/ folder is included in the ".gitignore" file. Can you please suggest how to proceed with this?

Last edited 17 months ago by MandadiMahendharReddy (previous) (diff)

comment:15 by Tim Graham, 17 months ago

You should edit the documentation in the docs folder, not in _build. The _build` directory contains the html files generated from the source txt files.

in reply to:  15 comment:16 by MandadiMahendharReddy, 17 months ago

Thank you Replying to Tim Graham:

You should edit the documentation in the docs folder, not in _build. The _build` directory contains the html files generated from the source txt files.

comment:17 by Mariusz Felisiak, 17 months ago

Has patch: set
Patch needs improvement: set

comment:18 by Faishal Manzar, 15 months ago

Owner: changed from Mandadi Mahendhar Reddy to Faishal Manzar

comment:19 by Mariusz Felisiak, 15 months ago

comment:20 by Mariusz Felisiak, 15 months ago

Patch needs improvement: unset
Triage Stage: AcceptedReady for checkin

comment:21 by Mariusz Felisiak <felisiak.mariusz@…>, 15 months ago

Resolution: fixed
Status: assignedclosed

In f4e72e6:

Fixed #32602 -- Clarified wording of TestCase class.

comment:22 by Mariusz Felisiak <felisiak.mariusz@…>, 15 months ago

In 25a61463:

[5.0.x] Fixed #32602 -- Clarified wording of TestCase class.

Backport of f4e72e6523e6968d9628dfbff914ab57dbf19e6b from main

Note: See TracTickets for help on using tickets.
Back to Top