Incomplete support for app-level testing

[Masklinn]

Django and its community have a pretty strong focus on the concept of reusable applications: applications as pluggable behavior blocks, independent from one another and from projects. And the test framework documentation and setup emphasize that: the documentation page is called ​Testing Django applications and the test themselves are saved at the application level not at the project level.

Yet Django provides no provision whatsoever for running tests from an application: a project (or at least a settings and a urls file) is required, and barring fairly weird setups (global settings and urls modules) there is no way to create a new app, write a test and run it just like that. Which means applications aren't as self-contained as they could be, and can't easily be tested "in a void" (independently of everything else).

[Russell Keith-Magee]

I concur 100%. The distinction between integration testing of new apps, and system testing of a specific app is a persistent problem, and the cause of more than one ticket in the Django system (e.g., #10976, #10755, and others). This is a problem for which we should have a solid answer.

[Manoj Govindan <egmanoj@…>]

Replying to russellm:

I concur 100%. The distinction between integration testing of new apps, and system testing of a specific app is a persistent problem, and the cause of more than one ticket in the Django system (e.g., #10976, #10755, and others). This is a problem for which we should have a solid answer.

Can I make a wish while we are on the topic? ;) Django allows me to write test suites (unittest.TestSuite objects) but won't allow me to run them. Adding one more level of granularity would help users to run any of Application/Test Suite/Test Class/Test Method tests. This would also make django's test running mechanism more similar to unittest's.

If you think this should be a separate ticket I can add one.

[Russell Keith-Magee]

Replying to Manoj Govindan <egmanoj@gmail.com>:

Replying to russellm:
Can I make a wish while we are on the topic? ;) Django allows me to write test suites (unittest.TestSuite objects) but won't allow me to run them. Adding one more level of granularity would help users to run any of Application/Test Suite/Test Class/Test Method tests. This would also make django's test running mechanism more similar to unittest's.

What you are proposing isn't a stylistic thing or a particular way that the original proposal could be interpreted - it's a completely new idea. If you have a separate feature request, open a separate ticket.

[Manoj Govindan <egmanoj@…>]

What you are proposing isn't a stylistic thing or a particular way that the original proposal could be interpreted - it's a completely new idea. If you have a separate feature request, open a separate ticket.

​Done.

[Aymeric Augustin]

Change UI/UX from NULL to False.

[Aymeric Augustin]

Change Easy pickings from NULL to False.

[Raphael Kimmig]

This ticket is really old, but I'd still love to see some progress here.
I've been thinking about this a bit lately and below are my thoughts so far.
We probably don't need to write a lot of code here, but rather add some
documentation to the advanced tutorial as well as the advanced testing
documentation.

From what I can see there are a few common things that should be documented.

1. How to run tests in a stand alone application (this requires settings).
2. How to manage settings for running tests in a stand alone application.
3. How to set up views/urls/middleware ... for tests.
4. How to add models specifically for tests inside applications.

For 1, 2 and 3. there are several solutions, to get started I'd probably describe
how to use django-admin test to run the tests with a custom test_settings.py
file.

For 4. there is a simple solution if the app doesn't provide any models itself.
In that case one can simply disable migrations for the app in the test settings.
Ideally I'd like to try implementing a @test_model decorator as suggested in
https://code.djangoproject.com/ticket/7835 which would also solve the general
case.

I'd love to have the documentation for reusable apps contain (directly, or via links
to the testing docs) all the information required to not only create an app, but also
make sure it has working tests.

I'll try to expand the relevant docs over the next few days. If anyone has some
input on the matter I'd be glad to hear it.

Did I miss any obvious work in that area? Maybe even somewhere in the documentation?
Do you have any suggestions on how specific aspects should be solved?

/e
After looking some more I've found the section in the advanced testing docs that covers
most of this already :-)

​https://docs.djangoproject.com/en/2.1/topics/testing/advanced/#using-the-django-test-runner-to-test-reusable-applications

I think the one thing that is really missing is some mention of that in the reusable apps introduction.
Also the part in advanced testing that covers how to use models only for tests could probably be worded
a bit more clearly.

[Raphael Kimmig]

I've added documentation to the resuable apps intro. ​PR

One possible way to improve this would be to add a --reusable flag to the test management command so that one could run the tests with django-admin test --reusable or something along those lines instead of copy & pasting a runtests.py file. Any thoughts?

[Simon Charette]

Is there a reason why python -m django test --settings=tests.settings tests (given tests.test_settings is renamed to settings) wouldn't work?

That's the pattern I use for all the third party apps I maintain and it seems to be working just fine.

e.g.

• ​https://github.com/charettes/django-seal/blob/36dde326783d51d1f45348fcb18cfd1418e93881/tox.ini#L20
• ​https://github.com/charettes/django-seal/tree/36dde326783d51d1f45348fcb18cfd1418e93881/tests

I'm not sure I'm following why a --reusable flag is required.

[Raphael Kimmig]

No good reason at all, that looks much nicer actually :-)

I'll update the PR and I'll change the advanced testing part about reusable applications as well,
unless you see a good reason to suggest using a runtests.py file there.

[Simon Charette]

I don't see a reason for shipping a runtests.py file with minimal configuration. The one we use for Django's testsuite happens to exist because we need to perform quite a lot of tweaks before running the tests which isn't the case for most Django third party apps.

[Carlton Gibson]

Patch looks good. Comments on PR (Summary: Just some rewording to the "Advanced testing topics" section linked from the new examples needed.)

[Raphael Kimmig]

I forgot about this completely. I'll just fix up the branch and add in the feedback.

[Mariusz Felisiak]

Replying to Raphael Kimmig:

I forgot about this completely. I'll just fix up the branch and add in the feedback.

Great, thanks!

[Raphael Kimmig]

I wasn't able to change the target branch on the old PR to main, so here is a new one incorporating feedback from the old PR

​https://github.com/django/django/pull/18018