Using a custom test suite, django.test.TestCase-based tests either fail silently (DEBUG = True) or fail with a generic error message (DEBUG = False)

[Chris Wagner <cw264701@…>]

I made a custom test script since I wanted to have tests in more places than just tests.py and models.py. In the script, I set up Django's "stuff" using

django.test.utils.setup_test_environment()
django.test.utils.create_test_db(verbosity=0, autoclobber=True)

just before auto-detecting and running all tests using the nose testing tool/framework. Under this configuration, I noticed that my django.test.TestCase-based tests could fail, but there would be no indication of failure when running them.

I looked at Django's code, to see how it ran the same tests, and the only noticeable difference that I saw, was, Django's test runner would set the "DEBUG" setting to False before initializing the test environment. I added that to my script, and now the result is slightly better, as I at least see some indication of failure, but the error message is a generic one ("TemplateDoesNotExist: 500.html") so I cannot easily determine the reason for failure.

Here's an example traceback:

ERROR: test_downloading_release_as_zip (tests.TestStuff)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chris/projects/easyweaze-py/easyweaze/test/web-level-tests/tests.py", line 160, in test_downloading_release_as_zip
    response = self.client.get("/members/zip_release/%d" % r.id)
  File "/usr/local/lib/python2.5/site-packages/django/test/client.py", line 203, in get
    return self.request(**r)
  File "/usr/local/lib/python2.5/site-packages/django/test/client.py", line 168, in request
    response = self.handler(environ)
  File "/usr/local/lib/python2.5/site-packages/django/test/client.py", line 38, in __call__
    response = self.get_response(request)
  File "/usr/local/lib/python2.5/site-packages/django/core/handlers/base.py", line 126, in get_response
    return callback(request, **param_dict)
  File "/usr/local/lib/python2.5/site-packages/django/views/defaults.py", line 88, in server_error
    t = loader.get_template(template_name) # You need to create a 500.html template.
  File "/usr/local/lib/python2.5/site-packages/django/template/loader.py", line 79, in get_template
    source, origin = find_template_source(template_name)
  File "/usr/local/lib/python2.5/site-packages/django/template/loader.py", line 72, in find_template_source
    raise TemplateDoesNotExist, name
TemplateDoesNotExist: 500.html

Let me know if you'd like to see my test script or anything. Thanks for any input on this.

[James Bennett]

The problem here is that running Django with DEBUG=False requires you to have a template named 500.html to display when an internal error occurs (and, correspondingly, 404.html for URLs that don't match), or to override handler500 and/or handler404 in the root URLConf; not having it swallows the original exception with a TemplateDoesNotExist, as you've found.

This seems to trip some folks up, and I'm not sure what the best course is; documenting it clearly is one option, and we should probably do that (though the traceback does tell you what you need to do if you read through it), but I don't know if we should also be including default templates or, if we do, how we should handle that.

[James Bennett]

Changing this to "documentation" for now since we need that either way, and since this is not specific to any part of Django -- doesn't matter if you're unit testing or not, you can trip this if you don't have a 500.html.

[Chris Wagner <cw264701@…>]

A) That still doesn't explain why my tests fail silently when DEBUG == True.
B) Shouldn't the test client have a special mechanism for catching errors that occur within a view? Everyone is going to have a failing test now and then, and if you have to dig beneath the extra rubbish every time -- well, that sure seems wasteful. And, in my case, I was taking a sort of test-driven development approach; I kinda expected the test to fail (although I was not sure of exactly where or how it would fail in this case).

[James Bennett]

Depends on whether your test is expecting to see an exception raised or is checking for an HTTP status code; regardless of DEBUG, Django does its best to catch any exception thrown during a request/response cycle. So if you're using the text client and you want to verify that something failed, check for HTTP 500 in the response. If there's some particular part of the view that you want to test for failure, using the client and doing a full request/response cycle may not be the best option; you can always pass a mock HttpRequest object and other arguments to the view directly, and look to see if it returns an HttpResponse or throws an exception.

[Jacob]

I actually don't think this *is* a documentation issue. Under the built-in test suite, a failing view called from a TestClient will re-raise the original exception (which is what Chris wants). I'm changing the component back to unit tests so that Russ will see it.

[Chris Wagner <cw264701@…>]

Coincidentally, I was writing this up just as Jacob posted. I'll go ahead and post it anyway....

---

According to the Django documentation (at ​http://www.djangoproject.com/documentation/testing/#test-client):

Exceptions

If you point the Test Client at a view that raises an exception, that exception will be visible in the test case. You can then use a standard try...catch block, or unittest.TestCase.assertRaises() to test for exceptions.

The only exceptions that are not visible in a Test Case are Http404, PermissionDenied and SystemExit. Django catches these exceptions internally and converts them into the appropriate HTTP responses codes.

Doesn't that contradict with what I'm seeing? I assume that, that means I should see any exception raised from one of my "views" (except for, of course, Http404, PermissionDenied and SystemExit exceptions), regardless of whether I have DEBUG set or not. Furthermore, I shouldn't see a TemplateDoesNotExist exception, because, if Django is already trying to render an error template, then it must have already captured the exception that was raised within my view.

[Russell Keith-Magee]

As long as your view is raising a normal exception (e.g., KeyError), you _should_ be able to catch the exception in a test case as long as settings.DEBUG = False.

I've tried to replicate your problem, but I haven't had any luck. Can you provide an example script that exhibits this problem?

[James Bennett]

Replying to jacob:

I actually don't think this *is* a documentation issue. Under the built-in test suite, a failing view called from a TestClient will re-raise the original exception (which is what Chris wants). I'm changing the component back to unit tests so that Russ will see it.

OK. I still think there's a related documentation issue here (probably worth a separate ticket, though) from the fact that we never warn people that they'll need 500.html and 404.html; as soon as they flip DEBUG of they'll start seeing TemplateDoesNotExist in places they never expected it...

[Chris Wagner <cw264701@…>]

I was going to upload an attachment -- a zip archive containing a Django project that seems to portray the problem -- but uploading attachments doesn't seem to be working (?). Let me know if you'd like me to email the archive to you, or something.

I didn't do anything special to reproduce the problem. It was actually less work than I thought it would be; I thought I would have to make a custom test-runner script, but I didn't even need to take it that far. I'm not sure what I'm doing wrong. I am running from Django's trunk and am up-to-date (at revision 5855).

[Russell Keith-Magee]

Replying to Chris Wagner <cw264701@ohiou.edu>:

I was going to upload an attachment -- a zip archive containing a Django project that seems to portray the problem -- but uploading attachments doesn't seem to be working (?). Let me know if you'd like me to email the archive to you, or something.

Sure. Mail a sample project to me privately at freakboy3742@….

[Russell Keith-Magee]

Ok; found the problem. The issue is caused by not having a 500.html template. The template isn't actually used - the underlying exception is caught and rethrown, but the template must exist. If it doesn't exist, the request handler throws another exception, which is the error you are reporting. My test environment (and the Django test environment) both provide a 500.html template, so I didn't see the problem.

Once I found the problem, the fix was pretty trivial. I just have to do some final testing; it should be in trunk shortly.

[Russell Keith-Magee]

(In [6023]) Fixed #4457 -- Corrected the handling of exceptions in the test client when the 500.html template is not available. Thanks to Chris Wager <cw264701@…> for his help in tracking down this problem.

[Chris Wagner <cw264701@…>]

Seems to be working for me. Thank you. :)