Add a test suite for core functionality

[nelson@…]

Congratulations on the (slightly earlier than expected) launch of Django. It looks very cool, and the site is nicely put together. Just what Python needs.

On initial evaluation I was surprised to find almost no test code. There are a few tests in one package, but no comprehensive unit test suite. Automated testing seems particularly important for the fiddly bits of code like your database caching system. It's also really valuable to have unit tests in an open source project; you're much less likely to get broken code if you require the tests pass.

Adding unit tests after the fact feels like a lot of work, but the earlier you get a good test framework in place the easier it will be. And the sooner it pays off.

Thanks for listening to my suggestion!

Nelson

[pb@…]

I think this is an excellent suggestion. I was surprised by the absence of tests as well (not shocked, you know, just surprised, given the quality of the project). Constructing the test suite is something that you could parcel out to your rapidly growing cadre of developer-users, if you were so inclined.

[Jason Huggins]

Actually, the lack of a unit test framework at this time is an opportunity in disguise... Perhaps it's time for a "real" project like Django to start using PyTest... With a mostly clean slate, there's not a lot of worry for moving away from the unittest module. Read more about PyTest here: ​http://agiletesting.blogspot.com/2005/01/python-unit-testing-part-3-pytest-tool.html

And regarding browser UI testing... Well, I'm only a little biased here... I suggest selenium (selenium.thoughtworks.com) :-) Selenium's 'sweet spot' is testing cross-browser, cross-platform bugs or features that depend on JavaScript (hello, AJAX!), but is also handy for 'end-to-end' and 'smoketest' tests. (i.e. open up ​http://localhost:8000/admin in firefox and make sure you don't get a stack trace.) I *think* the Plone guys like it, too. :-) (​http://plone.org/products/plone/roadmap/100)

[ianb@…]

Well, several real projects use py.test already (SQLObject and Paste among them), so there's nothing risky about it. unittest is very underfeatured, so you have to write all sorts of stuff to make it work decently in a large project, so the more "mature" choice is really a more risky proposition (or more of a pain in the butt, if you just punt on those extra features).

paste.tests.fixture offers a simple framework for acceptance tests (that kind of thing isn't really a unit test). The only really paste-specific aspect is the configuration loading, and perhaps suppressing exception catching; everything else is just WSGI. For application acceptance testing these are convenient; for framework acceptance tests they aren't as important (as it should probably be configuration-neutral, or you'll give explicit configuration as part of the test itself).

I think the advantage over Selenium is that you can run the tests more easily, with less configuration, and with less interaction. Assuming you aren't testing Javascript, of course. You can test apps that use Javascript still, but you'll generally be pretending the Javascript does its thing, you won't detect errors in that portion.

Another complementary option is doctesting your tutorials as a form of acceptance test. paste.tests.doctest_webapp has some support code to make that easier, though in the end it's just a doctest, nothing too fancy.

Unit tests themselves should be ad hoc. They shouldn't rely on configuration or state or anything. Well, not as true for database-related unit tests, but you'll have to make your own little framework for setup and teardown of those.

The relevant URLs for Paste:

• Doc on paste.tests.fixture: ​http://pythonpaste.org/docs/testing-applications.html
• Discussion: ​http://blog.ianbicking.org/functional-testing-in-paste.html
• The code: ​http://svn.pythonpaste.org/Paste/trunk/paste/tests/fixture.py
• Example of doc that uses doctest_webapp: ​http://svn.pythonpaste.org/Paste/trunk/docs/TodoTutorial.txt
• The code: ​http://svn.pythonpaste.org/Paste/trunk/paste/tests/doctest_webapp.py

[Simon Willison]

"Constructing the test suite is something that you could parcel out to your rapidly growing cadre of developer-users, if you were so inclined."

That's an excellent idea. The number of people willing to contribute to Django appears to be growing at an impressive rate, and if we can work out an intelligent way to co-ordinate the construction of a unit test suite it would really help the project. The people who worked on it would benefit by developing a much better understanding of the system, which in turn would make it easier for them to spot parts of the design that could be improved and make smart suggestions about it.

Sounds like a win-win to me.

[Adrian Holovaty]

#48 is a duplicate and has been closed.

[Adrian Holovaty]

As of [336], there's a unit-test framework for models. We'll keep expanding that framework to include more things than just models.

[nelson@…]

Great!