5th tutorial on turning Polls into a reusable app

[stumbles]

Katie Miller and I have been sprinting to create a tutorial to cover:

• reusable apps
• turning Polls into a reusable app
• Python packaging
• publishing your package

First draft attached.

[Aymeric Augustin]

This is a good addition to the docs.

I skimmed through the patch and I suggest the following improvements:

• Remove the TODO (to the best of my knowledge, the answer is "it can't be done")
• Warn before installing the app globally, since it can be unsettling for beginners to remove it directly from site-packages, and explain how to uninstall it.
• Remove the "Coming soon" section — those additional parts have been "coming soon" for years and nobody's about to write them.

[stumbles]

First draft of tutorial 5

[stumbles]

I've now added an updated patch to include the changes recommended by aaugustin. Sorry for the long delay.

[stumbles]

Extended README.txt, include "docs" in package and update tutorial index links.

[stumbles]

Minor readability tweak for intro.

[stumbles]

Reworded first section on reusable apps.

[stumbles]

Katie fixed some typos.

[stumbles]

Closing bracked added previously was not quite in the right place.

[stumbles]

Updated project layout for Django 1.4

[stumbles]

Removed some rougue tabs

[stumbles]

I've now updated the project layout of this tutorial to match the changes in Django 1.4. I also repeated the previous tutorials again myself which helped clarify a few issues, such as where the project-wide templates directory will be after completing tutorials 1-4.

[Mikhail Korobov]

Good work! Some notes:

• Templates won't be included in distribution (they should be added to package_data in setup.py);
• but I think it is bad idea to include templates in polls app because they usually tend to be too site-specific and overriding bundled templates is not always obvious;
• a couple of simple tests are must for a reusable app.

[stumbles]

Update tutorial to include templates directory via MANIFEST.in.

[stumbles]

I've added necessary instructions to include templates directory in the package. For illustrative purposes, I think it is useful to show the reader how to include the templates.

The alternative would be to:

• remove the templates directory
• remove the "Completing your reusable app" section of the tutorial
• explain to the reader that people may find the templates too site-specific and difficult to override and that it may be better not to include them at all
• get the reader to add instructions to the package on how to write templates

Plus it's no longer self-contained, so not quite as satisfying.

Sure it may not be best practise to include templates, but to teach the basic concepts, I think this is an acceptable compromise.

[stumbles]

Regarding tests, a doc test or unit test on models.Poll.was_published_today might be a useful example. What other tests do you think should be added?

[stumbles]

Update bullet point formatting.

[stumbles]

Just spoken to Russell at PyCon Australia sprints. He suggests that testing should be a separate tutorial of its own, rather than extending this one.

[Tim Graham]

Hey Ben & Katie,

This has sat for a long while and I'd like to try to get this committed! Couple of comments:

1. When I move the templates polls directory into the app folder, I had to add '/dir/to/mysite/' to settings.TEMPLATE_DIRS, previously I had only /path/to/mysite/mytemplates/ there.

2. The django-polls/docs isn't included in the package even though it's included in MANIFEST.in unless it has a file in it. Perhaps a note could be added about that.

3. Rather than suggest possible directories for dist/site-packages, use the command noted here: ​https://docs.djangoproject.com/en/dev/topics/install/#remove-any-old-versions-of-django

Nice work!

Tim

[Tim Graham]

I'll add that for dist-packages the command in the link I noted above output "/usr/lib/python2.7/dist-packages" while polls was actually installed in "/usr/local/lib/python2.7". I discovered the actual location by running this command:

>>> import polls
>>> polls
<module 'polls' from '/usr/local/lib/python2.7/dist-packages/polls/__init__.pyc'>

So perhaps that should be recommended as an alternative?

[stumbles]

Note about empty docs dir and easy way to find location of installed package

[stumbles]

Thanks for your notes Tim. Also looking forward to getting this in!

Regarding 1: I've just run through the dev version tutorials 1-4 and didn't find I needed to modify settings.TEMPLATE_DIRS. If you have 'django.template.loaders.app_directories.Loader', in your settings.TEMPLATE_LOADERS (the default), the templates should be found by Django when moved inside polls/templates/polls. Have I missed something?

Regarding 2: I've added a note about the docs directory not being included if empty, thanks.

Regarding 3: I've used the example you mentioned in comment 11 to describe how to find package files to delete.

[stumbles]

Also, I just noticed that while the templates directory is included in the package, it's not installed when I run python setup.py install.

After some hunting, I learnt that one way to ensure the templates are installed is to use packages=setuptools.find_packages() and include_package_data=True in my setup.py file as follows:

from distutils.core import setup
from setuptools import find_packages

setup(
    name='django-polls',
    version='0.1',
    packages=find_packages(),
    license='',
    long_description=open('README.txt').read(),
    url='http://www.example.com/',
    author='Your Name',
    author_email='yourname@example.com',
    include_package_data=True,
)

Is this the approach you would recommend? I'm not very experienced with packaging and noticed that the install behaved differently using after making these changes -- the installed files ended up in django_polls-0.1-py2.7.egg and easy-install.pth rather than the previous polls and django_polls-0.1.egg-info.

[stumbles]

Replying to answer my own question - the following looks like a neater approach:

from distutils.core import setup

setup(
    name='django-polls',
    version='0.1',
    packages=['polls'],
    package_data={'polls': ['templates/polls/*']},
    license='',
    long_description=open('README.txt').read(),
    url='http://www.example.com/',
    author='Your Name',
    author_email='yourname@example.com',
)

Please let me know if there's a better approach.

[stumbles]

Specify "package_data" explicitly and remove "include_package_data" from setup.py.

[Tim Graham]

Looks good. Trying again, yes, it looks like the TEMPLATE_DIRS change isn't needed.

Made a couple tweaks:
"In the Tutorial 3" -> "In Tutorial 3"
Under the virtualenv section, it says "This has some" and looks like "disadvantages" got lost (added back).

I've built this and stuck it up a server temporarily for review. I'm going to email django-developers so we can get some final feedback.

[stumbles]

Very minor additional changes:

• use url function in README rather than tuple - seems this is now the default approach for new projects
• change "cd django-polls" to "cd django-polls-0.1"

[stumbles]

Use url fn in README and add version number when cd'ing

[Travis Swicegood]

Just read through the tutorial. It looks good, but I don't know if it's the right thing to be teaching to someone at this point in their Django career. Reusable apps are something I would consider an advanced topic to be included as the final tutorial entry, if the topic belongs there at all. If you look at who the tutorial is geared at (someone definitely new to Django, but probably also new to Python), this tutorial is going to leave them scratching their head. What does in to the setup.py? (Can I include my own URL, or does it have to be example.com? Does every reusable app for Django have to have django- in front of the directory name?) What does the MANIFEST.in file do and why is it there? (Do .in files have some special meaning in Python in general? Can they do other things?)

In it's current state, I'm -1. I think this is a good foundation, but it needs to be fleshed out quiet a bit before it should be merged. I'm also -0 on whether reusable apps should be part 5 of the tutorial. I think tutorial 4's current list of next topics provides a really good starting point for further tutorials and that this is best left to a final spot or possibly even as part of the main documentation rather than the tutorial.

[Daniel Sokolowski]

I am +1 on this. One of the biggest challenges I faced when starting with Django is figuring out how to structure my projects/app and this tutorial clearly helps with that. Thank you to the authors working on this.

I suggest it ought to use distribute which is a replacement for setup_tools which was a successor for distuitls - python package installation landscape is hairy to say the least: ​Differences between distribute, distutils and setuptools?. I suggest the following setup.py.

from setuptools import setup, find_packages

setup(
    install_requires=['distribute'], # let's require the successor to setuptools    
    name='django-polls',
    version='0.1',
    packages=find_packages(),
    include_package_data=True, # this will use MANIFEST.in during install where we specify additional files
    license='',
    description=package.__doc__.strip(),
    long_description=open('README.txt').read(),
    url='http://www.example.com/',
    author='Your Name',
    author_email='yourname@example.com',
)

I am torn between using very explicit package_data or the include_package_data and MANIFEST.in to specify additional directories files. Former is more python however personally and from ​experience I settled on the latter as that is the only way to include root package files like README.txt and is the simplest - ​Including Data Files.

include LICENSE
include README.rst
recursive-include polls/templates *
recursive-include docs *

Also let's link to ​http://packages.python.org/distribute/index.html in the tutorial

[Anssi Kääriäinen]

I can see the problem of putting this into Tutorial 5. This stuff is really advanced if all you wanted was creating a simple web app. For that purpose "Tutorial 5: How to use reusable apps others wrote" would be much better.

So, could this be "Advanced tutorial: How to write reusable apps" or something like that.

One sign of problem is that I can't actually review the content of this tutorial for correctness, as I don't know the stuff this tutorial is introducing (I probably should, but that is another matter). Is this really something a newbie needs to be exposed on day 1?

[Tim Graham]

I think positioning this as an "advanced tutorial" and incorporating some of the feedback above, as well as from the thread on django-developers* makes sense.

• ​https://groups.google.com/forum/?fromgroups=#!topic/django-developers/I3uUlt2UdT4

[Tim Graham]

Updates based on feedback

[Tim Graham]

Thanks for the feedback guys. I've updated the patch in the following ways:

• Included this as a separate "Advanced tutorial" rather than simply "Tutorial 5"
• Added text and/or links to answer questions like:
  • What's the purpose of setup.py?
  • What does the MANIFEST.in file do and why is it there?
  • Does every reusable app for Django have to have django- in front of the directory name?

• Added a note that discusses naming conventions of packages.

• Added links to a couple more resources like readthedocs and djangopackages.com

You can view the HTML here: ​http://techytim.com/django/tutorial05/intro/reusable_apps.html

Since Django itself uses distutils and we have a tested patch using that, I think it makes to stick with that over distribute, although I am open to +1/-1 from any core devs.

The patch is based on the patch for #18715 Tutorial 3 refactor. Would appreciate a second set of eyes on that as well if you are interested.

[Russell Keith-Magee]

Another good looking documentation addition. Some notes:

• The shoutout to djangopackages.com -- I agree this is the best resource in this particular space at the moment -- but that's *at the moment*. There have been others in the past, and as a project, we dont' have any guarantees that djangopackages will be around next week. This may be a case of something that we need to fix at a project level by formalizing the relationship with djangopackages, rather than changing the text here.

• In crossover from my comments on #18715; The introduction of app-level template directories seems more appropriate here, but it might still be worth a shoutout in Tutorial 3 letting people know that this option exists (even if you don't go into details)

• The "Why nested" note -- an example would help here. Show how a template name like "base.html" would easily collide between apps.

• Regarding distutils vs distribute: This is one of those areas where I think Django has a responsibility to be a community leader. The ​Hitchhikers guide to Python Packaging (which is itself worth a callout in the docs) recommends distribute; and everything I've seen says that the features of distribute will ultimately become part of distutils2; it seems to me that recommending distribute is the right thing to do here. AFAIK, the only reason Django itself doesn't use distribute is historical; setuptools had some pathological behavior back in the day, and nobody has had enough of an itch to make the change. Call this a +0 for distribute from me.

• The packaging says to create a README; it doesn't say create a LICENSE file. We *must* say this. Code without a license is *useless*. Simliarly, the setup.py file doesn't specify a license definition. By way of communtiy guidance, we can say "Django, and many Django-compatible apps are distributed under the BSD license; however, you're free to pick your own license. However, be aware that your licensing choice will affect who is able to use your code".

• The setup.py also doesn't mention any trove classifiers. It should. At the very least, we should be able to say:

          'Environment :: Web Environment',
          'Framework :: Django',
          'Intended Audience :: Developers',
          'License :: OSI Approved :: BSD License',
          'Operating System :: OS Independent',
          'Programming Language :: Python',
          'Programming Language :: Python :: 2.6',
          'Programming Language :: Python :: 2.7',
          'Topic :: Internet :: WWW/HTTP',
          'Topic :: Internet :: WWW/HTTP :: Dynamic Content',
  

• Again, I'm a little hesitant to link out to third-party blogs and projects in the "more about" section. If we need to elaborate best practices, I'd say we should distil Eric's blog post into a couple of paragraphs of best practices; if we need to link to a list of packages, djangopackages.com would seem a better candidate (with the caveat from above).

[Carl Meyer]

Agreed, this is looking great.

I don't have clarity on how best to both keep it simple and recommend best practices in the morass of confusion that is Python packaging, when it comes to recommending distutils vs distribute. You can't really recommend distribute without explaining that it's a third-party package, not part of the Python standard library, meaning that you're preventing people who don't have it from installing your package - but at the same time, in practice everybody has it, so that's not really a problem. But that's an awful lot of extra potentially-very-confusing information to try to squeeze in (not to mention explaining _why_ you'd use distribute rather than the stdlib option), all of which argues for just using distutils.

The main problem I see is in recommending distutils and recommending "python setup.py install" in the same tutorial - because that's the one path that means no useful metadata is installed along with the package, and therefore automated uninstall is not possible (pip is not able to uninstall packages installed in this way). When you add in "sudo", it means our tutorial is telling people to install a test package as the root user, to their system site-packages, in such a way that it can't be removed without manually sudoing and using "rm -rf". That's... unfortunate. We have three choices (use distutils, demonstrate "python setup.py install", default to "sudo" rather than virtualenv) which I think are all individually the right choice for simplicity's sake, but the combination has a sub-optimal result. If we had to change any of those three, I'd probably go for switching from distutils to distribute.

Regardless, if the tutorial tells them to "sudo python setup.py install", I think we need to tell them how they can uninstall it. If we use distribute, we can just link to pip and say "pip can uninstall this." If we stick with distutils, we have to tell them they'd need to manually "sudo rm -rf" the polls directory (which they can find as demonstrated, by importing polls and checking polls.__file__).

One other note - in Python 2.7 it is no longer necessary to redundantly add files to MANIFEST.in that are listed in package_data (in this case, the templates). That was a bug in distutils in Python 2.6 and previous that was fixed in Python 2.7. I guess as long as we support 2.6 we should probably demonstrate a 2.6-compatible technique, but I wonder if it's worth a brief aside to note the redundancy, and that you don't need it if generating the sdist using Python 2.7. In any case, whenever Django drops 2.6 support this should be changed.

And lastly: if we keep the link to Eric's article, his last name is spelled "Holscher".

[Russell Keith-Magee]

One other piece of feedback - I'm in complete agreement that this is really "advanced tutorial 1" rather than "tutorial 5"; so when committed, the tutorial URL should probably reflect that.

[Tim Graham]

Thanks for the feedback, I'll work to incorporate these suggestions.

[Tim Graham]

Updates:

• Added a note in tutorial 3 regarding app-level template directories.
• Added an example of how template names could collide to the "Why nested" note.
• Changed from distutils to distribute and pip.
• Added a step of creating a license file
• Added trove classifiers to setup.py
• Removed link to Eric's blog post - if someone wants to add documentation on best practices it can be a separate ticket.

Other Notes:

• Since we switched to using distribute, it looks like Carl's point about redundancy between MANIFEST.in and package_data is moot.
• The updated doc is ​http://techytim.com/django/tutorial05/intro/reusable-apps.html

Todo:

• Should we keep the link to djangopackages.com?
• Russ, regarding the URL, I'm assuming you didn't like the 'intro' bit? Since this follows from tutorial 4, I felt like moving it to a whole new top level like /advanced-tutorials/reuseable-apps/ might be a bit too much (we won't get any next/previous links to and from tutorial 4) so I opted for /intro/reusable-apps/. I'm open to suggestions if you still don't like it. (And I just realized that maybe you saw tutorial05 in the url where I posted the built docs which is just a folder on my server where I put this.)

[Russell Keith-Magee]

I'm happy keeping the link to djangopackages.com; it's a reasonably robust part of the community, and we can start negotiations with Danny about making the arrangement a little more formal.

And yes, my objection was to the tutorial05 part in the URL. If that's just a build artefact, that's fine - I just raising it in case it was the proposed location in the source tree.

[Carl Meyer]

Some comments on the updated draft:

• Per its author, setuptools is not defunct, and he has made fixes in it since the distribute fork, so I think it's needlessly partisan to refer to it as "defunct" and claim that it has "bugs which will never be fixed." I recommend this alternative description of distribute: "It's a community-maintained fork of the older setuptools project." (Also, there's a mismatched parenthesis at the end of that sentence currently).

• I'm -0.5 on linking to the "state of Python packaging" diagram from the Hitchhiker's Guide, as it has some factual errors (pip doesn't work directly with distutils, it relies on setuptools/distribute) and makes assertions which at this point remain far from clear (i.e. that setuptools/distribute are headed for the dustbin, even though today they remain by far the most-used Python packaging tools, or even that distutils2 is headed for the stdlib - it was removed from Python 3.3 and so far there's been no movement to reintroduce it for Python 3.4, rather pieces of it are being rewritten under yet another name, "distlib"). On the other hand, I appreciate the attractiveness of having something simple to link to that demonstrates just how muddled things are, and I don't have a replacement link target to recommend in that role.

• Including distribute (or setuptools) in the install_requires in setup.py is wrong (unless your package actually imports and uses "setuptools" or "pkg_resources" at runtime - not in setup.py but in the actual packaged code - which the "polls" app does not). The note below claims that "the install_requires line ensure that distribute will be installed for anyone who is installing our package" but this is not true - if someone doesn't have setuptools or distribute installed to begin with, setup.py won't even run long enough to get to the install_requires, it'll raise ImportError on the first line. And having distribute in install_requires means that if a user has setuptools installed, installing your package will forcibly replace it with distribute, which is really unfriendly - the point of distribute being a drop-in replacement is that your package should work equally well for people using setuptools or distribute, and should not enforce one choice or the other on your users.

• Using open('README.txt').read() as the value of long_description will break if setup.py is executed from outside the current directory (e.g. python django-polls/setup.py install). For this to be reliable, an import os is needed up top, then a here = os.path.dirname(os.path.abspath(__file__)), then long_description=open(os.path.join(here, 'README.txt')).read().

• The line "only a limited set of files are included in the package by default" could be made more specific: "only Python modules and packages are included in the package by default."

Otherwise, looks good to me!

[Tim Graham]

In trying to address the fourth bullet point, I updated long_description as you suggested but the package still doesn't install properly when running setup.py outside django-polls-0.1:

$ sudo python django-polls-0.1/setup.py install
running install
Checking .pth file support in /usr/local/lib/python2.7/dist-packages/
/usr/bin/python -E -c pass
TEST PASSED: /usr/local/lib/python2.7/dist-packages/ appears to support .pth files
running bdist_egg
running egg_info
writing django_polls.egg-info/PKG-INFO
writing top-level names to django_polls.egg-info/top_level.txt
writing dependency_links to django_polls.egg-info/dependency_links.txt
warning: manifest_maker: standard file 'setup.py' not found

reading manifest file 'django_polls.egg-info/SOURCES.txt'
writing manifest file 'django_polls.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-i686/egg
running install_lib
warning: install_lib: 'build/lib.linux-i686-2.7' does not exist -- no Python modules to install

creating build/bdist.linux-i686/egg
creating build/bdist.linux-i686/egg/EGG-INFO
copying django_polls.egg-info/PKG-INFO -> build/bdist.linux-i686/egg/EGG-INFO
copying django_polls.egg-info/SOURCES.txt -> build/bdist.linux-i686/egg/EGG-INFO
copying django_polls.egg-info/dependency_links.txt -> build/bdist.linux-i686/egg/EGG-INFO
copying django_polls.egg-info/top_level.txt -> build/bdist.linux-i686/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/django_polls-0.1-py2.7.egg' and adding 'build/bdist.linux-i686/egg' to it
removing 'build/bdist.linux-i686/egg' (and everything under it)
Processing django_polls-0.1-py2.7.egg
creating /usr/local/lib/python2.7/dist-packages/django_polls-0.1-py2.7.egg
Extracting django_polls-0.1-py2.7.egg to /usr/local/lib/python2.7/dist-packages
Adding django-polls 0.1 to easy-install.pth file

Installed /usr/local/lib/python2.7/dist-packages/django_polls-0.1-py2.7.egg
Processing dependencies for django-polls==0.1
Finished processing dependencies for django-polls==0.1

A directory is created: /usr/local/lib/python2.7/dist-packages/django_polls-0.1-py2.7.egg
but the polls subdirectory is missing.

[Carl Meyer]

Replying to timo:

In trying to address the fourth bullet point, I updated long_description as you suggested but the package still doesn't install properly when running setup.py outside django-polls-0.1:
...
A directory is created: /usr/local/lib/python2.7/dist-packages/django_polls-0.1-py2.7.egg
but the polls subdirectory is missing.

Interesting, I've never seen that. My first guess would be that it's a bug in find_packages? I generally don't use that, even when using setuptools/distribute, I just list packages explicitly. Does removing find_packages fix it?

[Tim Graham]

With packages=['polls']:

sudo python django-polls-0.1/setup.py install
running install
Checking .pth file support in /usr/local/lib/python2.7/dist-packages/
/usr/bin/python -E -c pass
TEST PASSED: /usr/local/lib/python2.7/dist-packages/ appears to support .pth files
running bdist_egg
running egg_info
creating django_polls.egg-info
writing django_polls.egg-info/PKG-INFO
writing top-level names to django_polls.egg-info/top_level.txt
writing dependency_links to django_polls.egg-info/dependency_links.txt
writing manifest file 'django_polls.egg-info/SOURCES.txt'
warning: manifest_maker: standard file 'setup.py' not found

error: package directory 'polls' does not exist

It may be a path issue. For example, if I rename django-polls-0.1 to django-polls and then modify setup.py to packages=['django-polls/polls'] it works; of course, this breaks it when running setup.py in the django-polls directory.

Yes, if I add:

my_path = os.path.abspath(__file__)
os.chdir(os.path.normpath(os.path.join(my_path, os.pardir)))

after here=os.path.dirname... in setup.py, I can run setup.py from any path. Fun, fun.

[Carl Meyer]

Looks like even distutils itself is built to assume that setup.py is in the current working directory. Interesting, I didn't realize that. I think I'd never tried to actually run sdist or install from a different directory, only simple metadata-getting commands (python setup.py egg_info or e.g. python setup.py --long-description).

That definitely reduces the importance of point 4, but I still think it's worth doing this the robust way, because at least then a few more (simple) uses of setup.py will work from a different directory. I've definitely seen real bugs filed against real packages because the handling of README.txt wasn't robust against execution from a different directory.

[Tim Graham]

Updates:

• Updated language for setuptools vs. distribute per Carl's suggestion
• Removed link to "state of python packaging" due to inaccuracies.
• Removed install_requires=['distribute'] from setup.py
• Updated open('README.txt') to allow executing setup.py from any directory
• Made language regarding what files are included by default more specific per Carl's suggestion

[Tim Graham <timograham@…>]

In 08cf54990ae112083b159aa4e263c1f64f396f39:

Fixed #16671 - Added a tutorial on reuseable apps

Thank-you Katie Miller and Ben Sturmfels for the initial draft,
as well as Russ and Carl for the reviews.

[Tim Graham <timograham@…>]

In 5c7406b236b641ac1802d1471f04b27166909121:

[1.5.X] Fixed #16671 - Added a tutorial on reuseable apps

Thank-you Katie Miller and Ben Sturmfels for the initial draft,
as well as Russ and Carl for the reviews.

Backport of 08cf54990a from master