Visualizing Django source code changeset dependencies

When following the workflow I describe in [1] to be able to maintain a local copy of Django based on the 0.96 release with selected changes backported from the SVN trunk tip I soon found myself in following situation:

Certain changeset C that I'd want to use wasn't being considered by [2]darcs as a potential candidate so it didn't offer me to apply it.

Later the cause became evident: The C changeset wasn't being considered by darcs because I had previously answered "no" when it offered me to apply a (chronologically previous) changeset B on which C depends (dependency in this context means both B and C modify some overlaping section of some file).

Given the fact that in darcs patches (changesets, patchsets) are first class citizens, it is relatively easy (when compared with other revision control systems) to track and visualize the dependencies among them by using it together with some related [3]tools and [4]Graphviz.

If we add a couple of ad-hoc scripts and some tips from the experience gained to the mix we can get something like this (partial output):

Scripts

filter1.sh

#!/bin/sh

# This filter changes the labels of the nodes of a Graphviz dot
# file as generated by darcs-deps to be just the patchset
# number.
#
# It depends on the labels being of the form
# "[tailor-project-name @ changeset-number]"
#
# It needs one command line parameter: The tailor-project-name
#

if test $# -ne 1; then
    echo "Usage:"
    echo "$(basename $0) tailor-project-name"
    exit 1
fi

/bin/sed "s/\[$1 @ \\([0-9]\+\\)\]/\1/g"

filter2.sh

#!/bin/sh

# This filter modifies the properties of the nodes of a Graphviz dot
# file to add a relative URL "href" property with the same value as
# the node label.
#
# It depends on the labels being the number of the revision/changeset
# represented by the node (possibly generated by filtering the darcs-deps
# output through filter1.sh).
#
# It has been created to facilitate the navigation of the Django trunk
# changeset dependancy graph, enabling (for the relevant output formats: pdf,
# image maps) the user to click on a node and getting a web browser window
# opened with the Django Trac page showing the corresponding changeset.
#
# For this to work, a graph level property href=http://code.djangoproject.com/changeset/
# must be also present, it can be added with a -Ghref=... commandline switch
# in the dot(1) invocation.

if test $# -ne 0; then
    echo "Usage:"
    echo "$(basename $0)"
    exit 1
fi

/bin/sed "s#label = \"\\([0-9]\+\\)\"#label = \"\1\", href=\"\1\"#g"

Usage

ramiro@tabaqui:~/src/django/inmutable $ darcs-deps | filter1.sh django096 | filter2.sh | unflatten | dot -Tps2 -Gsize=50,50 -Ghref=http://code.djangoproject.com/changeset/ -o ~/django-deps.ps -Tpng -o ~/django-deps.png
Examining patch 249 of 249.

ramiro@tabaqui:~/src/django/inmutable $ cd

ramiro@tabaqui:~ $ poster -p1x3A4 -o django-deps-mpage.ps django-deps.ps

ramiro@tabaqui:~ $ ps2pdf django-deps-mpage.ps django-deps.pdf

Explanation

darcs-deps is ran on the inmutable darcs repository (please refer to [1] to see what that repository is and how to create it), it generates a Graphviz dot language description of the dependancy graph.

We filter that output first through our two custom filtering scripts (see above) to change some graph node attributes, and then through unflatten(1) (part of the Graphviz suite) to enhance the layout of the graphic representation.

Finally, we feed the result to dot(1) (also part of Graphviz) to get our final output. For some output methods (PDF [via Postscript], server/client side image maps) we'll be able to click on the nodes and get our Web browser to go to the Django Trac page associated with the changeset at hand.

For the PDF output we use poster(1) on the intermediate Postscript output to create a legible non-cropped multi-page output.

Files

Generated as of revision [5165] (May 8 2007 3:43 UTC).

Some files are hosted elsewhere while the Trac crew solve the ​bug that keeps us from attaching binary files to the Wiki and tickets.

DOT files

• Graph as generated by darcs-deps, before post-processing: django-deps-r5165-orig.dot​
• After post-processing, before being feed to dot: django-deps-r5165-final.dot

Example output

• PNG output: ​django-deps-r5165.png
• PDF output: ​django-deps-r5165.pdf

Notes

• The "Tailorization" node represents an initial synthetic patch created by darcs from which every other leaf node depends on, perhaps it would be better to find a way to eliminate it from the graph.
• I attempted without success modifying the darcs-deps script to eliminate the nodes that have no other nodes depending on them, next try will be using Graphviz's gvpr(1) tool.
• The PDF output for this iteration has some problems, something that didn't happen before; it seems there is some non-deterministic interaction between poster and the dot PS output we feed to it. It would be nice to be able to create multi-page output directly from dot, ideas welcome.

References

1. ​Django: My post 0.96 branch
2. ​darcs
3. ​darcs-deps (darcs repo: ​http://scratchbox.org/~ttimonen/repos/darcs-deps) (Perl script)
4. ​GraphViz
5. ​Ghostscript
6. ​poster

Feedback

Please direct all feedback to ​Ramiro Morales.