Add contributor facing documentation for the ORM

[Mads Jensen]

Overall, Django's documentation is quite good for the average user, and it also covers topics on how to write custom aggregate-functions and expressions, custom lookups and more. However, documentation on how the ORM works internally is mostly something that can be achieved through talks from YouTube, and from reading the code (albeit, there are some comments, and IDE tools letting you jump between symbols and methods do help) this takes time.

Something more elaborate on the compilers would be nice (SQLInsertCompiler, SQLDeleteCompiler, SQLAggregateCompiler, SQLDeleteCompiler, SQLCompiler), and structures such as PathInfo are missing, the different methods in Query, and how Query in the django.db.models.models and django.db.models.sql.query, as well as the chain of calls that are called to generate and optimize the code/query. The structure of a backend and what files that are necessary is also a bit absent. It's a bit tricky to see where this would fit into the current documentation structure, but it would augment Django a lot. In ​https://docs.djangoproject.com/en/dev/howto/custom-lookups/ there's a comment about the compiler.

SQLalchemy outlines the structure of its architecture in a chapter in Architecture of Open Source Applications: ​http://aosabook.org/en/sqlalchemy.html My requests aims at providing a bit of this in some form.

Talks on YouTube that were helpful in getting some idea on how the ORM works. Since the ORM was built for relational databases, this is also the scope of this documentation request.

1. ​https://www.youtube.com/watch?v=CGF-0csOjPwa
2. ​https://opbeat.com/community/posts/expressions-by-josh-smeaton/
3. ​https://www.youtube.com/watch?v=OCq2eUmrCfI

[Shai Berger]

Hi, Thanks for filing this request. I think it is an awesome idea.

There's a simple, "bureaucratic" problem with it: The Django project is committed to backwards-compatibility with respect to documented features and behaviors. We don't want that to apply to implementation details, neither those of the ORM nor otherwise. So we'd need to either create a "non-committing" section of the documentation (with a different appearance?), or put this part out of the regular documentation.

[Josh Smeaton]

I agree, I think the ORM in particular should be documented where possible, with the caveat that certain APIs are not subject to backwards compatibility. An ORM architecture document sounds like the right place for such a thing. It could start off quite high level by documenting the modules and how a query is created. It could then go further over time by documenting helpers like JoinPromoter and PathInfo, and especially documenting all of the feature flags.

It'll help future ORM contributors quite a bit while supporting those in the community that are already writing 3rd party backends with trial and error.

It's a big job though. The tradeoff will be maintaining such a document in the face of internal refactorings.

[Tim Graham]

Contributor facing documentation (as proposed in #24989 for migrations) is a good idea. For example, Ansii started writing up some explanation of ​join promotion that would be suitable for inclusion.