Context Navigation

← Previous Ticket
Next Ticket →

#15578 closed Cleanup/optimization (fixed)

loaddata and processing order of fixtures

Reported by:	Luc Saffre	Owned by:	Leo Suarez
Component:	Documentation	Version:	1.2
Severity:	Normal	Keywords:
Cc:		Triage Stage:	Accepted
Has patch:	yes	Needs documentation:	no
Needs tests:	no	Patch needs improvement:	yes
Easy pickings:	no	UI/UX:	no

Description

As the documentation for loaddata correctly says: "the order in which fixture files are processed is undefined."
That's a pity because I use fixtures to provide combinable and interdependent sets of demo data.

Would it be difficult to implement a rule for the order in which fixtures are loaded?
I'd suggest the following rule:
If there are 3 applications "a", "b" and "c" and you invoke

manage.py loaddata foo bar baz

(or you specify

fixtures = ['foo','bar','baz']

in a TestCase, then loaddata would first load all fixtures 'foo' for all applications, then all fixtures named 'bar', then all fixtures 'baz'.

Attachments (1)

doc_patch.diff (1.0 KB ) - added by trez 14 years ago.: doc path for fixture ordering - maybe just a draft

Download all attachments as: .zip

Change History (18)

comment:1 by Luke Plant, 14 years ago

Type:	→ New feature

comment:2 by Luke Plant, 14 years ago

Severity:	→ Normal

comment:3 by Jacob, 14 years ago

Component:	Uncategorized → Core (Management commands)
Triage Stage:	Unreviewed → Design decision needed
Type:	New feature → Cleanup/optimization

Interesting - that clause in the docs dates from r4659, the original commit of fixtures. But as far as I can tell, it's never been true: the code actually does exactly what you suggest and does load fixtures in the order given. I think we should just strike that paragraph from the docs, but I'd like to see if Russ remembers why he wrote that in the first place.

comment:4 by Russell Keith-Magee, 14 years ago

Component:	Core (Management commands) → Documentation
Triage Stage:	Design decision needed → Accepted

The statement is still accurate -- just imprecise.

Firstly, fixtures *are* loaded in the "foo", "bar", "baz" order. However, "foo" doesn't refer to a single fixture; any fixture named "foo" *in any app* will be loaded. This is what allows the "initial_data" load to occur, but the same trick works for any fixture name. The admonition that is there is intended to wave people off of the idea that specifying fixture order is enough to guarantee 100% predictable. This isn't true -- you also need to take into account the order in which applications are defined.

The admonition was probably also a partial "commit to nothing" statement, giving us some wiggle room if we needed to change things. Fixtures have been in the water for long enough that this reason no longer holds.

So I agree - fixing the documentation is all that is required here. Just deleting the line is the simple approach; a slightly better approach would expand the statement to include a the app ordering part of the process.

comment:5 by Luc Saffre, 14 years ago

Glad to read that this is just a documentation bug. Maybe something like this:

If several fixtures are specified, then the application order is also important: the first fixture is loaded first for all applications (in the order specified by INSTALLED_APPS), then the second fixture, and so on.

by trez, 14 years ago

Attachment:	doc_patch.diff added

doc path for fixture ordering - maybe just a draft

comment:6 by trez, 14 years ago

Easy pickings:	unset
Has patch:	set
UI/UX:	unset

The patch attached contains this explanation:

SQL data files are executed in the same order than their ordering
in the fixtures.

For example:

fixtures = ['foo','bar','baz']
foo will be executed first then bar, then baz.

However, fixtures are not loaded for one application but for any
applications which contains this fixture.

Therefore, you should also take into account the order in which application
are defined.

Hope this help but may surely be crystal clear with some improvements

comment:7 by Aymeric Augustin, 13 years ago

Patch needs improvement:	set

There are a couple issues with this patch.

It should change the following sentences in ref/django-admin.txt:

"Note that the order in which fixture files are processed is undefined." — the section about fixtures in topics/testing.txt links to this paragraph.
"Note that the order in which the SQL files are processed is undefined." — not sure about this one, but if the order is deterministic, let's fix it too.

Finally, the patch isn't relative to the trunk directory of the source distribution.

comment:8 by pascal chambon, 9 years ago

I'd like to highlight another imprecise point in this documentation paragraph : as described in (duplicate) ticket #15578, even if the order of fixture import gets precised, users may not always rely on the "in transaction" behaviour of loaddata, regarding relationships between records.

Indeed DBs like MySQL do not have defered constraint checking, so referential integrity must be true for *each* SQL operation, not just as a whole. Others like PostgreSQL can have it on demand (http://www.postgresql.org/docs/9.1/static/sql-set-constraints.html), but I have alas no experience with these and the way Django uses them.

Does anybody have experience with loading interdependent fixtures on several different engines ? Can we simply state "If your engine supports defered constraint checking etc. etc." ?

comment:9 by Muskan Vaswan, 4 years ago

Is this patch better framed as the following?

Fixtures are processed in the same order as the one specified in the list.
This means that if Fixtures = ['foo', 'bar', 'baz'] then the order in which they are loaded will be 'foo' followed by 'bar' and then 'baz'. Where, "foo" is any fixture named 'foo' in any application. This implies that in cases where there are several fixtures with the same name, the order in which applications are defined in INSTALLED_APPS is also a taken into account.
This order may not always be predictable. If the database backend supports row-level constraints, these constraints will be checked at the end of the transaction.