Use the natural key instead of the primary key when serializing

[Chris Beaven]

This would allow existing objects to be updated and missing objects to be added across databases (e.g. when the autonumber primary keys may not match).

[Chris Beaven]

This also fixes #11486 by proxy.

[Russell Keith-Magee]

I accept the use case - that you should be able to say "update the existing object that matches the natural key, or create a new object if one doesn't exist". However I'm not convinced that providing a natural key for the PK value is the right way to do this.

Essentially, this approach means specifying the natural key data twice - once in the fields of the object, and once in the primary key value. This strikes me as especially un-DRY.

I haven't given this a lot of thought, but there might be a way of using the natural key without to specify it in the pk attribute. A natural key needs to be unique, so if a model defines a natural key, you shouldn't need to have the PK value specifically (or the natural key value as the primary key) in order to reconstruct the natural key for an object. This does leave the question of when the 'update or create' behavior should be invoked, but I think this will be less prone to error than reproducing the natural key.

[Chris Beaven]

now with tests/docs

[Paul Winkler]

fwiw, patch 3 works for me in initial testing. Really useful feature!

[Tai Lee]

Updated patch to work on r14992. Appears to work well locally. serializers_regress tests pass (on sqlite at least).

[Alex Gaynor]

Fixed by [14994].

[Alex Gaynor]

Reopening.

[Tai Lee]

I've just updated the patch after feedback from freakboy3742 on IRC.

The new patch adds --natural-foreign and --natural-primary options to the dumpdata management command, maps the -n and --natural options to --natural-foreign and adds a note in the documentation that the old options are deprecated. This should retain backwards compatibility with anyone who was depending on data dumped with natural keys still containing a pk field for models that support natural keys. It also adds the corresponding use_natural_foreign_keys and use_natural_primary_keys arguments to serializers.serialize() and maps use_natural_keys to use_natural_foreign_keys.

I've left a comment in the source code where I think we should probably raise a deprecation warning (but not actually raised it) if that ends up being the desired behaviour.

One quirk is that the code beginning at line 30 in django/core/serializers/python.py appears to affect the order of fields in serialized output depending on how the data dict is constructed, even though the resulting data dict contains the same data. If I first create the data dict with model and fields items and then conditionally add a pk item (as the old patch did), then the serialized output is different to when I create the data dict with model, pk and fields items. Rather than change a tonne of hard coded tests, I construct the data dict the same way that it would have been before the patch which results in a couple of duplicated lines of code.

[patchhammer]

13252-natural-key-serializing-r14995.diff fails to apply cleanly on to trunk

[Claude Paroz]

Updated to apply cleanly on current trunk

[Chris Beaven]

The new behaviour needs to be optional to be backwards compatible.

[Claude Paroz]

Updated to current trunk

[Tai Lee]

I've updated this patch again on a branch at GitHub.

​https://github.com/thirstydigital/django/tree/tickets/13252-natural-key-serializing

SmileyChris, can you elaborate on making the new behaviour optional? Unless I'm missing something, it is? The --natural option maps to --natural-foreign (which behaves the same, with a pending deprecation warning) and we now have a new --natural-primary option as well.

I believe I've incorporated all the changes russellm requested and this should be RFC now.

[Simon Charette]

FWIW all tests pass on python 2.7.3rc1 sqlite3. I guess the backward compatibility or must opt-in issue SmileyChris was referring to is no more with two new flags?

[anonymous]

I would be very pleased to see this feature enhancement reach the master branch asap 'cause I actually pretty much need it and have write myself some kind of the same..

[Tai Lee]

Rebased onto master and opened a pull request.

​https://github.com/django/django/pull/894

[Tai Lee]

Rebased onto master and changes for feedback integrated.

[Tim Graham]

Left a couple more comments on the PR.

[Tim Graham <timograham@…>]

In e527c0b6d808cb8e4bedf79ded3dc4ad1a7e17a8:

Fixed #13252 -- Added ability to serialize with natural primary keys.

Added --natural-foreign and --natural-primary options and
deprecated the --natural option to the dumpdata management
command.

Added use_natural_foreign_keys and use_natural_primary_keys
arguments and deprecated the use_natural_keys argument to
django.core.serializers.Serializer.serialize().

Thanks SmileyChris for the suggestion.

[Tim Graham <timograham@…>]

In c3336e7e4f146fc62272d462288a00f8d78c1f83:

Removed dumpdata --natural option and serializers use_natural_keys parameter.

Per deprecation timeline; refs #13252.