Loading fixtures with one-to-one inheritance on abstract model and M2M fails since 1.7 Beta

[stanislas.guerra@…]

Hi,

I am testing a project with the 1.7Beta release / master branch and Django can't load some fixtures when I am running my test-suite:

======================================================================
ERROR: test_validation_ajouter_double_page (fab4.front.tests.tests.DossierTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/stan/src/venv_dj_1_7/lib/python2.7/site-packages/Django-1.8.dev20140410233300-py2.7.egg/django/test/testcases.py", line 175, in __call__
    self._pre_setup()
  File "/Users/stan/src/venv_dj_1_7/lib/python2.7/site-packages/Django-1.8.dev20140410233300-py2.7.egg/django/test/testcases.py", line 747, in _pre_setup
    self._fixture_setup()
  File "/Users/stan/src/venv_dj_1_7/lib/python2.7/site-packages/Django-1.8.dev20140410233300-py2.7.egg/django/test/testcases.py", line 869, in _fixture_setup
    return super(TestCase, self)._fixture_setup()
  File "/Users/stan/src/venv_dj_1_7/lib/python2.7/site-packages/Django-1.8.dev20140410233300-py2.7.egg/django/test/testcases.py", line 788, in _fixture_setup
    **{'verbosity': 0, 'database': db_name, 'skip_checks': True})
  File "/Users/stan/src/venv_dj_1_7/lib/python2.7/site-packages/Django-1.8.dev20140410233300-py2.7.egg/django/core/management/__init__.py", line 167, in call_command
    return klass.execute(*args, **defaults)
  File "/Users/stan/src/venv_dj_1_7/lib/python2.7/site-packages/Django-1.8.dev20140410233300-py2.7.egg/django/core/management/base.py", line 337, in execute
    output = self.handle(*args, **options)
  File "/Users/stan/src/venv_dj_1_7/lib/python2.7/site-packages/Django-1.8.dev20140410233300-py2.7.egg/django/core/management/commands/loaddata.py", line 60, in handle
    self.loaddata(fixture_labels)
  File "/Users/stan/src/venv_dj_1_7/lib/python2.7/site-packages/Django-1.8.dev20140410233300-py2.7.egg/django/core/management/commands/loaddata.py", line 89, in loaddata
    self.load_label(fixture_label)
  File "/Users/stan/src/venv_dj_1_7/lib/python2.7/site-packages/Django-1.8.dev20140410233300-py2.7.egg/django/core/management/commands/loaddata.py", line 146, in load_label
    obj.save(using=self.using)
  File "/Users/stan/src/venv_dj_1_7/lib/python2.7/site-packages/Django-1.8.dev20140410233300-py2.7.egg/django/core/serializers/base.py", line 176, in save
    setattr(self.object, accessor_name, object_list)
  File "/Users/stan/src/venv_dj_1_7/lib/python2.7/site-packages/Django-1.8.dev20140410233300-py2.7.egg/django/db/models/fields/related.py", line 1183, in __set__
    manager = self.__get__(instance)
  File "/Users/stan/src/venv_dj_1_7/lib/python2.7/site-packages/Django-1.8.dev20140410233300-py2.7.egg/django/db/models/fields/related.py", line 1169, in __get__
    through=self.field.rel.through,
  File "/Users/stan/src/venv_dj_1_7/lib/python2.7/site-packages/Django-1.8.dev20140410233300-py2.7.egg/django/db/models/fields/related.py", line 821, in __init__
    (instance, source_field_name))
ValueError: Problem installing fixture '/Users/stan/Dropbox/Projets/Aden/Publish/repos/publish/fab4/fabrication/fixtures/tests/fabrication/fabrication2.json': "<Assistant: >" needs to have a value for field "user" before this many-to-many relationship can be used.

My models are :

from django.contrib.sites.models import Site
from django.contrib.auth.models import User


class SupportManager(models.Manager):
    def get_current(self):
        return self.get(pk=settings.SITE_ID)

    def get_by_natural_key(self, short_name):
        return self.get(short_name=short_name)


class Support(Site):
    short_name = models.CharField('nom court', max_length=10, unique=True)
    logo = models.ImageField('logo', upload_to='images/supports', blank=True)
    site_ptr = models.OneToOneField(Site, primary_key=True)  # allow to do select related without an extra query to Site.
    objects = SupportManager()

    def __unicode__(self):
        return self.name

    def natural_key(self):
        return (self.short_name,)


class Organisation(models.Model):
    name = models.CharField(max_length=100)
    short_name = models.CharField('nom court', max_length=16)


class Employe(User):
    supports = models.ManyToManyField(Support)
    organisation = models.ForeignKey('Organisation', null=True, blank=True)
    filtre_organisation = models.BooleanField(default=False)
    telephone = models.CharField(u'téléphone', max_length=100, blank=True)

    class Meta:
        abstract = True

    def __unicode__(self):
        return self.get_full_name()


class Assistant(Employe):
    class Meta:
        verbose_name = 'chargé de clientèle'

The fixture causing the crash is a concat from the following dumps (master version of dumpdata):

python manage.py dumpdata sites.Site --indent=2 --natural-foreign --pk=1
python manage.py dumpdata parametrage.Support --indent=2 --natural-foreign --pk=1
python manage.py dumpdata auth.Group --indent=2 --natural-foreign --pk=2
python manage.py dumpdata auth.User --indent=2 --natural-foreign --pk=8
python manage.py dumpdata parametrage.Assistant --indent=2 --natural-foreign --pk=8

Which is :

[
{
  "fields": {
    "domain": "proprietesdefrance.com",
    "name": "Propri\u00e9t\u00e9s de France"
  },
  "model": "sites.site",
  "pk": 1
},
{
  "fields": {
    "logo": "images/supports/logo_PDF.jpg",
    "short_name": "PDF"
  },
  "model": "parametrage.support",
  "pk": 1
},
{
  "fields": {
    "name": "Relation Client",
    "permissions": [
      [
        "add_logentry",
        "admin",
        "logentry"
      ],
      [
        "change_statut_refus_bat_to_service_fab",
        "fabrication",
        "changementstatut"
      ],
      [
        "see_all_status",
        "fabrication",
        "changementstatut"
      ],
      [
        "see_chemindefer",
        "fabrication",
        "chemindefer"
      ],
      [
        "add_dossier",
        "fabrication",
        "dossier"
      ],
      [
        "change_dossier",
        "fabrication",
        "dossier"
      ],
      [
        "change_prix_dossier",
        "fabrication",
        "dossier"
      ],
      [
        "see_bat_pdf_dossier",
        "fabrication",
        "dossier"
      ],
      [
        "change_logoconstructeur",
        "saisie",
        "logoconstructeur"
      ],
      [
        "delete_logoconstructeur",
        "saisie",
        "logoconstructeur"
      ]
    ]
  },
  "model": "auth.group",
  "pk": 2
},
{
  "fields": {
    "username": "igixxxxx",
    "first_name": "Isxxxxx",
    "last_name": "Gxxxxxx",
    "is_active": true,
    "is_superuser": false,
    "is_staff": true,
    "last_login": "2014-04-04T10:34:28",
    "groups": [
      [
        "Relation Client"
      ]
    ],
    "user_permissions": [],
    "password": "sha1$17783$7152b7f25xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "email": "igibxxxxx@xxxxxxxxx.fr",
    "date_joined": "2009-10-07T16:08:28"
  },
  "model": "auth.user",
  "pk": 8
},
{
  "fields": {
    "organisation": 1,
    "telephone": "01 xx xx xx xx",
    "groups": [
      [
        "Relation Client"
      ]
    ],
    "user_permissions": [],
    "filtre_organisation": false,
    "supports": [
      [
        "PDF"
      ]
    ]
  },
  "model": "parametrage.assistant",
  "pk": 8
}
]

Any idea ?

Thanks.

[stanislas.guerra@…]

Looks like it came from this commit in django/db/models/fields/related.py#ForeignObject.get_instance_value_for_fields(instance, fields) :

​https://github.com/django/django/commit/244e2b71f512605f3d0a8e1ba4c9d6b538acf69d#diff-301
Which is related to the ticket #20820

-                ret.append(instance.pk)
-            else:
-                ret.append(getattr(instance, field.attname))
+                possible_parent_link = opts.get_ancestor_link(field.model)
+                if not possible_parent_link or possible_parent_link.primary_key:
+                    ret.append(instance.pk)
+                    continue
+            ret.append(getattr(instance, field.attname))
         return tuple(ret)

And especially the last line: ret.append(getattr(instance, field.attname)) (<assistant obj>.id in that case) which add a None to the returned Tuple raising an Exception in create_many_related_manager() :

          self.related_val = source_field.get_foreign_related_value(instance)
            if None in self.related_val:
                raise ValueError('"%r" needs to have a value for field "%s" before '
                                 'this many-to-many relationship can be used.' %
                                 (instance, source_field_name))

Because the Assistant object has not been saved yet.

FYI, the fixture loading fails when trying to save the Assistant object (which is a one-to-one to User) because of the M2M user_permissions relation.

Hope this helps.

[stanislas.guerra@…]

Here a patch on the tests that cause a failure.

I just added a proxy model.

$ git diff
diff --git a/tests/fixtures_regress/fixtures/special-article.json b/tests/fixtures_regress/fixtures/special-article.json
index a36244a..75f2d65 100644
--- a/tests/fixtures_regress/fixtures/special-article.json
+++ b/tests/fixtures_regress/fixtures/special-article.json
@@ -2,14 +2,14 @@
     {
         "pk": 1,
         "model": "fixtures_regress.article",
-        "fields": {"title": "foof"
+        "fields": {
+            "title": "foof"
         }
     },
     {
         "pk": 1,
         "model": "fixtures_regress.specialarticle",
         "fields": {
-            "title": "Article Title 1",
             "channels": []
         }
     }
diff --git a/tests/fixtures_regress/models.py b/tests/fixtures_regress/models.py
index 95f9488..e440184 100644
--- a/tests/fixtures_regress/models.py
+++ b/tests/fixtures_regress/models.py
@@ -66,7 +66,14 @@ class Article(models.Model):
 
 
 # Subclass of a model with a ManyToManyField for test_ticket_20820
-class SpecialArticle(Article):
+class CommonSpecialArticle(Article):
+    subtitle = models.CharField(max_length=255)
+
+    class Meta:
+        abstract = True
+
+
+class SpecialArticle(CommonSpecialArticle):
     pass
 

And the failure (tested with sqlite and MySQL):

PYTHONPATH=..:$PYTHONPATH ./runtests.py --settings=test_sqlite fixtures_regress.tests.TestFixtures.test_ticket_20820
Testing against Django installed in '/Users/stan/src/Django/repos/django/django/django'
Creating test database for alias 'default'...
Creating test database for alias 'other'...
E
======================================================================
ERROR: test_ticket_20820 (fixtures_regress.tests.TestFixtures)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/stan/src/Django/repos/django/django/tests/fixtures_regress/tests.py", line 480, in test_ticket_20820
    verbosity=0,
  File "/Users/stan/src/Django/repos/django/django/django/core/management/__init__.py", line 167, in call_command
    return klass.execute(*args, **defaults)
  File "/Users/stan/src/Django/repos/django/django/django/core/management/base.py", line 337, in execute
    output = self.handle(*args, **options)
  File "/Users/stan/src/Django/repos/django/django/django/core/management/commands/loaddata.py", line 60, in handle
    self.loaddata(fixture_labels)
  File "/Users/stan/src/Django/repos/django/django/django/core/management/commands/loaddata.py", line 90, in loaddata
    self.load_label(fixture_label)
  File "/Users/stan/src/Django/repos/django/django/django/core/management/commands/loaddata.py", line 147, in load_label
    obj.save(using=self.using)
  File "/Users/stan/src/Django/repos/django/django/django/core/serializers/base.py", line 176, in save
    setattr(self.object, accessor_name, object_list)
  File "/Users/stan/src/Django/repos/django/django/django/db/models/fields/related.py", line 1185, in __set__
    manager = self.__get__(instance)
  File "/Users/stan/src/Django/repos/django/django/django/db/models/fields/related.py", line 1171, in __get__
    through=self.field.rel.through,
  File "/Users/stan/src/Django/repos/django/django/django/db/models/fields/related.py", line 823, in __init__
    (instance, source_field_name))
ValueError: Problem installing fixture '/Users/stan/src/Django/repos/django/django/tests/fixtures_regress/fixtures/special-article.json': "<SpecialArticle: SpecialArticle object>" needs to have a value for field "article" before this many-to-many relationship can be used.

----------------------------------------------------------------------
Ran 1 test in 0.021s

FAILED (errors=1)
Destroying test database for alias 'default'...
Destroying test database for alias 'other'...

[stanislas.guerra@…]

Patch against master causing a test failure.

[stanislas.guerra@…]

replace the previous patch (removed subtitle field to narrow down the bug).

[stanislas.guerra@…]

Edit:

This is not a proxy model but an abstract one.
And it does not looks like it is related to the natural keys since the regression test fails without them.

[Ramiro Morales]

PR is ready for review at ​https://github.com/django/django/pull/2679

[Ramiro Morales <cramm0@…>]

In 862e1ff2340a1e28a3e7c6904d2b0283085f34c8:

Fixed #22421 -- Regression in fixtures loading.

Loading fixtures were failing since the refactoring in 244e2b71f5 for
inheritance setups where the chain contains abstract models and the
root ancestor contains a M2M relation.

Thanks Stanislas Guerra for the report.

Refs #20946.

[Ramiro Morales <ramiro@…>]

In d731f48ecea6ceeabc2b015b48655a9f9428a730:

Merge pull request #2679 from ramiro/t22421

Fixed #22421 -- Regression in fixtures loading.

[Tim Graham <timograham@…>]

In fb45e666c204d7188c38469ca1f661a9b113b85b:

[1.7.x] Fixed #22421 -- Regression in fixtures loading.

Loading fixtures were failing since the refactoring in 244e2b71f5 for
inheritance setups where the chain contains abstract models and the
root ancestor contains a M2M relation.

Thanks Stanislas Guerra for the report.

Refs #20946.

Backport of 862e1ff234 from master