Confusing error when ForeignKey lookup fails while serializing

[Kenneth Arnold]

I got a confusing error when running manage.py dumpdata:

Error: Unable to serialize database:

Turns out that a ForeignKey had been marked blank=True but not null=True.

The error message gave no hint as to even which model to look in; the real exception lay hidden behind two levels of error handlers. Maybe serializers should pass on exceptions?

[Etienne Robillard]

You can use --traceback to debug this:

$ bin/debug.sh manage dumpdata -v 2 --traceback foo
/usr/local/lib/python2.6/site-packages/MySQL_python-1.2.2-py2.6-linux-i686.egg/M
ySQLdb/__init__.py:34: DeprecationWarning: the sets module is deprecated
/home/erob/foo/lib/foo/captcha/models.py:8: DeprecationWarni
ng: the md5 module is deprecated; use hashlib instead
  import md5
Traceback (most recent call last):
  File "/home/erob/foo/bin/manage.py", line 7, in <module>
    execute_manager(settings_module)
  File "/home/erob/django.readonly/django/core/management/__init__.py", line 340
, in execute_manager
    utility.execute()
  File "/home/erob/django.readonly/django/core/management/__init__.py", line 295
, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/erob/django.readonly/django/core/management/base.py", line 195, in
 run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/erob/django.readonly/django/core/management/base.py", line 222, in    execute
    output = self.handle(*args, **options)
  File "/home/erob/django.readonly/django/core/management/commands/dumpdata.py",    line 48, in handle
    return serializers.serialize(format, objects, indent=indent)
  File "/home/erob/django.readonly/django/core/serializers/__init__.py", line 86   , in serialize
    s.serialize(queryset, **options)
  File "/home/erob/django.readonly/django/core/serializers/base.py", line 47, in    serialize
    self.handle_fk_field(obj, field)
  File "/home/erob/django.readonly/django/core/serializers/python.py", line 41,    in handle_fk_field
    related = getattr(obj, field.name)
  File "/home/erob/django.readonly/django/db/models/fields/related.py", line 248   , in __get__
    rel_obj = QuerySet(self.field.rel.to).get(**params)
  File "/home/erob/django.readonly/django/db/models/query.py", line 309, in get
    % self.model._meta.object_name)
foo.models.DoesNotExist: Foo matching query does not exist.

Afaik, a simple print statement or two in serializers/python.py would not hurt much and
would help identifying buggy code.

[ajitomatix]

Even something as simple as the following is helpful to find out why the exception is thrown.

raise self.model.DoesNotExist("%s matching query does not exist. args were %s. Lookup parameters were %s"
                    % (self.model._meta.object_name, args, kwargs))

[googol]

or add even more valuable information - like the table name:

raise self.model.DoesNotExist("%s matching query does not exist. Table name: %s. Args were %s. Lookup parameters were %s"

% (self.model._meta.object_name, self.model._meta.db_table, args, kwargs))

[Ryan Witt]

management command to find serialization errors

[Ryan Witt]

Hopefully the command above will help people track down the offending objects. Should work with Django 1.1-1.2.3 at least. There's a gist of it at ​https://gist.github.com/724466 if people feel like forking it, but probably should re-attach improved versions here.

See ​http://docs.djangoproject.com/en/dev/howto/custom-management-commands/ for using the script if you're not familiar with custom commands.

[Dan Loewenherz]

What's the status on this ticket? I'm removing the 1.3 milestone since 1.3 has already been released.

What documentation is needed for this? Perhaps a paragraph explaining the issue?

[Russell Keith-Magee]

The answers you seek can be partially found by reading the ticket history.

The ticket *was* DDN until it was marked "Fixed on a branch" by an anonymous user (who also marked it for milestone 1.3). Since the anonymous person didn't give any explanation, you can probably assume that that update was spam. That means the ticket is waiting on a design decision.

A patch has been provided; that patch isn't a diff against trunk, and it doesn't include tests. It also introduces a new management command, but includes no documentation for that management command. Assuming that the patch is the right fix, the patch isn't ready for trunk because of these missing elements.

However, I'm not convinced the patch is the right approach. We already have a "find the errors" command -- it's called validate. I don't see the use case for an additional management command to catch serialization errors. In any case, blank=True without null=True shouldn't pose a serialization problem, but the reporter doesn't provide a clear test case to validate the problem they describe.

Given that this problem was reported 3 years ago, there have been many improvements in serialization error reporting since that time, and many edge cases caught in serialization since then, I'm going to close this needsinfo. If someone can verify that this is still a problem, feel free to reopen, providing a specific reproduction test case.

[littke]

Even though I'm unable to provide a specific reproduction test case, I can report that I'm having this issue in Django 1.3 with validate returning 0 errors while the dumpdata reports "Error: Unable to serialize database: Package matching query does not exist."

The --traceback command does not help much.

In my case, I believe this occured because of a row with a foreignkey pointing to a parent that was deleted.

[ethanp]

I encountered this exact same problem with Django 1.3 when working with some legacy data. Whilst validate returned no errors, dumpdata reported the error "Space matching query does not exist." The problem was that the legacy data allowed a value of 0 for a blank foreign key, instead of NULL. Django was (logically) unable to match the ID 0 to an existing foreign row. To fix the issue, I updated the legacy data to use NULL instead of 0 for blank foreign keys. I agree that the error message could be more explicit or maybe validate could catch these types of problems?

[anonymous]

Replying to ethanp:

I encountered this exact same problem with Django 1.3 when working with some legacy data. Whilst validate returned no errors, dumpdata reported the error "Space matching query does not exist." The problem was that the legacy data allowed a value of 0 for a blank foreign key, instead of NULL. Django was (logically) unable to match the ID 0 to an existing foreign row. To fix the issue, I updated the legacy data to use NULL instead of 0 for blank foreign keys. I agree that the error message could be more explicit or maybe validate could catch these types of problems?

Thank you!
a simple, bare, <null> (without the brackets) in place of a 0 for null foreign key properties in an import json fixture will also take care of the problem.

[nathank@…]

still a problem in 1.3.1.