Serializer infinite recursion on M2M field if reference vars in init

[Arthur Hanson]

This is a bit of a strange one. serializers.serialize to json will get an infinite recursion error on an M2M field if that M2M field has a custom init method that references two class variables. A quick example below that reproduces the error - create a new django project and create an app called testit and put the following into models.py :

from django.db import models

# Create your models here.
class Tag(models.Model):
    name = models.CharField(max_length=200)
    position = models.IntegerField(default=0)

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

        self._name = self.name
        self._original_position = self.position


class Item(models.Model):
    name = models.CharField(max_length=200)
    position = models.IntegerField(default=0)

    tags = models.ManyToManyField(
        to=Tag,
        related_name='+',
        blank=True
    )

Then a management command that does the serialization:

from django.core.management.base import BaseCommand, CommandError
from testit.models import Item, Tag
from django.core import serializers


class Command(BaseCommand):
    help = ""

    def handle(self, *args, **options):
        tag, created = Tag.objects.get_or_create(name="tag1")
        item, created = Item.objects.get_or_create(name="item1")
        item.tags.add(tag)

        json_str = serializers.serialize('json', [item])
        print(json_str)

When you run the management command the following error is produced produced:

Traceback (most recent call last):
  File "/Users/ahanson/dev/test/m2mjson/m2mjson/manage.py", line 22, in <module>
    main()
  File "/Users/ahanson/dev/test/m2mjson/m2mjson/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/Users/ahanson/dev/test/m2mjson/m2mjson/testit/management/commands/testcmd.py", line 14, in handle
    json_str = serializers.serialize('json', [item])
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/core/serializers/__init__.py", line 134, in serialize
    s.serialize(queryset, **options)
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/core/serializers/base.py", line 167, in serialize
    self.handle_m2m_field(obj, field)
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/core/serializers/python.py", line 93, in handle_m2m_field
    self._current[field.name] = [m2m_value(related) for related in m2m_iter]
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/core/serializers/python.py", line 93, in <listcomp>
    self._current[field.name] = [m2m_value(related) for related in m2m_iter]
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/query.py", line 516, in _iterator
    yield from iterable
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/query.py", line 122, in __iter__
    obj = model_cls.from_db(
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/base.py", line 582, in from_db
    new = cls(*values)
  File "/Users/ahanson/dev/test/m2mjson/m2mjson/testit/models.py", line 11, in __init__
    self._name = self.name
...
 File "/Users/ahanson/dev/test/m2mjson/m2mjson/testit/models.py", line 12, in __init__
    self._original_position = self.position
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/query_utils.py", line 178, in __get__
    instance.refresh_from_db(fields=[field_name])
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/base.py", line 724, in refresh_from_db
    db_instance = db_instance_qs.get()
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/query.py", line 633, in get
    num = len(clone)
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/query.py", line 380, in __len__
    self._fetch_all()
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/query.py", line 1881, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/query.py", line 122, in __iter__
    obj = model_cls.from_db(
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/base.py", line 582, in from_db
    new = cls(*values)
  File "/Users/ahanson/dev/test/m2mjson/m2mjson/testit/models.py", line 11, in __init__
    self._name = self.name
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/query_utils.py", line 178, in __get__
    instance.refresh_from_db(fields=[field_name])
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/base.py", line 724, in refresh_from_db
    db_instance = db_instance_qs.get()
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/query.py", line 633, in get
    num = len(clone)
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/query.py", line 380, in __len__
    self._fetch_all()
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/query.py", line 1881, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/query.py", line 122, in __iter__
    obj = model_cls.from_db(
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/base.py", line 582, in from_db
    new = cls(*values)
  File "/Users/ahanson/dev/test/m2mjson/m2mjson/testit/models.py", line 12, in __init__
    self._original_position = self.position
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/query_utils.py", line 178, in __get__
    instance.refresh_from_db(fields=[field_name])
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/base.py", line 707, in refresh_from_db
    db_instance_qs = self.__class__._base_manager.db_manager(
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/query.py", line 1436, in filter
    return self._filter_or_exclude(False, args, kwargs)
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/query.py", line 1454, in _filter_or_exclude
    clone._filter_or_exclude_inplace(negate, args, kwargs)
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/query.py", line 1461, in _filter_or_exclude_inplace
    self._query.add_q(Q(*args, **kwargs))
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/sql/query.py", line 1545, in add_q
    clause, _ = self._add_q(q_object, self.used_aliases)
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/sql/query.py", line 1576, in _add_q
    child_clause, needed_inner = self.build_filter(
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/sql/query.py", line 1462, in build_filter
    if isinstance(value, Iterator):
  File "/Users/ahanson/.pyenv/versions/3.10.6/lib/python3.10/abc.py", line 119, in __instancecheck__
    return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded in comparison
Exception ignored in: <generator object cursor_iter at 0x102ec03c0>
Traceback (most recent call last):
  File "/Users/ahanson/dev/test/m2mjson/venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 2096, in cursor_iter
    cursor.close()
sqlite3.ProgrammingError: Cannot operate on a closed database.

What is strange is that two variables have to be referenced in the init, if you only reference one it will work fine. The assignment in the init is just for clarity, just the reference to the var will cause the issue:

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

        self.name
        # second one needed to make it fail
        self.position

[Arthur Hanson]

Sample project of model and management command that reproduces the issue

[David Sanders]

Thanks for the report!

Initial investigation monitoring the SQL it goes into a death spiral as it alternates between selecting name & position from the database.

[David Sanders]

Hi Arthur,

Just FYI this appears to make it work if you're looking for an immediate fix:

serializers.serialize("json", Item.objects.prefetch_related('tags'))

The docs state that serialize() expects a queryset rather than a material list, however, without prefetching we get the same infinite recursion… so I'd wager this will remain accepted.

EDIT: I doubt there's something that can be done to fix this other than warn in the docs about the dangers of accessing deferred field attributes during initialisation. Messing with the model during initialisation is ​warned against in the docs, though for other reasons.

[Simon Charette]

I agree with your assessment David, this came up when adjusting cascade deletion to limit the number of fields that get selected (#30191).

In order for your model definition __init__ override to adequately support field deferral, which is a feature the serialization framework make use of, you must use self.__dict__.get(field_name) to retrieve possibly deferred values.

[David Sanders]

Moved keywords from cc to keywords ;)