Missing chunk_size throws exception when serializing many-to-many Model field

[Erica Pisani]

Related issue: https://code.djangoproject.com/ticket/35238

I'm currently upgrading a project from v4.x to v5.0.1 and, when attempting to save a model with a prefetched many-to-many field, the "ValueError: chunk_size must be provided when using QuerySet.iterator() after prefetch_related()" exception is thrown within the core Django ​serializer:

Here's the trace within my project that leads to the exception:

File ".../site-packages/django/db/models/base.py", line 822, in save
    self.save_base(
  File ".../site-packages/django/db/models/base.py", line 924, in save_base
    post_save.send(
  File ".../site-packages/django/dispatch/dispatcher.py", line 189, in send
    response = receiver(signal=self, sender=sender, **named)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

# The revisions files below are referring to the django-reversion library

  File ".../site-packages/reversion/revisions.py", line 340, in _post_save_receiver
    add_to_revision(instance, model_db=using)
  File ".../site-packages/reversion/revisions.py", line 209, in add_to_revision
    _add_to_revision(obj, db, model_db, True)
  File ".../site-packages/reversion/revisions.py", line 184, in _add_to_revision
    serialized_data=serializers.serialize(
                    ^^^^^^^^^^^^^^^^^^^^^^
  
File ".../site-packages/django/core/serializers/__init__.py", line 134, in serialize
    s.serialize(queryset, **options)
  File ".../site-packages/django/core/serializers/base.py", line 143, in serialize
    self.handle_m2m_field(obj, field)
  File ".../site-packages/django/core/serializers/python.py", line 91, in handle_m2m_field
    queryset_iterator(obj, field),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../site-packages/django/core/serializers/python.py", line 86, in queryset_iterator
    .iterator()
     ^^^^^^^^^^
  File ".../site-packages/django/db/models/query.py", line 532, in iterator
    raise ValueError(
ValueError: chunk_size must be provided when using QuerySet.iterator() after prefetch_related().

I confirmed that applying the same fix as the issue linked above addresses the issue.

Proposed solution:
I've put up a pull request with the patch which mirrors what was done in the linked issue above (conditionally adding a chunk_size to the .iterator call).

[Tim Graham]

The patch was backported to the stable/5.0.x branch and released in 5.0.3. See ticket:35238#comment:7.

[Erica Pisani]

Hi Tim,

The patch that you're referring to addresses a similar issue, but in a different area of the code.

I opened a [PR ​https://github.com/django/django/pull/18917] with a patch that mirrors the one that you linked if you'd like to see what I'm referring to.

It's still missing tests but I'm looking to add some and would appreciate any guidance on where the best place to do so would be.

[Sarah Boyce]

Have you upgraded to Django 5.1.4 and can you please share the model/model managers so that we can reproduce?

[Erica Pisani]

Hi Sarah,

I've tried upgrading to Django 5.1.4 and the issue still exists (and it still does on the main branch of the Django repository as well).

For steps to reproduce:

We can use these models within the Django test suite in tests/serializers/models/base.py:

class CategoryMetaData(models.Model):
    kind = models.CharField(max_length=10)
    name = models.CharField(max_length=10)
    value = models.CharField(max_length=10)
    objects = CategoryMetaDataManager()

class Category(models.Model):
    name = models.CharField(max_length=20)
    meta_data = models.ForeignKey(
        CategoryMetaData, models.SET_NULL, null=True, default=None
    )

class Article(models.Model):
    author = models.ForeignKey(Author, models.CASCADE)
    headline = models.CharField(max_length=50)
    pub_date = models.DateTimeField()
    categories = models.ManyToManyField(Category)
    meta_data = models.ManyToManyField(CategoryMetaData)
    topics = models.ManyToManyField(Topic)

if test_serialize_prefetch_related_m2m in tests/serializers/tests.py were updated from:

        with self.assertNumQueries(4):
            serializers.serialize(
                self.serializer_name,
                Article.objects.prefetch_related("categories", "meta_data", "topics"),
            )
        with self.assertNumQueries(7):
            serializers.serialize(self.serializer_name, Article.objects.all())

to the following:

        with self.assertNumQueries(4):
            serializers.serialize(
                self.serializer_name,
                Article.objects.prefetch_related(
                    "meta_data",
                    "topics",
                    Prefetch(
                        "categories",
                        queryset=Category.objects.prefetch_related("meta_data"),
            )
        with self.assertNumQueries(7):
            serializers.serialize(self.serializer_name, Article.objects.all())

the ValueError related to missing chunk_size will be raised in the JSON, YAML, and XML serializers.

[Sarah Boyce]

Thank you Erica!

[Sarah Boyce <42296566+sarahboyce@…>]

In 20f9f61:

Fixed #35996 -- Fixed database serialization crash when serializing a many-to-many field that had a prefetch.