GenericRelation and prefetch_related: wrong caching with cyclic prefetching.

[Finn Stutzenstein]

Hello @all!

I encountered an issue with GenericRelations. Here is an example to reproduce this issue: ​https://github.com/FinnStutzenstein/GenericRelatedPrefetch
Just do a migrate and runserver. The code showing the error is started automatically in main/apps.py. Please start with --noreload not to have the output twice.

Whats the problem?
I have a generic model (Tag) that have a content_object. Then there are multiple (in the example 2) models, Book and CD, that have exactly one tag assigned. In the real application (OpenSlides) this invariant is ensured in other places; in the example the objects are created in a way, that this invariant holds.

All these content objects have a property tag, that should return the one assigned tag. This is done by adding a helper field tags=GenericRelation(Tag) and the property accesses self.tags.all()[0]. The .all()[0] instead of a simple .get() is required for the prefetching to work. See main/models.py in the example.

Now all tags should be loaded because in OpenSlides they would be serialized. For each tag the content_object is accessed as well as content_object.tag. Because this would result in many DB queries (in the real application about 10000 Tags, and in sum 10000 content objects) the models are prefetched with: Tag.objects.prefetch_related("content_object", "content_object__tag") (Note: The executed code is in main/apps.py). This results in a constant amount of queries (4 in this case) instead of something proportional to the amount of objects. In the example you can set N, the amount of objects created, to a higher amount to verify, that the amount of queries stays constant.

What is expected: If I have a tag tag, tag.content_object.tag should be equal to tag.
Output from the example (with N=1):

Got 'Tag to book0':
    -the content object: Book0
    -the content objects tag (should be the same as 'Tag to book0'!):Tag to book0
Got 'Tag to cd0':
    -the content object: CD0
    -the content objects tag (should be the same as 'Tag to cd0'!):Tag to book0

This is not the case: 'Tag to cd1' -> 'cd1' -> 'Tag to book1'.

I tracked this a bit showing, that _prefetched_objects_cache holds the wrong value, which is accessed through .all() -> .get_queryset() where the cached/prefetched result is taken.

Thanks!

[Mariusz Felisiak]

Thanks for the report. This is definitely related with prefetch_related(), because get() and first() works properly.

Reproduced at 519016e5f25d7c0a040015724f9920581551cab0.

[Can Sarıgöl]

​PR

[Mariusz Felisiak <felisiak.mariusz@…>]

In f66021f3:

Refs #30493 -- Added GenericRelatedObjectManager.get_content_type() hook.

[Mariusz Felisiak <felisiak.mariusz@…>]

In dffa3e19:

Fixed #30493 -- Fixed prefetch_related() for GenericRelation with different content types.

Co-Authored-By: Mariusz Felisiak <felisiak.mariusz@…>

Thanks Simon Charette for the review.

[Finn Stutzenstein]

Thanks to Can Sarıgöl, felixxm and charettes for the fix! Great work!