Add the option in the prefetch_related to have the prefetched results returned as a list or as a queryset

[Arnaud B]

In his answer in ticket #26676, Simon Charette suggested me to open this ticket.

Currently, the result stored by prefetch_related is by default a queryset, unless Prefetch with to_attr is used, in which case it is a list. According to the ​documentation about Prefetch, this trades the flexibility offered by the queryset for a considerable performance improvement.

I can see the rationale to have this behavior by default, but it seems to me it would be nicer to have an option to explicitly declare whether the results should be returned as a queryset or as a list:

1. In many cases, there is no need to chain further queries to the default prefetch_related. By explicitly asking for a list it would then become possible to benefit from the speed improvement.
2. Conversely, some developers may be happy to take the performance hit in order to further refine a Prefetch with to_attr.

This could be done via an optional boolean in prefetch_related, is_chainable or force_queryset or something like this (if left unset the current default behavior would apply). Does that sound like a good idea?

[Tim Graham]

Simon, care to add your thoughts?

[Simon Charette]

Personally I'm -0 on adding such a feature for the following reasons:

In many cases, there is no need to chain further queries to the default prefetch_related. By explicitly asking for a list it would then become possible to benefit from the speed improvement.

The default prefetch_related() results are stored in a QuerySet() in order to make sure the returned instances' prefetched relations still behave the same as if they were not prefetched. Storing lists in there would make such instances/querysets unusable by many components of Django such as forms and model level validation where it would fail in non-obvious ways.

Using Prefetch('relation', to_attr='relation_as_list') already serves this purpose and don't think introducing another way to do it such as Prefetch('relation', as_list=True) is worth it. If we'd really want to speed up default prefetches I think we should investigate assigning objects that quack like QuerySet (e.g. PrefetchedQuerySet) but only lazily build their underlying query when required as this is the part that slow down performance here.

Conversely, some developers may be happy to take the performance hit in order to further refine a Prefetch with to_attr.

I've never personally required to refine a prefetched result, that sounds like an anti-pattern to me.

[Simon Charette]

FWIW the PrefetchedQuerySet idea was already suggested 3 years ago by Anssi (#20577).