djorm-ext-filtered-contenttypes impossible on Django 1.9

[Michał Pasternak]

Hi,

I'm the author of a small package, that allows you to filter GenericForeignKey fields using custom query magic introduced in Django 1.7, ​https://github.com/mpasternak/djorm-ext-filtered-contenttypes/ . It works on Django 1.7 and 1.8.

On 1.9 my package is impossible to implement. Reason: ​https://github.com/django/django/commit/9ed82154bd0bd01c6195942db84302e791ad366f which fixed https://code.djangoproject.com/ticket/23791 .

I realize, that the fix for #23791 is very important and I'm glad this bug was solved. I have nothing against it and it's cool that it was implemented; having OneToOne as a primary key seems an interesting use case.

On the other hand, I'd love to be able to use my package on Django 1.9 without too much hackery (there's already some involved).

The reason for GenericForeignKey filtering and for that app of mine... in PostgreSQL you can create a compound index of (content_type_id, object_id) and use it for querying. If I can write Python code like:

StorageRecord.objects.filter(object__in=OtherModel.objects.filter(...))

... and my package can translate it to:

SELECT * FROM app_storage_record WHERE (content_type_id, object_id) IN (SELECT content_type_id, object_id FROM app_other_model WHERE ... ) 

... then PostgreSQL will be able to use compound index on (content_type_id, object_id) and the query will be insanely fast.

I use this to create a large cache table for my app, that holds records from 5 other tables. It uses GenericForeignKey to hold references to those tables. I use my FilteredGenericForeignKey field to filter stuff in that big table. On PostgreSQL with reasonable indexes it is very fast.

If you happen to download my project, just please run tox and see the problem. 1.7 and 1.8 work fine. On 1.9 I get bugs like:

======================================================================
ERROR: test_single_object (filtered_contenttypes.tests.test_fields.TestFilteredContentTypes)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "../filtered_contenttypes/tests/test_fields.py", line 32, in test_single_object
    qry = StorageRecord.objects.filter(item=self.l)
  File "/Users/mpasternak/Programowanie/djorm-ext-filtered-contenttypes/.tox/py35-django19/lib/python3.5/site-packages/django/db/models/manager.py", line 122, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/Users/mpasternak/Programowanie/djorm-ext-filtered-contenttypes/.tox/py35-django19/lib/python3.5/site-packages/django/db/models/query.py", line 790, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "/Users/mpasternak/Programowanie/djorm-ext-filtered-contenttypes/.tox/py35-django19/lib/python3.5/site-packages/django/db/models/query.py", line 808, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
  File "/Users/mpasternak/Programowanie/djorm-ext-filtered-contenttypes/.tox/py35-django19/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1240, in add_q
    clause, _ = self._add_q(q_object, self.used_aliases)
  File "/Users/mpasternak/Programowanie/djorm-ext-filtered-contenttypes/.tox/py35-django19/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1266, in _add_q
    allow_joins=allow_joins, split_subq=split_subq,
  File "/Users/mpasternak/Programowanie/djorm-ext-filtered-contenttypes/.tox/py35-django19/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1174, in build_filter
    self.check_related_objects(field, value, opts)
  File "/Users/mpasternak/Programowanie/djorm-ext-filtered-contenttypes/.tox/py35-django19/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1071, in check_related_objects
    self.check_query_object_type(value, opts, field)
  File "/Users/mpasternak/Programowanie/djorm-ext-filtered-contenttypes/.tox/py35-django19/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1055, in check_query_object_type
    (value, opts.object_name))
ValueError: Cannot query "Laptop object": Must be "StorageRecord" instance.

----------------------------------------------------------------------

[Michał Pasternak]

... on the other hand, I can just set is_relation to False on the field and have it done.

On the other hand, I'd really LOVE someone from core team have a look at my small project. Being able to filter GenericForeignKeys out of the box is not only possible AND efficient (with PostgreSQL and some indexing), it also gives a lot of extra power to the ORM layer.

I think this bug report may be closed as I found solution by looking at the code right after submitting this bug report. So, this bug report was actually useful.

On the other hand, if there's anyone on the other side of the internet, caring about GenericForeignKeys the way I do, please contact me.

[Tim Graham]

Is the requested functionality described in #3006?

[Michał Pasternak]

#3006 is similar but different.

My app is a drop-in replacement for GenericForeignKey, you just replace this with FilteredGenericForeignKey in model definition and you're set. You don't need to use any functions, just the ORM. #3006 uses additional function to achieve that.

Also, I'd need to have a look at queries generated by #3006. My module is built for PostgreSQL and efficiency; if you index both (content_type_id, object_id) fields, my module is going to produce queries that will make PostgreSQL use that index.

Also, I don't support aggregate() and annotate() in my module.

[Tim Graham]

Do you propose to try to incorporate the functionality your package provides in Django in some way. If so, I guess a new ticket for that functionality would be clearer. I'm unsure if you want to keep this ticket open at this point?

[Michał Pasternak]

I had a look at #3006. My package allows you to generate queries like:

SELECT ... WHERE (content_type_id, object_id) = (SELECT (1, id) FROM ...)

... but it does NOT allow you to filter stuff in ORM like the other package does.

So, ATM I'd not incorporate this into Django. Both #3006 and djorm-ext-filtered-contenttypes have their place. Merging those packages in some form of a cool API and adding some docs would make a great addition to Django. ATM I'm not going to work on that, perhaps in the future.

I think this is very good we had this short discussion right here; this functionality is really useful in some corner cases and I guess there won't be many people needing it.