DB Backend cannot specify query class.

[Samuel Bishop]

It would be beneficial if the Queryset class (django.db.models.query.QuerySet) could use some mechanism to find out which query class the database backend wants to use.

Without some mechanism to specify the query class, non sql based database backends are made drastically more complex.

Adding something like "get_query" on the database backends that returns 'sql.Query' in the existing backends would enable new backends to provide their own query class.

[Anssi Kääriäinen]

The problem is that QuerySets aren't tied to any given backend. What should happen if you have two backends, one for mongodb and other for postgres, and the user does User.objects.filter(id=10).using('postgres')? Maybe this could be an error?

I'm not sure what we could do to help nonsql backends. Some sort of User.objects.all() returns nonsql QuerySet seems like a possible addition, but I'm not sure what that mechanism should be.

[Josh Smeaton]

I'm not sure what the solution to this problem is, but after discussing it at pycon with techdragon I agree that it'd be a nice thing to fix.

Here's a couple of things off the top of my head:

1) Meta option per model: Meta: query_backend = 'django.db.models.sql.query'
2) A DATABASES option
3) A new queryset method Model.objects.with('django.db.models.sql.query')

But I also think that we need to define the API between queryset and query, so that actually writing different query backends is realistic. Not a requirement for a solution here though.

[Samuel Bishop]

RE: akaariai
--
The situation between 'comparable' database backends should actually be more like what will happen if you did the same query User.objects.filter(id=10).using('postgres') with a MySQL and PostgreSQL backend. I can't say I've done that myself, but if it raises an error, it should raise an error between a sql and a nonsql backend.
'Non comparable' backends should probably raise an error, something like NotImplemented perhaps.
--
RE: jarshwah
--
1 - This offers little improvement over a custom model that uses the Meta API, and overrides the save method.
2 - This is more like what I'm thinking. While its definitely possible to enhance the flexibility by providing a SETTINGS powered override, I'm not sure that defining the 'default' query for any particular backend in the SETTINGS dictionary is the right way to go.
3 - Just like suggestion 1, this doesn't improve the 'compatibility' situation significantly, beyond what can be done with existing mechanisms such as custom models using the Meta API and overriding their save methods.
--

Ideally with a database backend that supports all the correct methods, we should be able to run Django entirely on top of it, so that means either extensive modifications to contrib.user and other contrib modules, to support some kind of 'optional non sql path', or we need to 'fix' the 'issues' that currently exist with this and related section of code that result in the public API to the ORM mandating that any Django database backend behave like an SQL database.

I'm fairly sure the optimal path is shifting some of this back into the database backend 'layer'. Possibly creating a new "base_sql" class that the existing SQL database backends inherit from instead of db.backends.base and things like defining 'what is my query class' can be set here and shared by all the SQL backends. Otherwise we just implement the new behaviour in each backend.

[Anssi Kääriäinen]

Its also worth noting that there are three ways to implement nonsql backends:

1. Implement Compiler
2. Implement Query (and possibly Compiler)
3. Implement QuerySet (and possibly Query and Compiler)

Currently implementing a different Compiler for nonsql backend is the suggested approach. In principle it should work. In practice, I don't know. I guess the biggest problem is that when doing this, the author is targeting an often changing internal API.

I'm not sure implementing Query is that much better. The API for Query isn't public. How about implementing QuerySet itself? Now that iterator is separated from QuerySet, it isn't that much more work when compared to just implementing Query and/or Compiler.

For .using() incompatibility - I think we should throw an error. And the default backend should be used to decide what is the default queryset type. So, if you have default db = mongodb, other = postgres, then User.objects.filter(foo=bar).using('other') should throw an error, but User.objects.using('other').filter(foo=bar) shouldn't.

It is also worth noting that maybe the right answer is to use different *models* altogether - Django's default user, permission and groups system for example isn't a good design for non-relational databases. If you control the models, then you also control the default queryset type, and we don't have a problem to begin with.

I guess I am +½ to manager .get_queryset() checking the query/queryset class from the default backend, with the idea that objects.using('other') will get you the other backend's query/queryset class.

Finally, I think whatever we come up with here, we should present the solution on django-developers and ask for opinions.

[Mariusz Felisiak]

I don't think there is a way to move it forward without PoC and a discussion on DevelopersMailingList.

[GitHub <noreply@…>]

In 14c8504a:

Refs #25265 -- Allowed Query subclasses to build filters.

[Mariusz Felisiak <felisiak.mariusz@…>]

In 547656c8:

Refs #25265 -- Allowed customizing Query's datastructure classes.