LiveServerTestCase fails when query expressions are used for model ordering

[Gergely Kalmár]

Reproducing the bug

Create a models.py file as follows:

from django.db import models
from django.db.models.functions import Lower


class Order(models.Model):
    id = models.CharField(max_length=10, primary_key=True)

    class Meta:
        ordering = [Lower('id')]


class Offer(models.Model):
    order = models.OneToOneField(Order, primary_key=True, on_delete=models.PROTECT)

Create a test case:

from django.contrib.staticfiles.testing import LiveServerTestCase


class Test(LiveServerTestCase):
    def test_case(self):
        pass

Run the tests with ./manage.py test:

> ./manage.py test
Creating test database for alias 'default'...
Traceback (most recent call last):
  File "./manage.py", line 17, in <module>
    main()
  File "./manage.py", line 13, in main
    execute_from_command_line(sys.argv)
  File ".../lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File ".../lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File ".../lib/python3.8/site-packages/django/core/management/commands/test.py", line 23, in run_from_argv
    super().run_from_argv(argv)
  File ".../lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File ".../lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File ".../lib/python3.8/site-packages/django/core/management/commands/test.py", line 55, in handle
    failures = test_runner.run_tests(test_labels)
  File ".../lib/python3.8/site-packages/django/test/runner.py", line 725, in run_tests
    old_config = self.setup_databases(aliases=databases)
  File ".../lib/python3.8/site-packages/django/test/runner.py", line 643, in setup_databases
    return _setup_databases(
  File ".../lib/python3.8/site-packages/django/test/utils.py", line 179, in setup_databases
    connection.creation.create_test_db(
  File ".../lib/python3.8/site-packages/django/db/backends/base/creation.py", line 90, in create_test_db
    self.connection._test_serialized_contents = self.serialize_db_to_string()
  File ".../lib/python3.8/site-packages/django/db/backends/base/creation.py", line 136, in serialize_db_to_string
    serializers.serialize("json", get_objects(), indent=None, stream=out)
  File ".../lib/python3.8/site-packages/django/core/serializers/__init__.py", line 129, in serialize
    s.serialize(queryset, **options)
  File ".../lib/python3.8/site-packages/django/core/serializers/base.py", line 90, in serialize
    for count, obj in enumerate(queryset, start=1):
  File ".../lib/python3.8/site-packages/django/db/backends/base/creation.py", line 133, in get_objects
    yield from queryset.iterator()
  File ".../lib/python3.8/site-packages/django/db/models/query.py", line 353, in _iterator
    yield from self._iterable_class(self, chunked_fetch=use_chunked_fetch, chunk_size=chunk_size)
  File ".../lib/python3.8/site-packages/django/db/models/query.py", line 51, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File ".../lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1162, in execute_sql
    sql, params = self.as_sql()
  File ".../lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 513, in as_sql
    extra_select, order_by, group_by = self.pre_sql_setup()
  File ".../lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 56, in pre_sql_setup
    order_by = self.get_order_by()
  File ".../lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 372, in get_order_by
    resolved = expr.resolve_expression(self.query, allow_joins=True, reuse=None)
  File ".../lib/python3.8/site-packages/django/db/models/expressions.py", line 247, in resolve_expression
    c.set_source_expressions([
  File ".../lib/python3.8/site-packages/django/db/models/expressions.py", line 248, in <listcomp>
    expr.resolve_expression(query, allow_joins, reuse, summarize)
  File ".../lib/python3.8/site-packages/django/db/models/expressions.py", line 678, in resolve_expression
    c.source_expressions[pos] = arg.resolve_expression(query, allow_joins, reuse, summarize, for_save)
  File ".../lib/python3.8/site-packages/django/db/models/expressions.py", line 578, in resolve_expression
    return query.resolve_ref(self.name, allow_joins, reuse, summarize)
  File ".../lib/python3.8/site-packages/django/db/models/sql/query.py", line 1754, in resolve_ref
    join_info = self.setup_joins(field_list, self.get_meta(), self.get_initial_alias(), can_reuse=reuse)
  File ".../lib/python3.8/site-packages/django/db/models/sql/query.py", line 1623, in setup_joins
    path, final_field, targets, rest = self.names_to_path(
  File ".../lib/python3.8/site-packages/django/db/models/sql/query.py", line 1537, in names_to_path
    raise FieldError("Cannot resolve keyword '%s' into field. "
django.core.exceptions.FieldError: Cannot resolve keyword 'id' into field. Choices are: order, order_id

If there is any workaround that I can use in the meantime please let me know. Thank you!

[Tim Graham]

We'll have to figure out exactly what ORM query is being constructed here. The error doesn't happen on Django's main branch after 3089018e951dcca568574c0e0ebf9d8aab112389 but I'd guess the underlying problem still exists.

[Simon Charette]

You'll want to make sure to generate migrations for the app containing these models otherwise you won't be able to reproduce.

This bug is two fold. First we likely just want to have the querysets generated during serialization order by the primary key without involving possible related object semantic

diff --git a/django/db/backends/base/creation.py b/django/db/backends/base/creation.py
index 81cb34bd9f..f49c31edaf 100644
--- a/django/db/backends/base/creation.py
+++ b/django/db/backends/base/creation.py
@@ -129,7 +129,7 @@ def get_objects():
                         ):
                             queryset = model._base_manager.using(
                                 self.connection.alias,
-                            ).order_by(model._meta.pk.name)
+                            ).order_by('pk')
                             yield from queryset.iterator()
         # Serialize to a string
         out = StringIO()

The above happens to address the reported crash but it doesn't deal with the fundamental issue which is that Referent.objects.order_by('reference') crashes if Referred.Meta.ordering contains expressions because they won't be transposed to the origin of the queryset (e.g. Lower('id') -> Lower('order__id') in the reported case). It can be reproduced with the same set of models and the following test

from django.test import TestCase

from .models import Offer

class Test(TestCase):
    def test_case(self):
        list(Offer.objects.order_by('order'))

The issue lies in SQLCompiler.find_ordering_name but ​is not trivial to address as we'd need a way to prefix all field references in an Expression so I'd suggest we raise a NotImplementedError instead for now. ​Not sure when this path is covered by the suite but I would assume it would only work when MTI inheritance is involved and referenced field references don't need to be prefixed as they are inherited?

[Gergely Kalmár]

Thank you for the help! It seems that I can work around this by setting

DATABASES = {
    'default': {
        ...
        'TEST': {'SERIALIZE': False}
    }
}

which is fine for now, I think. It would be great if the recommended patch for at least fixing the immediate issue could be added though in case we'd need tests to be serialized for some reason in the future.

[Mariusz Felisiak]

Fixed in 2798c937deb6625a4e6a36e70d4d60ce5faac954 (#29538).