Got `django.core.exceptions.FieldError` when adding GeneratedField

[wd0517]

Here is an exists model

class Foo1(models.Model):
    age = models.IntegerField()

Then adding GeneratedField

class Foo1(models.Model):
    age = models.IntegerField()
    name = models.CharField(max_length=10)
    lower_name = models.GeneratedField(
        expression=Lower("name"),
        output_field=models.CharField(max_length=11),
        db_persist=True,
    )

Run makemigrations get below file

class Migration(migrations.Migration):

    dependencies = [
        ('temp_django_test', '0006_foo1'),
    ]

    operations = [
        migrations.AddField(
            model_name='foo1',
            name='lower_name',
            field=models.GeneratedField(db_persist=True, expression=django.db.models.functions.text.Lower('name'), output_field=models.CharField(max_length=11)),
        ),
        migrations.AddField(
            model_name='foo1',
            name='name',
            field=models.CharField(default='a', max_length=10),
            preserve_default=False,
        ),
    ]

Then run migrate, will got error

temp-django-5.0 ❯ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions, temp_django_test
Running migrations:
  Applying temp_django_test.0007_foo1_lower_name_foo1_name...Traceback (most recent call last):
  File "/Users/wangdi/Desktop/test_scripts.nosync/temp-django-5.0/manage.py", line 22, in <module>
    main()
  File "/Users/wangdi/Desktop/test_scripts.nosync/temp-django-5.0/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/Users/wangdi/Desktop/test_scripts.nosync/temp-django-5.0/.venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/Users/wangdi/Desktop/test_scripts.nosync/temp-django-5.0/.venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/wangdi/Desktop/test_scripts.nosync/temp-django-5.0/.venv/lib/python3.11/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Users/wd0517/Desktop/test_scripts.nosync/temp-django-5.0/.venv/lib/python3.11/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wd0517/Desktop/test_scripts.nosync/temp-django-5.0/.venv/lib/python3.11/site-packages/django/core/management/base.py", line 107, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wd0517/Desktop/test_scripts.nosync/temp-django-5.0/.venv/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
                         ^^^^^^^^^^^^^^^^^
  File "/Users/wd0517/Desktop/test_scripts.nosync/temp-django-5.0/.venv/lib/python3.11/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wd0517/Desktop/test_scripts.nosync/temp-django-5.0/.venv/lib/python3.11/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wd0517/Desktop/test_scripts.nosync/temp-django-5.0/.venv/lib/python3.11/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wd0517/Desktop/test_scripts.nosync/temp-django-5.0/.venv/lib/python3.11/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/Users/wd0517/Desktop/test_scripts.nosync/temp-django-5.0/.venv/lib/python3.11/site-packages/django/db/migrations/operations/fields.py", line 108, in database_forwards
    schema_editor.add_field(
  File "/Users/wd0517/Desktop/test_scripts.nosync/temp-django-5.0/.venv/lib/python3.11/site-packages/django/db/backends/mysql/schema.py", line 103, in add_field
    super().add_field(model, field)
  File "/Users/wd0517/Desktop/test_scripts.nosync/temp-django-5.0/.venv/lib/python3.11/site-packages/django/db/backends/base/schema.py", line 709, in add_field
    definition, params = self.column_sql(model, field, include_default=True)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wd0517/Desktop/test_scripts.nosync/temp-django-5.0/.venv/lib/python3.11/site-packages/django/db/backends/base/schema.py", line 369, in column_sql
    " ".join(
  File "/Users/wd0517/Desktop/test_scripts.nosync/temp-django-5.0/.venv/lib/python3.11/site-packages/django/db/backends/base/schema.py", line 336, in _iter_column_sql
    generated_sql, generated_params = self._column_generated_sql(field)
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wd0517/Desktop/test_scripts.nosync/temp-django-5.0/.venv/lib/python3.11/site-packages/django/db/backends/base/schema.py", line 434, in _column_generated_sql
    expression_sql, params = field.generated_sql(self.connection)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wd0517/Desktop/test_scripts.nosync/temp-django-5.0/.venv/lib/python3.11/site-packages/django/db/models/fields/generated.py", line 58, in generated_sql
    resolved_expression = self.expression.resolve_expression(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wd0517/Desktop/test_scripts.nosync/temp-django-5.0/.venv/lib/python3.11/site-packages/django/db/models/expressions.py", line 975, in resolve_expression
    c.source_expressions[pos] = arg.resolve_expression(
                                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wd0517/Desktop/test_scripts.nosync/temp-django-5.0/.venv/lib/python3.11/site-packages/django/db/models/expressions.py", line 854, in resolve_expression
    return query.resolve_ref(self.name, allow_joins, reuse, summarize)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wd0517/Desktop/test_scripts.nosync/temp-django-5.0/.venv/lib/python3.11/site-packages/django/db/models/sql/query.py", line 2014, in resolve_ref
    join_info = self.setup_joins(
                ^^^^^^^^^^^^^^^^^
  File "/Users/wd0517/Desktop/test_scripts.nosync/temp-django-5.0/.venv/lib/python3.11/site-packages/django/db/models/sql/query.py", line 1867, in setup_joins
    path, final_field, targets, rest = self.names_to_path(
                                       ^^^^^^^^^^^^^^^^^^^
  File "/Users/wd0517/Desktop/test_scripts.nosync/temp-django-5.0/.venv/lib/python3.11/site-packages/django/db/models/sql/query.py", line 1772, in names_to_path
    raise FieldError(
django.core.exceptions.FieldError: Cannot resolve keyword 'name' into field. Choices are: age, id, lower_name

Environment:

• Python 3.11.8
• Django 5.0.4
• MySQL 8.0.34

I think the bug is caused by the order of operations in Migration, after adjusting it, then can be migrate successfully

[Simon Charette]

The auto-detector has to be adapted to make the AddField of generated field depend on any other AddField or AlterField of field it references through .expression. Since GeneratedField cannot depend on other GeneratedField the easier way of doing to avoid introspecting .expression is likely to make them depend on all other non-GeneratedField fields being added per model so they are always last.

[David Sanders]

For anyone picking up this ticket: Simon & I were discussing ways to get fields from expressions in this PR: ​https://github.com/django/django/pull/16054

[Simon Charette]

Thanks David, I knew this was implemented somewhere but I lost track of where exactly!

I think that relying on this property once it lands is definitely the way to go so it's reused by multiple code paths but in the case of stable/5.x we might be better off by always adding them last. Thoughts?

[David Sanders]

I think that relying on this property once it lands is definitely the way to go so it's reused by multiple code paths but in the case of stable/5.x we might be better off by always adding them last. Thoughts?

Hm can't fks depend on these though because the constraints for those are also grouped together last. 🤔

[Simon Charette]

Hm can't fks depend on these though because the constraints for those are also grouped together last. 🤔

Right. Since GeneratedField cannot be foreign keys though it should still allow for the following grouping of fields?

non_fk_non_generated_field | fk_fields | generated_fields

[Bhuvnesh]

​PR

[Sarah Boyce <42296566+sarahboyce@…>]

In 97d48cd:

Refs #34007, Refs #35359 -- Added Q.referenced_based_fields property.

Thank you to Mariusz Felisiak and Natalia Bidart for the reviews.

[nessita <124304+nessita@…>]

In 9aeb38c2:

Fixed #35359 -- Fixed migration operations ordering when adding fields referenced by GeneratedField.expression.

Thank you to Simon Charette for the review.

[Natalia <124304+nessita@…>]

In fa202d5c:

[5.0.x] Refs #34007, Refs #35359 -- Added Q.referenced_based_fields property.

Thank you to Mariusz Felisiak and Natalia Bidart for the reviews.

Backport of 97d48cd3c6f409584b5cc19fbddfca917bae78fd from main

[Natalia <124304+nessita@…>]

In 24f54c3:

[5.0.x] Fixed #35359 -- Fixed migration operations ordering when adding fields referenced by GeneratedField.expression.

Thank you to Simon Charette for the review.

Backport of 9aeb38c296c69532c7e64b5e3d706a5eb17b3f12 from main.

[nessita <124304+nessita@…>]

In ac9e18f:

[5.0.x] Refs #35359 -- Fixed OperationTests.test_add_generate_field() test on PostgreSQL.

Concat() in Django 5.0 is not immutable on PostgreSQL and cannot be used
in GeneratedField, see 6364b6ee1071381eb3a23ba6b821fc0d6f0fce75.