Migration crash when renaming a field referenced in GeneratedField.expression

[Sarah Boyce]

Given a model (migrated) on SQLite

class Foo(models.Model):
    name = models.CharField(max_length=10)
    lower_name = models.GeneratedField(
        expression=Lower("name"),
        output_field=models.CharField(max_length=10),
        db_persist=True,
    )

when you update to

class Foo(models.Model):
    surname = models.CharField(max_length=10)
    lower_name = models.GeneratedField(
        expression=Lower("surname"),
        output_field=models.CharField(max_length=10),
        db_persist=True,
    )

and makemigrations, you then get asked Was foo.name renamed to foo.surname (a CharField)? [y/N] and say y (because it was)
You then get a crash when migrating:

Operations to perform:
  Apply all migrations: app3
Running migrations:
  Applying app3.0005_rename_name_foo_surname_alter_foo_lower_name...Traceback (most recent call last):
  File "path_to_project\mysite\manage.py", line 22, in <module>
    main()
  File "path_to_project\mysite\manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "path_to_project\venv\Lib\site-packages\django\core\management\__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "path_to_project\venv\Lib\site-packages\django\core\management\__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "path_to_project\venv\Lib\site-packages\django\core\management\base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "path_to_project\venv\Lib\site-packages\django\core\management\base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path_to_project\venv\Lib\site-packages\django\core\management\base.py", line 107, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path_to_project\venv\Lib\site-packages\django\core\management\commands\migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
                         ^^^^^^^^^^^^^^^^^
  File "path_to_project\venv\Lib\site-packages\django\db\migrations\executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path_to_project\venv\Lib\site-packages\django\db\migrations\executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
            ^^^^^^^^^^^^^^^^^^^^^
  File "path_to_project\venv\Lib\site-packages\django\db\migrations\executor.py", line 255, in apply_migration
    state = migration.apply(state, schema_editor)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path_to_project\venv\Lib\site-packages\django\db\migrations\migration.py", line 132, in apply
    operation.database_forwards(
  File "path_to_project\venv\Lib\site-packages\django\db\migrations\operations\fields.py", line 241, in database_forwards
    schema_editor.alter_field(from_model, from_field, to_field)
  File "path_to_project\venv\Lib\site-packages\django\db\backends\base\schema.py", line 875, in alter_field
    or old_field.generated_sql(self.connection)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path_to_project\venv\Lib\site-packages\django\db\models\fields\generated.py", line 58, in generated_sql
    resolved_expression = self.expression.resolve_expression(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path_to_project\venv\Lib\site-packages\django\db\models\expressions.py", line 1052, in resolve_expression
    c.source_expressions[pos] = arg.resolve_expression(
                                ^^^^^^^^^^^^^^^^^^^^^^^
  File "path_to_project\venv\Lib\site-packages\django\db\models\expressions.py", line 874, in resolve_expression
    return query.resolve_ref(self.name, allow_joins, reuse, summarize)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path_to_project\venv\Lib\site-packages\django\db\models\sql\query.py", line 2010, in resolve_ref
    join_info = self.setup_joins(
                ^^^^^^^^^^^^^^^^^
  File "path_to_project\venv\Lib\site-packages\django\db\models\sql\query.py", line 1863, in setup_joins
    path, final_field, targets, rest = self.names_to_path(
                                       ^^^^^^^^^^^^^^^^^^^
  File "path_to_project\venv\Lib\site-packages\django\db\models\sql\query.py", line 1768, in names_to_path
    raise FieldError(
django.core.exceptions.FieldError: Cannot resolve keyword 'name' into field. Choices are: id, lower_name, surname

[Natalia Bidart]

Thank you Sarah, I reproduced and confirmed that this is not a dupe of #35359 (or at least the fix for that ticket is not fixing this issue).

This is a release blocker for 5.0 given that GeneratedField was added in that Django version. I also reproduced in f333e3513e8bdf5ffeb6eeb63021c230082e6f95.

[Simon Charette]

I don't think the problem is specific to SQLite.

The non-invasive back portable solution I can think of is for FieldError to be caught and also result in a ValueError ​pointing out that the generated field must be removed first.

In other words, if we can't generate the SQL for the old field anymore but we can for the new one then it was certainly changed and the same lack of alteration support limitation applies.

[Sarah Boyce]

In other words, if we can't generate the SQL for the old field anymore but we can for the new one then it was certainly changed and the same lack of alteration support limitation applies.

Makes sense! Agree that I found this on SQLite is irrelevant

[Mariusz Felisiak]

​PR

[Sarah Boyce <42296566+sarahboyce@…>]

In 91a4b9a8:

Fixed #35422 -- Fixed migrations crash when altering GeneratedField referencing rename field.

Thanks Sarah Boyce for the report and Simon Charette for the
implementation idea.

[Sarah Boyce <42296566+sarahboyce@…>]

In c544f1a:

[5.0.x] Fixed #35422 -- Fixed migrations crash when altering GeneratedField referencing rename field.

Thanks Sarah Boyce for the report and Simon Charette for the
implementation idea.

Backport of 91a4b9a8ec2237434f06866f39c7977e889aeae6 from main.

[nessita <124304+nessita@…>]

In e72049aa:

Refs #35422 -- Fixed typo in docs/releases/5.0.5.txt.

[Natalia <124304+nessita@…>]

In e18e931:

[5.0.x] Refs #35422 -- Fixed typo in docs/releases/5.0.5.txt.

Backport of e72049aa6302411d8cdf2e798e15fa38e76b92fc from main.