AlterField migration on ForeignKey column fails on fresh DB when referenced model has been subsequently modified

[djangobugreport]

I have run into a specific sequence of migrations that work one by one but fail when applied in sequence. The sequence is as follows (generate and apply migrations after each below step to reproduce the bug):

1. Create two models with one referencing the other like so:

class Foo(models.Model):
    hoge = models.IntegerField(unique = True)

class Bar(models.Model):
    foo = models.ForeignKey(Foo, on_delete = models.SET_NULL, null = True)

2. Change the ForeignKey field on the referencing model to pull a default instance from a python function:

class Foo(models.Model):
    hoge = models.IntegerField(unique = True)

def get_default_foo():
    foo, created = Foo.objects.get_or_create(
        hoge = 0
    );
    return foo.id

class Bar(models.Model):
    #foo = models.ForeignKey(Foo, on_delete = models.SET_NULL, null = True)
    foo = models.ForeignKey(Foo, on_delete = models.SET_DEFAULT, default = get_default_foo)

3. Add a new field to the referenced model:

class Foo(models.Model):
    hoge = models.IntegerField(unique = True)
    piyo = models.IntegerField(default = 0)

def get_default_foo():
    foo, created = Foo.objects.get_or_create(
        hoge = 0
    );
    return foo.id

class Bar(models.Model):
    #foo = models.ForeignKey(Foo, on_delete = models.SET_NULL, null = True)
    foo = models.ForeignKey(Foo, on_delete = models.SET_DEFAULT, default = get_default_foo)

Now, delete the DB and try to recreate from scratch, and the migration created in the second step above will fail complaining that the column "piyo" does not exist. Evidently, the migration runtime is not correctly referencing the old version of Foo without the piyo field, but rather the current version in the source code. Here is my stack trace from Visual Studio:

Operations to perform:
  Apply all migrations: TheApp, admin, auth, contenttypes, sessions
Running migrations:
  Applying TheApp.0001_initial... OK
  Applying TheApp.0002_auto_20210114_1330...Traceback (most recent call last):
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\sqlite3\base.py", line 413, in execute
    return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: no such column: TheApp_foo.piyo

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\sandbox\django\three\DjangoWebProject1\manage.py", line 17, in <module>
    execute_from_command_line(sys.argv)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\core\management\__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\core\management\__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\core\management\base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\core\management\base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\core\management\base.py", line 85, in wrapped
    res = handle_func(*args, **kwargs)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\core\management\commands\migrate.py", line 245, in handle
    fake_initial=fake_initial,
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\migrations\executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\migrations\executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\migrations\executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\migrations\migration.py", line 124, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\migrations\operations\fields.py", line 236, in database_forwards
    schema_editor.alter_field(from_model, from_field, to_field)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\sqlite3\schema.py", line 138, in alter_field
    super().alter_field(model, old_field, new_field, strict=strict)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\base\schema.py", line 572, in alter_field
    old_db_params, new_db_params, strict)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\sqlite3\schema.py", line 360, in _alter_field
    self._remake_table(model, alter_field=(old_field, new_field))
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\sqlite3\schema.py", line 200, in _remake_table
    'default': self.quote_value(self.effective_default(new_field))
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\base\schema.py", line 303, in effective_default
    return field.get_db_prep_save(self._effective_default(field), self.connection)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\base\schema.py", line 282, in _effective_default
    default = field.get_default()
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\models\fields\related.py", line 960, in get_default
    field_default = super().get_default()
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\models\fields\__init__.py", line 831, in get_default
    return self._get_default()
  File ".\TheApp\models.py", line 11, in get_default_foo
    hoge = 0
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\models\manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\models\query.py", line 573, in get_or_create
    return self.get(**kwargs), False
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\models\query.py", line 425, in get
    num = len(clone)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\models\query.py", line 269, in __len__
    self._fetch_all()
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\models\query.py", line 1308, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\models\query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\models\sql\compiler.py", line 1156, in execute_sql
    cursor.execute(sql, params)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\utils.py", line 98, in execute
    return super().execute(sql, params)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "D:\sandbox\django\three\DjangoWebProject1\env\lib\site-packages\django\db\backends\sqlite3\base.py", line 413, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such column: TheApp_foo.piyo

I cannot see any obvious way to work around this problem without diving into the internals of the migration engine. Similar migrations work fine, it's only this combination of changing "on_delete" and "default" in this way that I've seen cause the problem. Although this isn't personally blocking a release for me, I've marked this bug as "Release Blocker" as IMO it is fairly severe because it could easily block the update of a production database if the above migrations had to be applied in sequence. (It would be possible to rewind the commit history and apply the migrations one by one along with the accompanying source code, but this could potentially be a huge amount of work, especially if dealing with an automated deployment system, etc.).

I have attached a zip containing a test Visual Studio Django project with the code above. I simply created an empty project with the Visual Studio Django wizard, updated requirements.txt to the most recent version of Django (3.1.5), changed the parts of the default project that caused errors on Django 3, and then wrote the test case above. There's nothing Visual Studio centric about this problem so it should reproduce on any platform as far as I can tell.

If there is a workaround for this bug, please advise. Thank you.

[djangobugreport]

minimal reproduction for ticket #32351 migration bug

[djangobugreport]

After some more tinkering I've found the following work around:

1. Modify the "default" argument of the models.ForeignKey call in the second generated migration to be some invalid ID value, like -1:

field=models.ForeignKey(default=TheApp.models.get_default_foo, on_delete=django.db.models.deletion.SET_DEFAULT, to='TheApp.foo'),
# change to
field=models.ForeignKey(default=-1, on_delete=django.db.models.deletion.SET_DEFAULT, to='TheApp.foo'),

2. Generate a new fourth migration, which will run after the "piyo" field has been added, and change the "default" setting back to the callback function.

This works for me although it will have to be done again every time the model Foo changes.

I have also changed the level of this ticket to Normal since I noticed your FAQ states that "Release blocker" is for things blocking a release of Django (I had assumed it was for things blocking the release of user projects).

[Mariusz Felisiak]

Thanks for this report, however IMO it's not a supported to use callables that create new objects in the ​Field.default. Django evaluates default before applying a migration, in your case there is a mismatch between model and table states. To avoid this issue in the RunPython() operation you can get the model from the versioned app registry, e.g. apps.get_model("theapp", "Foo"). I'm not sure if it will work in the Field.default.