Squash migration is not marked as applied when the migrations it replaces are

[Martin Häcker]

(I am attempting a shorter and clearer description of this bug, but leaving the original description intact below - carljm).

In Django 1.8, consider an app A with migrations 1 and 2 and a squashed migration 1_squashed_2 that replaces both 1 and 2. Consider a deployment of this app which has only 1 applied. When it receives the update including 2 and 1_squashed_2 and is migrated, 2 is marked as applied in the database but 1_squashed_2 is not.

This does not immediately appear to be a problem, because as long as migrations 1 and 2 exist and 1_squashed_2 is marked as replacing them, Django automatically considers 1_squashed_2 to be applied (and shows it as such in showmigrations). But Django never actually records 1_squashed_2 itself as applied in the database.

At some point, once all deployments have migrated through 2, the idea is that 1 and 2 can be removed, and the replaces tag removed from 1_squashed_2. At this point we have a problem, because now Django considers 1_squashed_2 to not be applied, and tries to apply it again, causing errors because tables already exist, etc. The only solution is to manually --fake squashed migrations on all deployments when their replaces tag is removed, which is a pain and eliminates much of the convenience of the migration system; --fake should not be necessary in the normal course of things.

The solution appears simple: anytime a migration is marked as having been applied, Django should check if it is part of a replaced set, and if that replaced set is now fully applied, the replacing migration should also be marked as applied.

It may be that this check should be performed by migrate even when it hasn't applied any migrations in the current run, as that would help to correct the corruption already caused by this bug in many existing databases, but not yet noticed because the replaced migrations haven't yet been removed. (This correction should probably be mentioned to the user so it doesn't happen silently.)

Original report follows:

After some time in the chat with MarkusH, we decided that this should be a bug report.

Some context: I have quite some migrations (33) of which the first already was a squash of several migrations. It did get it's 'replaced' field removed though, as the original migrations it replaced are long gone already. The goal now was to get that long list of migrations down to one again, for peace of mind.

This led to the following symptoms: When I created the squashed migration everything seemed fine. ./manage.py migrate did say that nothing was to be done (but then again, all the migrations to be squashed where already applied).

But adding another migration after that ended in tears - ./manage migrate didn't want to apply it and told me so in no uncertain terms.

(pycess)dwt@atlan ~/Code/Projekte/pycess/pycess (git)-[master] % ./manage.py migrate
Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/dwt/Code/Projekte/pycess/django/django/core/management/__init__.py", line 330, in execute_from_command_line
    utility.execute()
  File "/Users/dwt/Code/Projekte/pycess/django/django/core/management/__init__.py", line 322, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/dwt/Code/Projekte/pycess/django/django/core/management/base.py", line 347, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Users/dwt/Code/Projekte/pycess/django/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/Users/dwt/Code/Projekte/pycess/django/django/core/management/commands/migrate.py", line 86, in handle
    executor = MigrationExecutor(connection, self.migration_progress_callback)
  File "/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/executor.py", line 19, in __init__
    self.loader = MigrationLoader(self.connection)
  File "/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/loader.py", line 47, in __init__
    self.build_graph()
  File "/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/loader.py", line 281, in build_graph
    _reraise_missing_dependency(migration, parent, e)
  File "/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/loader.py", line 264, in _reraise_missing_dependency
    raise exc
  File "/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/loader.py", line 274, in build_graph
    self.graph.add_dependency(migration, key, parent)
  File "/Users/dwt/Code/Projekte/pycess/django/django/db/migrations/graph.py", line 124, in add_dependency
    parent
django.db.migrations.graph.NodeNotFoundError: Migration process.0002_auto_20150411_1005 dependencies reference nonexistent parent node ('process', '0001_squashed_initial_2')

Here we had quite some discussion in #django-dev with MarkusH, of which the result to me was that a) django doesn't seem to ever record that a squashed migration is applied, at least as long as it is recognizable by django as a squashed migration (i.e. it has a .replaced property). b) there seems to be no way to tell django that the replacing squashed migration really is already applied for this database.

The last one turned out to be achievable if you remove the old migrations and the .replaces property from the squashed migration and then apply it with --fake

MarkusH might want to say more here that he can describe better.

For ease of reproduction I'm attaching the project where this occurred for me.

[Martin Häcker]

Since the upload limit was too small, I've put it on dropbox: ​https://dl.dropboxusercontent.com/u/765996/pycess.small.zip

[Markus Holtermann]

The behavior from the second part of that ticket (once the replaced migrations are removed and the replaces section is gone) sounds to me like a regression in 1.8: in 1.7 the squashed migration would automatically be fake applied due to 1.7's automatic behavior. This changed in 1.8 and at least requires a documentation update!

[Carl Meyer]

I just ran into this as well. I think it is definitely a bug and should be fixed in code, not just in documentation.

[Carl Meyer]

PR: ​https://github.com/django/django/pull/4738

[Carl Meyer <carl@…>]

In 492537a:

Fixed #24628 -- Fixed applied status for squashed migrations.

[Carl Meyer <carl@…>]

In efdcd13:

[1.8.x] Fixed #24628 -- Fixed applied status for squashed migrations.

Backport of 492537ac18df56c36f8a2335d773aa0fa01a61a3 from master.

[Carl Meyer <carl@…>]

In 5c085ea7:

Refs #24628 -- Added a second test and a docstring comment to avoid regression.

[Carl Meyer <carl@…>]

In feed5ad:

[1.8.x] Refs #24628 -- Added a second test and a docstring comment to avoid regression.

Backport of 5c085ea7b3f3ff10389aeed327e018581791876a from master.