Squashing migrations isn't very effective in complicated projects; rewrite docs to provide a preferred manual trimming process

[Mike Lissner]

I'm creating this ticket as a tracker, having already discussed this briefly on the mailing list:

​https://groups.google.com/g/django-developers/c/xpeFRpMTBZw/m/lDq78EedAwAJ

and having already created a PR for this:

​https://github.com/django/django/pull/14408

My impression is that to get this back on the review queue I need this ticket.

[Carlton Gibson]

OK, happy to review a suggestion.Thanks.

[Carlton Gibson]

Not sure that squashmigrations is not the preferred option for most/many projects. It works perfectly well for a lot of cases. 🤔
Probably _Automatically squash_ vs _Manually squash_ rather than _trim_ vs _squash_ (for me)
Comments on ​PR

[David Sanders]

@Carlton & Nessita:

I agree there needs to be a bit in the docs for this; Mike has stepped down as he doesn't have any time but I have some time:

• I agree that it should be "manually squashing" vs "automatic squashing"
• I've only tried squashmigrations twice but both times failed and ended up costing me quite a few hours that my boss wasn't too happy about :D
• The first time there was this circular dependency nightmare that I couldn't escape
• The second I had to write some custom serialisation to deal with django-pgtrigger & in the end it didn't work out (I don't remember all the details as this was a few months ago)
• I decided to finally manually squash them, but using the "replaces" feature that squashmigrations uses – and it worked a treat plus it didn't take very long!
• The key thing to remember is that you *must* note down all the non-elidable operations.
• These are the steps I used (they'll need to be tailored for general public):
  • I rm'd all my migrations
  • ran makemigrations (giving it a distinct new initial name)
  • used git to restore all the rm'd migrations
  • copy all the filenames into the replaces attribute
  • copied all my non-elidable operations *plus any dependencies* into the new initial migration
  • wait until all deployments have been migrated out of the "replaces zone" then follow the rest of the instructions on the docs

Sounded like Jessamyn had some issues with missing the custom user model - I didn't encounter this issue even though my user model was in the app I squashed 🤔

I'd like to forward a PR if that's ok.

[Natalia Bidart]

Replying to David Sanders:

@Carlton & Nessita:

Hello! Thanks for including me.

I agree there needs to be a bit in the docs for this; Mike has stepped down as he doesn't have any time but I have some time:

• I agree that it should be "manually squashing" vs "automatic squashing"

I don't object to this though I don't understand exactly which scenario would be "automatic squashing": even when running manage.py squashmigrations arg1 arg2, the user has to perform manual steps afterwards (unless I'm missing something, which can be the case!)

What I mean is that, in my experience, squashing migrations is always a manual operation (beyond running the management command, that is). There is more or less steps, and one may or may not use the management command. So if you ask me I would organize the docs roughly like this:

1. Squashing migrations

You can use the provided management command ... <details here for this option>.

If the above does not work or does not suit your needs for whatever reason (<examples>), you could instead:

<details for option 2>
<details for option 3>

I guess my point is that at most I would use labels to distinguish the options like: using the mgmt command, resetting, replacing.

• I've only tried squashmigrations twice but both times failed and ended up costing me quite a few hours that my boss wasn't too happy about :D

I've did it multiple times are previous work but it required a non trivial amount of coordination involving migration landing freezes, deployment, squashing, migration unfreeze. We only did this when migrate was too slow when running tests.

• The first time there was this circular dependency nightmare that I couldn't escape

Luckily we did not have this! But we did have a test check to ensure apps would not create circular dependencies, so I'm sure that helped.

• The second I had to write some custom serialisation to deal with django-pgtrigger & in the end it didn't work out (I don't remember all the details as this was a few months ago)
• I decided to finally manually squash them, but using the "replaces" feature that squashmigrations uses – and it worked a treat plus it didn't take very long!
• The key thing to remember is that you *must* note down all the non-elidable operations.

:see_no_evil:

• These are the steps I used (they'll need to be tailored for general public):
  • I rm'd all my migrations
  • ran makemigrations (giving it a distinct new initial name)
  • used git to restore all the rm'd migrations
  • copy all the filenames into the replaces attribute
  • copied all my non-elidable operations *plus any dependencies* into the new initial migration
  • wait until all deployments have been migrated out of the "replaces zone" then follow the rest of the instructions on the docs

I never tried this procedure!

Sounded like Jessamyn had some issues with missing the custom user model - I didn't encounter this issue even though my user model was in the app I squashed 🤔

I'd like to forward a PR if that's ok.

Yes please!

[David Sanders]

Thanks nessita :)

Also I'm guilty of posting here then cross-posting in the forum as well where Carlton had replied: ​https://forum.djangoproject.com/t/doc-update-describe-steps-for-manually-squashing-migrations/20624

[David Sanders]

Draft PR up for feedback on progress: ​https://github.com/django/django/pull/16843