Moving production data to a new database

[liwee]

Hi,

When I move my production data to a new server, ideally I try to follow the steps

Old server

1. ftp source files from repo
2. run migration
3. backup database

New server

4. ftp source files from repo (migration directory is non-existent)
5. restore database
6. run migration (faked)

The problem occurs when there are some model changes in repo and steps 1 & 2 are left out.

E.g.

In old server, both database and model.py match, i.e. both have foo1:

  class foo(models.Model):
    foo1 = models.CharField()

however, the model.py in repo has been rename from foo1->foo2
and when step 6 is executed, migration detects the table foo and fakes the initial migration. it assumes that the table foo contains foo2 according to model.py while in reality the table foo contains foo1

  class foo(models.Model):
    foo2 = models.CharField()

at this point, no errors are detected. the problem occurs during the next code change, i.e. foo2->foo3. migration cannot locate foo2 in the table foo.

  class foo(models.Model):
    foo3 = models.CharField()

Is it possible to add some sort of field verification when the initial migration is faked at step 6?

[Tim Graham]

Sorry, but I can't really make sense of what you are doing. Maybe try avoiding making database schema and code changes at the same time if that's what's going on?

[Aymeric Augustin]

I would also suggest using a (D)VCS for moving code around rather than FTP'ing uncommitted changes. That would certainly spare you lots of trouble.

[liwee]

I am no expert in Django but I was just wondering if it is possible to change how Django handle migration with existing tables.

Run python manage.py migrate. Django will see that the tables for the initial migrations already exist and mark them as applied without running them.

The above is taken from Django documentation. Instead of marking them as applied unconditionally, it would be really helpful if Django could throw out some sort of error if the existing database structure doesn't match model.py (via inspectdb perhaps?).

[Tim Graham]

The idea is that all changes to your database schema should have Django migrations. If you are changing the schema in other ways, then you are on your own.

[Carl Meyer]

I do think it's odd that migrate will auto-fake your initial migrations if it sees that the tables already exist, as documented in the "upgrading from South" section that liwee quoted. I would think that situation should raise an error instead. I see how the auto-fake behavior is convenient in the upgrade-to-1.7 scenario, but as liwee points out, it's making some hard-to-justify assumptions. I think it should require a special flag to trigger that behavior.

(Of course it would be nice to keep the auto-fake behavior AND verify that the db state actually matches the models, as liwee suggested, but I don't think that's practically feasible at all.)

I'll look into this more to verify that I'm not missing anything and perhaps file a new bug to require a flag for auto-fake.

[liwee]

(Of course it would be nice to keep the auto-fake behavior AND verify that the db state actually matches the models, as liwee suggested, but I don't think that's practically feasible at all.)

I didn't mean for both to work at the same time.

1. For initial migration with existing tables, Django will first check if db state match the model
2. if match, continue to auto-fake
3. if doesn't match, raise an error and auto-fake is aborted
4. User will adjust the db table manually and rerun initial migration again

[Carl Meyer]

I understand. That's the same thing I meant. But I think you are under-estimating the practical difficulty of implementing "Django will first check if db state match the model" correctly. The code for "inspectdb" doesn't really get us very far in that direction.

[Carl Meyer]

I filed #24184 to follow up on the issue of the auto-fake behavior being unsafe.

[liwee]

This might be a very stupid idea ...

Assume the database already exist, say database name is mydb

Migration will

1. create another db called mydb_compare using model.py
2. export structure sql (no data) for mydb
3. export structure sql for mydb_compare
4. do a diff for these 2 files?