Migrations are crashing after changing foreign key from auth.Group model to a proxy model of auth.Group

[tinloaf]

Hi!

I have created a proxy model for the User model like this (in core.models):

from django.contrib.auth.models import User as OriginalUser

class User(OriginalUser):
	objects = UserManager()
	all_objects = AllUserManager()

	class Meta:
		proxy = True
[...]

I then changed some ForeignKeys that have been pointing to the original User to my proxy. Now makemigrations creates a migration that looks like this:

# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.db import models, migrations


class Migration(migrations.Migration):

    dependencies = [
        ('finance', '0011_auto_20141126_1447'),
    ]

    operations = [
        migrations.AlterField(
            model_name='financialdata',
            name='user',
            field=models.OneToOneField(to='core.User'),
            preserve_default=True,
        ),
        migrations.AlterField(
            model_name='mandate',
            name='user',
            field=models.ForeignKey(to='core.User'),
            preserve_default=True,
        ),
        migrations.AlterField(
            model_name='userrate',
            name='user',
            field=models.ForeignKey(to='core.User'),
            preserve_default=True,
        ),
    ]

Which looks OK so far. If I try to run this migration, it fails like this:

(venv)tinloaf@janeway alumnet % python manage.py migrate                         
Operations to perform:
  Synchronize unmigrated apps: gis, bootstrap3, dajaxice, autocomplete_light
  Apply all migrations: sessions, ml, auth, admin, local, finance, core, contenttypes
Synchronizing apps without migrations:
  Creating tables...
  Installing custom SQL...
  Installing indexes...
Running migrations:
  Applying finance.0012_auto_20150319_1316...Traceback (most recent call last):
  File "/mnt/data/home/tinloaf/src/alumnet/venv/lib/python3.3/site-packages/django/apps/registry.py", line 148, in get_app_config
    return self.app_configs[app_label]
KeyError: 'core'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/data/home/tinloaf/src/alumnet/venv/lib/python3.3/site-packages/django/db/migrations/state.py", line 84, in render
    model = self.apps.get_model(lookup_model[0], lookup_model[1])
  File "/mnt/data/home/tinloaf/src/alumnet/venv/lib/python3.3/site-packages/django/apps/registry.py", line 202, in get_model
    return self.get_app_config(app_label).get_model(model_name.lower())
  File "/mnt/data/home/tinloaf/src/alumnet/venv/lib/python3.3/site-packages/django/apps/registry.py", line 150, in get_app_config
    raise LookupError("No installed app with label '%s'." % app_label)
LookupError: No installed app with label 'core'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/mnt/data/home/tinloaf/src/alumnet/venv/lib/python3.3/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/mnt/data/home/tinloaf/src/alumnet/venv/lib/python3.3/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/mnt/data/home/tinloaf/src/alumnet/venv/lib/python3.3/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/mnt/data/home/tinloaf/src/alumnet/venv/lib/python3.3/site-packages/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
  File "/mnt/data/home/tinloaf/src/alumnet/venv/lib/python3.3/site-packages/django/core/management/commands/migrate.py", line 161, in handle
    executor.migrate(targets, plan, fake=options.get("fake", False))
  File "/mnt/data/home/tinloaf/src/alumnet/venv/lib/python3.3/site-packages/django/db/migrations/executor.py", line 68, in migrate
    self.apply_migration(migration, fake=fake)
  File "/mnt/data/home/tinloaf/src/alumnet/venv/lib/python3.3/site-packages/django/db/migrations/executor.py", line 96, in apply_migration
    if self.detect_soft_applied(migration):
  File "/mnt/data/home/tinloaf/src/alumnet/venv/lib/python3.3/site-packages/django/db/migrations/executor.py", line 140, in detect_soft_applied
    apps = project_state.render()
  File "/mnt/data/home/tinloaf/src/alumnet/venv/lib/python3.3/site-packages/django/db/migrations/state.py", line 94, in render
    raise ValueError(msg.format(field=operations[0][1], model=lookup_model))
ValueError: Lookup failed for model referenced by field finance.FinancialData.user: core.User

So, for some reason, it cannot find the core module and/or the core.User model. The core module is in my installed apps, and the whole application works if I just fake the migration, so the model itself works, etc. Still, having to fake a migration in this case should probably not be the intended solution.

Also, I can't fake that migration in tests, making testing essentially impossible in this case.

[Tim Graham]

Could you provide a minimal project that we could download to reproduce the issue? Did you test with Django 1.8?

[Marco Fucci]

The bug seems to have been fixed in Django 1.8.

I've created a github repo ​https://github.com/marcofucci/django-ticket-24506 containing two projects using the same code but different django versions.

The one using django 1.7 breaks and the one using django 1.8 doesn't.

[Petros Moisiadis]

I get a similar exception chain on 1.8.2 when trying to migrate an app that contains a model with a foreign key to another app's model defined as a proxy model of Django's auth application Group model. In case this is relevant, the application that contains the proxy model has a custom but very simple AppConfig subclass, that only overrides ready() method just to import a 'signals' module.

[Petros Moisiadis]

The traceback occurs when rendering model states:

Operations to perform:
  Synchronize unmigrated apps: rest_framework, django_python3_ldap, debug_toolbar, messages, staticfiles
  Apply all migrations: contenttypes, users, issues, ...
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  Rendering model states...Traceback (most recent call last):
  File "/path/to/virtualenv/lib/python3.4/site-packages/django/apps/config.py", line 159, in get_model
    return self.models[model_name.lower()]
KeyError: 'group'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/path/to/virtualenv/lib/python3.4/site-packages/django/db/migrations/state.py", line 238, in __init__
    model = self.get_model(lookup_model[0], lookup_model[1])
  File "/path/to/virtualenv/lib/python3.4/site-packages/django/apps/registry.py", line 202, in get_model
    return self.get_app_config(app_label).get_model(model_name.lower())
  File "/path/to/virtualenv/lib/python3.4/site-packages/django/apps/config.py", line 162, in get_model
    "App '%s' doesn't have a '%s' model." % (self.label, model_name))
LookupError: App 'users' doesn't have a 'group' model.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/path/to/virtualenv/lib/python3.4/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
    utility.execute()
  File "/path/to/virtualenv/lib/python3.4/site-packages/django/core/management/__init__.py", line 330, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/path/to/virtualenv/lib/python3.4/site-packages/django/core/management/base.py", line 390, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/path/to/virtualenv/lib/python3.4/site-packages/django/core/management/base.py", line 441, in execute
    output = self.handle(*args, **options)
  File "/path/to/virtualenv/lib/python3.4/site-packages/django/core/management/commands/migrate.py", line 221, in handle
    executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
  File "/path/to/virtualenv/lib/python3.4/site-packages/django/db/migrations/executor.py", line 100, in migrate
    state.apps  # Render all real_apps -- performance critical
  File "/path/to/virtualenv/lib/python3.4/site-packages/django/utils/functional.py", line 60, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/path/to/virtualenv/lib/python3.4/site-packages/django/db/migrations/state.py", line 166, in apps
    return StateApps(self.real_apps, self.models)
  File "/path/to/virtualenv/lib/python3.4/site-packages/django/db/migrations/state.py", line 248, in __init__
    raise ValueError(msg.format(field=operations[0][1], model=lookup_model))
ValueError: Lookup failed for model referenced by field issues.SomeModel.group: users.Group

[Tim Graham]

Could you please add a minimal project that reproduces the issue?

[Petros Moisiadis]

I added an example project as an attachment. Running 'migrate' reproduces the problem. The traceback above is shown when running the 'migrate' command for a second time as well as for subsequent runs. The first run also produces a traceback, which seems to be the same error but triggered in a different code path.

The project's history/migrations:
1) 'myapp' application has a MyModel model with a text field / The Django's default user model is being used (no AUTH_USER_MODEL is set).
2) 'myapp' application's MyModel gets a foreign key to 'auth.Group'
3) 'users' application is added to INSTALLED_APPS. It contains a custom User model.
4) 'users' application gets a new Group model as a proxy to auth.Group and MyModel's foreign key is changed to point to 'users.Group'. AUTH_USER_MODEL is changed to 'users.User'
5) 'myapp' application's MyModel gets a new text field. Migration file for this change is created without problems, but running 'migrate' fails with the trace above.

[Marco Fucci]

I had a good look at this.
The problem has nothing to do with the auth app and proxy models.

The issue is caused by AlterField not being able to build dependencies properly.
If you take a look at the attached project, you'll realise that myproject.myapp.migrations.0003_auto_20150524_2202.py should depend on ('users', '0002_group').

If you do add that to the dependencies list, the migrate command works.

I tried to search in Trac, there are many related bugs but nothing exactly like this. The best one might be #22944 .
I've actually found a way to fix it but I don't have enough time atm.

[Asif Saifuddin Auvi]

I deleted all old migrations and ran makemigrations and migrate to fix the issue in django 1.8.7