Support unpickled models (e.g. read from cache) in migrations

[Odero]

Trying to create the migration below which is failing. It seems like the same issue discussed and resolved in #24282.

def generate_invoices(apps, schema_editor):
    A = apps.get_model('app1', 'A')
    B = apps.get_model('app1', 'B')
    User = apps.get_model('accounts', 'User')

    a = A.objects.last()

    b = B(
        client=a.user,
        status=5
    )
    b.save()

Also tried

    b = B(
        client=User.objects.get(pk=a.user.pk),
        status=5
    )

In both cases I get the error:

Cannot assign "<User: Some User>": "B.client" must be a "User" instance.

Investigating the issue shows that it's due to the models being read from cache.

[Tim Graham]

I can't reproduce this. Could you provide a sample project and confirm your Django version?

[Odero]

I discovered that it was a caching issue. I have caching enabled by default. It seems that whenever the item is fetched it creates and returns the cached version which is of the *real* type i.e. the one you'd get by doing

from app1.models import A

which is different from what you'd get by doing

apps.get_model('app1', 'A')

[Tim Graham]

What does it mean to have caching enabled by default? Is it a third-party app?

[Odero]

I meant automatic caching. I have set up django-cacheops to auto-manage the caching for me. BUT, even without cacheops, just manual caching also causes the error. Here's a contrived example to illustrate this effect and which fails when using cached data.

def generate_invoices(apps, schema_editor):
    A = apps.get_model('app1', 'A')
    B = apps.get_model('app1', 'B')
    User = apps.get_model('accounts', 'User')

    for item in A.objects.all():
        u = User.objects.get(pk=item.user.pk)

        # Simulate fetching cached data
        # start caching-segment
        cache.set('user', u)
        u = cache.get('user')
        # end caching-segment

        b = B(client=u, status=1)
        b.save()

If I remove those 2 lines in the "caching-segment", thereby eliminating caching, the migration runs successfully.

[Tim Graham]

I guess it probably infeasible or impractical to try to allow that to work, but CCing Markus for a second opinion.

[Markus Holtermann]

This is not really feasible as the unpickling of models from cache uses the global app registry (​https://github.com/django/django/blob/d5c0eb7d686ea89d6dbf787abc5bf18302c4e428/django/db/models/base.py#L1770). We would need to look up the models from the app registry at this state of the migration process. I have no idea how we could achieve that w/o passing apps around all the time.

I also think it's not a good idea to cache objects during the migration process. You can easily end up with objects from previous model states that are completely invalid at a later stage because too many things changed in between that wasn't taken care of for cached model instances.

I'm closing this as won't fix for the time being. If this is possible at some point we could re-evaluate the feature, but for now I doubt it's going to work out.