Query cloning fails with models having Lock as instance member

[Claude Paroz]

Here is a basic test case:

import threading
from django.db import models

class BaseObject(models.Model):
    status      = models.CharField(max_length=10)
    foreign     = models.ForeignKey('MyModel')

class MyModel(models.Model):
    name = models.CharField(max_length=10)

    def __init__(self, *args, **kwargs):
        super(MyModel, self).__init__(*args, **kwargs)
        self.checkout_lock = threading.Lock()

Now as soon as a query with MyModel instance has to be cloned, this fails.

from testapp.models import MyModel, BaseObject
mm = MyModel(name='Test1')
mm.save()
bo = BaseObject(status='a',foreign=mm)
bo.save()

print BaseObject.objects.filter(foreign=mm).order_by('foreign__name')
(full traceback here)
Error: un(deep)copyable object of type <type 'thread.lock'>

[Claude Paroz]

Also note that this doesn't happen with 1.1

[George Vilches]

If this doesn't happen in 1.1, but does in 1.2, the should this be marked as a regression for 1.2 release? Marking the milestone so a core dev will notice and decide, since that fits the criteria from other tickets.

[Russell Keith-Magee]

I'm looking at this right now; it's a problem with the way db_prep_save is used on foreign keys. Should have a patch soon.

[Russell Keith-Magee]

(In [12865]) Fixed #13227 -- Modified ForeignKeys to fully honor the db_prep/prep separation introduced by multidb. This was required to ensure that model instances aren't deepcopied as a result of being involved in a filter clause. Thanks to claudep for the report, and Alex Gaynor for the help on the patch.

[Claude Paroz]

I'm so sorry, but I think this is not completely resolved. Here is an updated test case where the same error appear. Model is unchanged.

from testapp.models import MyModel, BaseObject
mm = MyModel(name='Test1')
mm.save()
bo = BaseObject(status='a',foreign=mm)
bo.save()

mms = MyModel.objects.all()
mms_id_list = [m.id for m in mms]
print BaseObject.objects.filter(foreign__in=mms).order_by('foreign__name')

Note the mms_id_list creation (apparently unrelated to the problem) which access the id of MyModel objects. Without this line, the query is successfull.

[Russell Keith-Magee]

The issue is that by evaluating mms_id_list (which evaluates the mms queryset), you populate the mms queryset cache with object instances; when you then try to evaluate the BaseObject query, you need to be able to clone those instances.

On closer inspection, it looks like the original problem was also present in 1.1... but this problem *isn't*, so it's a regression. However I also need to backport r12865 to 1.1.

[Russell Keith-Magee]

(In [12963]) [1.1.X] Refs #13227 -- Partial backport of r12865; backported the changes to Where tree cloning logic to ensure that unclonable objects in a where() clause don't break querying.

[Russell Keith-Magee]

(In [12970]) Fixed #13227 -- Ensure that the query cache is flushed when a QuerySet is deepcopied, avoiding problems when an evaluated queryset is used as a subquery. Thanks to claudep for the report.

[Claude Paroz]

Thanks Russell, it's fine now!

[Jacob]

Milestone 1.2 deleted

[Mariusz Felisiak <felisiak.mariusz@…>]

In 1dd96f73:

Refs #13227 -- Adjusted a test to avoid making a shared test model unpickable.

This allowed the Note model to be used in setUpTestData() which requires
assigned model instances to be copy.deepcopy()'able.