Unable to pickle certain QuerySet objects due to use of curry function in RelatedField

[Michael Malone]

QuerySet objects containing Model objects that have a RelatedField attribute are not pickle-able due to the use of the curry function in RelatedField. The Python pickle/cPickle modules can only serialize objects that are defined at the top level of a module. Since the curry function returns a nested function, pickling fails.

In this case, the use of curry to create a partial function seems unnecessary. Equivalent functionality can be achieved by storing the curry parameters in an instance variable. This appears to be the simplest fix to this problem, so it's probably the right one. I've attached a diff for related.py that implements this fix. Alternatively, re-implementing the curry function as a callable class may work, but not without some trickery in getstate to get the callable to pickle.

[Michael Malone]

Patch to remove curry function from RelatedField init

[Jacob]

I think this is a duplicate of #7204, but this is actually the root cause. I'm going to keep both open until I verify.

[Jacob]

mmalone: do you have a minimal test case that can replicate this error? Simply calling pickle(qs) for some arbitrary qs doesn't fail; it appears you need to do something else to make that failure happen.

[Michael Malone]

jacob: I think the model objects inside the QuerySet has to have a RelatedField that you're filtering by. The RelatedField.contribute_to_class() method has to be called. I'm working on narrowing it down at the moment. Here's a dpaste from the console showing the exception, but it's obviously specific to our code base (probably not that helpful, but w/e): ​http://dpaste.com/57631/

I'll keep working on a minimal test-case to reproduce the error.

[Michael Malone]

Ok, I've got a working test-case. I've reproduced it using a OneToOneField, not sure if other types of RelatedFields trigger the problem.

Complete models.py:

from django.db import models

class Sandwich(models.Model):
    pass

class Pickle(models.Model):
    sandwich = models.OneToOneField(Sandwich, primary_key=True, editable=False)

Exception in interpreter:

>>> from pickledcurry.app.models import *
>>> from django.db.models.query import QuerySet
>>> qs = QuerySet(Pickle).filter(pk=1)
>>> import pickle
>>> pickle.dumps(qs)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 1366, in dumps
    Pickler(file, protocol).dump(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 224, in dump
    self.save(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 419, in save_reduce
    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 419, in save_reduce
    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 419, in save_reduce
    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 600, in save_list
    self._batch_appends(iter(obj))
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 615, in _batch_appends
    save(x)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 562, in save_tuple
    save(element)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 419, in save_reduce

    save(state)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <function _curried at 0x12c42f0>: it's not found as django.utils.functional._curried
>>> 

And here's a little function that will dump a trace of where the PicklingError is being raised, along with the output when run w/ the qs I created in the last example:

def trace_pickle(obj):
  if not hasattr(obj, '__dict__'):
    return
  for k in obj.__dict__:
    try:
      pickle.dumps(obj.__dict__[k])
    except:
      print ("%s -> %s" % (k, obj.__dict__[k]))[:120]
      trace_pickle(obj.__dict__[k])

>>> trace_pickle(qs)
query -> SELECT `app_pickle`.`sandwich_id` FROM `app_pickle` WHERE `app_pickle`.`sandwich_id` = 1 
connection -> <django.db.backends.mysql.base.DatabaseWrapper object at 0x1068750>
connection -> <_mysql.connection open to 'localhost' at a38610>
encoders -> {<type 'int'>: <function Thing2Str at 0x10696b0>, <type 'datetime.timedelta'>: <function DateTimeDelta2liter
string_decoder -> <function string_decoder at 0x12f1330>
unicode_literal -> <function unicode_literal at 0x12f12f0>
where -> (AND: ('app_pickle', 'sandwich_id', <django.db.models.fields.related.OneToOneField object at 0x12b0c30>, 'exact
children -> [('app_pickle', 'sandwich_id', <django.db.models.fields.related.OneToOneField object at 0x12b0c30>, 'exact',

Hope that all makes sense. Let me know if you need any other info.

[Jacob]

You only put one pickle on your sandwiches? What a sad, sad thing!

(Thanks; I'll play around with this and see if it's what I need.)

[anonymous]

I've been able to reproduce this with ForeignKey relationships in r7726 of trunk.

[anonymous]

Here's my question, why did this problem start occurring recently? The curried function was added as part of magic removal in [2381] and merged into trunk in [2809]. Pickling ORM objects with relationships was working just fine for me as of r6626 of trunk.

[Michael Malone]

When was qs-rf merged? Here's a thread from two months ago where Malcolm acknowledges that QuerySet pickling is broken: ​http://groups.google.com/group/django-users/browse_thread/thread/32143d024b17dd00/76b336189c3c3757?lnk=st#76b336189c3c3757

The curry function could have been there/can stay. It just can't be pickled. So maybe in r6626 it wasn't being pickled..? One could get fancy and implement getstate/re-init the object when it's retrieved, or something like that, but that seems like more trouble than it's worth. I'm wondering why the curry function is used here to begin with? As far as I can tell, the change I made maintains the same functionality and is as easy/easier to understand than the curry version.

[JustinTArthur]

Yea, definitely a valid point. Your patch sounds fine. QuerySet Refactor was merged into trunk at r7477.

[Malcolm Tredinnick]

Fixed in [7773].

[wfagan@…]

I am running Django 1.1 beta and having this same problem I think.

Can't pickle <function _curried at 0x9eb8e9c>: it's not found as django.utils.functional._curried

Here is my query.

query = Model.objects.all().annotate(liked_count=Count('liked_users')).exclude(liked_count=0).order_by('-liked_count')

liked_users is a related name from my UserProfile objects.

im doing ...

pickle.dumps(query)

[Russell Keith-Magee]

@wfagan Your problem is slightly different, and is already logged as #10197.

[wfagan@…]

russellm: thanks

[Paul Garner]

Just upgraded our server to 1.2-beta-1 and started getting this error.
Am trying to cache a simple queryset of a model which has a ForeignKey field... looks like this bug is back?

[Paul Garner]

Replying to anentropic:

Just upgraded our server to 1.2-beta-1 and started getting this error.
Am trying to cache a simple queryset of a model which has a ForeignKey field... looks like this bug is back?

the qs I'm trying to cache (from a Manager method) is:
self.filter(content_type=ct.pk)

...where content_type field is a ForeignKey.

Hope that helps.

[Russell Keith-Magee]

Please don't reopen an old ticket; This ticket was fixed in [7773], and the fix included a test case, so we know that this specific case can't re-emerge. Instead, open a new ticket, with a specific model and code example that is failing.

[Paul Garner]

Ok sorry, have opened a new one at: ​http://code.djangoproject.com/ticket/13119

[Jacob]

Milestone 1.1 deleted