DoesNotExist is not picklable

[Łukasz Langa]

>>> import django
>>> django.contrib.auth.models.User.DoesNotExist
<class 'django.contrib.auth.models.DoesNotExist'>
>>> import pickle
>>> pickle.dumps(django.contrib.auth.models.User.DoesNotExist())
Traceback (most recent call last):
...
PicklingError: Can't pickle <class 'django.contrib.auth.models.DoesNotExist'>: it's not found as django.contrib.auth.models.DoesNotExist

The example uses contrib.auth.models.User but this is true for all models AFAICT.

The reason why picklability of this class is important is for multiprocessing exception handling, for instance in Celery tasks. Now every other exception in Celery gets a nice traceback in the logs but for DoesNotExist exceptions it's a useless:

Traceback (most recent call last):
 File "/home/virtualenv/lib/python2.7/site-packages/celery/concurrency/processes/pool.py", line 199, in worker
   put((READY, (job, i, result)))
 File "/usr/lib/python2.7/multiprocessing/queues.py", line 392, in put
   return send(obj)
MaybeEncodingError: Error sending result: '<ExceptionInfo: ObjectDoesNotExist('LeagueMember matching query does not exist.',)>'. Reason: 'Can't pickle <class 'psl.league.models.DoesNotExist'>: attribute lookup psl.league.models.DoesNotExist failed'.

[Łukasz Langa]

Unfortunately, after further investigation it looks like this is an inherent limitation of pickling. The Python documentation states that "picklable functions and classes must be defined in the top level of a module".

Source: ​http://docs.python.org/library/pickle.html#what-can-be-pickled-and-unpickled

So that's a WONTFIX, worth keeping for reference though.

[Jeremy Dunck]

Reopening because this problem affects the combination of Django, Celery, and Raven/Sentry, a very common thing indeed.

I am not sure if there's a fix available, but I'll look into it. Meantime, there was no discussion on this ticket and it seemed to be closed by reporter.

[Jeremy Dunck]

Example failure in sentry:

(This is an Endorsement.DoesNotExist exception being captured under raven.)

<MaybeEncodingError: Error sending result: '<ExceptionInfo: ObjectDoesNotExist('Endorsement matching query does not exist.',)>'. Reason: 'Can't pickle <class 'candidates.models.DoesNotExist'>: attribute lookup candidates.models.DoesNotExist failed'.>

[anonymous]

As I wrote in my WONTFIX explanation, this has to be fixed at the pickling level, specifically by introducing a new pickle protocol that can pickle inner classes. Good luck! ;)

[ask@…]

You can make this pickleable by defining reduce, e.g.

class DoesNotExist(..):

     def __reduce__(self):
         return _unpickle_DoesNotExist, (self.app_label, self.model, self.args)


 def _unpickle_DoesNotExist(app_label, model, args):
     return get_model(app_label, model).DoesNotExist(*args) 

[anonymous]

Exactly, there is at least one precedent in Django: r9272.

[Luke Plant]

Assuming the fix as described above is uncomplicated, this seems worth fixing. Test and patch needed.

[Luke Plant <L.Plant.98@…>]

In [a54a8bab0c6a96c03452040e92b2a3141695a363]:

Fixed #17776 - DoesNotExist is not picklable

Thanks to ambv for the report