Opened 15 years ago

Closed 15 years ago

Last modified 12 years ago

#12121 closed (fixed)

deepcopy(model_instance) causes infinite recursion since r11681

Reported by: Johannes Dollinger Owned by: nobody
Component: Database layer (models, ORM) Version: dev
Severity: Keywords:
Cc: immel@… Triage Stage: Unreviewed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

deepcopy(mode_instance) causes infinite recursion in Model.__reduce__() since r11681.

$ python tests/runtests.py queries
      ...
    RuntimeError: maximum recursion depth exceeded
      ...

Attachments (2)

12121.reduce_deferred.diff (3.4 KB ) - added by Johannes Dollinger 15 years ago.
t12121-r11683.diff (1.9 KB ) - added by Russell Keith-Magee 15 years ago.
Possible patch for weird reduce behavior.

Download all attachments as: .zip

Change History (20)

comment:1 by Russell Keith-Magee, 15 years ago

Resolution: worksforme
Status: newclosed

I ran the full test suite before I committed, and I just re-ran the queries test specifically - I don't see this failure.

comment:2 by Russell Keith-Magee, 15 years ago

To clarify - I don't see this behaviour under MacOS, using Python 2.5, under SQLite, Postgres and MySQL. I'm willing to accept that you're seeing this problem, but we're going to need a lot more detail that you have provided so far.

comment:3 by Johannes Dollinger, 15 years ago

$ sqlite3 --version
3.6.5
$ svn up
At revision 11690.
$ export PYTHONPATH=`pwd`
$ export DJANGO_SETTINGS_MODULE=settings
$ cat settings.py 
DEBUG = True
DATABASE_ENGINE = 'sqlite3'
DATABASE_NAME = 'test.db'

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
)
$ python django/bin/django-admin.py syncdb --noinput
$ python django/bin/django-admin.py shell
Python 2.5 (r25:51918, Sep 19 2006, 08:49:13) 
[GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django.contrib.contenttypes.models import ContentType
>>> ct = ContentType.objects.all()[0]
>>> ct
<ContentType: content type>
>>> from django.db.models import Q
>>> Q(ct=ct)
<django.db.models.query_utils.Q object at 0xdd0a30>
>>> Q(ct=ct) | Q(foo=True)
Traceback (most recent call last):
    ...
RuntimeError: maximum recursion depth exceeded
>>>
$ python tests/runtests.py queries
======================================================================
FAIL: Doctest: regressiontests.queries.models.__test__.API_TESTS
...
$

comment:4 by Alex Gaynor, 15 years ago

The test passes for me under sqlite3, python2.5. My SQLite is version 3.6.16 however.

comment:5 by Karen Tracey, 15 years ago

Perhaps it might shed some light if you did not trim all of the traceback, that might give a clue what is causing the recursion. FWIW I cannot recreate on Ubuntu nor Windows, with a somewhat lower level sqlite3:

kmt@lbox:~/django/trunk$ svn up
At revision 11690.
kmt@lbox:~/django/trunk$ svn diff
kmt@lbox:~/django/trunk$ sqlite3 --version
3.5.9
kmt@lbox:~/django/trunk$ export PYTHONPATH=`pwd`
kmt@lbox:~/django/trunk$ export DJANGO_SETTINGS_MODULE=settings
kmt@lbox:~/django/trunk$ cat settings.py
DATABASE_ENGINE = 'sqlite3'
DATABASE_NAME = 'test.db'

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
)

kmt@lbox:~/django/trunk$ python django/bin/django-admin.py syncdb --noinput
kmt@lbox:~/django/trunk$ python django/bin/django-admin.py shell
Python 2.5.2 (r252:60911, Oct  5 2008, 19:24:49)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django.contrib.contenttypes.models import ContentType
>>> ct = ContentType.objects.all()[0]
>>> ct
<ContentType: content type>
>>> from django.db.models import Q
>>> Q(ct=ct)
<django.db.models.query_utils.Q object at 0x84f09cc>
>>> Q(ct=ct) | Q(foo=True)
<django.db.models.query_utils.Q object at 0x84f0c4c>
>>> quit()
kmt@lbox:~/django/trunk$ python tests/runtests.py queries
----------------------------------------------------------------------
Ran 6 tests in 1.185s

OK
kmt@lbox:~/django/trunk$

I don't have any Macs to try.

comment:6 by Johannes Dollinger, 15 years ago

Resolution: worksforme
Status: closedreopened

Searching for a python bug, I stumbled over a blog post from Malcolm: http://www.pointy-stick.com/blog/2009/03/23/yak-shaving-advanced-players/. Let me quote:

"Next morning, somebody reports an infinite recursion problem. That was very odd, since the code I wrote looked pretty correct in that area. The key seemed to be that they were using Python 2.4. Even creating a diff between the pickle module for Python 2.4 and the one for 2.5 was unenlightening, so the problem was deeper than that.

I'm not 100% sure if this is a bug in Python versions prior to 2.5, or just an undocumented side-effect. The end result is that if you write a __reduce__() method, you can't do the natural thing and dispatch to the same method on the superclass if you don't need to do anything special. At least, not in Python 2.3 or 2.4. Django supports those versions, so I had to work around.

Not a big problem, once I diagnosed it (turns out the existing test suite caught the problem when I ran it under Python 2.4), and fixed easily enough."

The ticket was #10547, the commit r10099. Apparently, my Apple "python 2.5" has some 2.4 bugs in it.

comment:7 by Johannes Dollinger, 15 years ago

Here's the requested traceback:

>>> Q(ct=ct) | Q(foo=True)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/emulbreh/Projekte/django/trunk/django/db/models/query_utils.py", line 159, in __or__
    return self._combine(other, self.OR)
  File "/Users/emulbreh/Projekte/django/trunk/django/db/models/query_utils.py", line 154, in _combine
    obj = deepcopy(self)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/copy.py", line 173, in deepcopy
    y = copier(memo)
  File "/Users/emulbreh/Projekte/django/trunk/django/utils/tree.py", line 61, in __deepcopy__
    obj.children = deepcopy(self.children, memodict)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/copy.py", line 162, in deepcopy
    y = copier(x, memo)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/copy.py", line 227, in _deepcopy_list
    y.append(deepcopy(a, memo))
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/copy.py", line 162, in deepcopy
    y = copier(x, memo)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/copy.py", line 234, in _deepcopy_tuple
    y.append(deepcopy(a, memo))
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/copy.py", line 181, in deepcopy
    rv = reductor(2)
  File "/Users/emulbreh/Projekte/django/trunk/django/db/models/base.py", line 356, in __reduce__
    return super(Model, self).__reduce__()
...
  File "/Users/emulbreh/Projekte/django/trunk/django/db/models/base.py", line 356, in __reduce__
    return super(Model, self).__reduce__()
RuntimeError: maximum recursion depth exceeded

by Johannes Dollinger, 15 years ago

Attachment: 12121.reduce_deferred.diff added

comment:8 by Johannes Dollinger, 15 years ago

Has patch: set
Patch needs improvement: set

Just replacing super(Model, self).__reduce__() broke the cache tests.

comment:9 by Johannes Dollinger, 15 years ago

Note: The patch is bad. Model.__reduce__() cannot be removed. It's required for dynamically created models.

by Russell Keith-Magee, 15 years ago

Attachment: t12121-r11683.diff added

Possible patch for weird reduce behavior.

comment:10 by Russell Keith-Magee, 15 years ago

@emulbreh: I've just attached a patch which passes for me - however, I don't have a copy of Python 2.4 (or a broken version of Python 2.5) available. Can you validate that this patch works for you?

in reply to:  10 comment:11 by Karen Tracey, 15 years ago

Replying to russellm:
I've got Python2.4 on both Ubuntu and Windows. Was able to see the infinite recursion failure once the early level of Python was identified as the cause, just confirmed the queries test passes with the patch.

comment:12 by Russell Keith-Magee, 15 years ago

Resolution: fixed
Status: reopenedclosed

(In [11692]) [1.1.X] Fixed #12121 -- Modified reduce on a model to avoid an infinite recursion problem that occurs on Python 2.4. Thanks to emulbreh for the report.

Backport of r11691 from trunk.

in reply to:  10 ; comment:13 by Johannes Dollinger, 15 years ago

Replying to russellm:

@emulbreh: I've just attached a patch which passes for me - however, I don't have a copy of Python 2.4 (or a broken version of Python 2.5) available. Can you validate that this patch works for you?

It works, thanks.

in reply to:  13 comment:14 by anonymous, 15 years ago

Replying to emulbreh:

Replying to russellm:

@emulbreh: I've just attached a patch which passes for me - however, I don't have a copy of Python 2.4 (or a broken version of Python 2.5) available. Can you validate that this patch works for you?

It works, thanks.

Which means: with my broken Python 2.5, too.

comment:15 by Ryan Fugger, 15 years ago

I can confirm this bug on Red Hat under python 2.4.3. Where I see it is attempting to pickle an object for caching (using locmem cache) -- infinite recursion on Model.reduce.

Using python 2.5 works fine.

comment:16 by Russell Keith-Magee, 15 years ago

@rfugger - are you reporting that this is *still* a problem? This should have been fixed by r11691; if it still exists for Python 2.4.3, then we have more work to do; a test case that triggers the infinite recursion would be most helpful.

comment:17 by anonymous, 15 years ago

Cc: immel@… added

comment:18 by Claude Paroz <claude@…>, 12 years ago

In [146aff3bac974e56ec8cb597c2720d1cd4f77b26]:

Fixed #18590 - Reverted Python 2.4 workaround for Model pickling

Revert of 08d521efa0. Refs #10547, #12121.
Thanks Michal Petrucha for the report.

Note: See TracTickets for help on using tickets.
Back to Top