Exceptions in GEOS I/O object destructor at process exit

[mroach@…]

I see this traceback in GeoDjango when the application exits. Not exactly
sure but there may be some memory leakage here.

Exception TypeError: "'NoneType' object is not callable" in <bound method_WKTReader.__del__ ofdjango.contrib.gis.geos.prototypes.io._WKTReader object at 0x6b1a790>> ignored

Exception TypeError: "'NoneType' object is not callable" in <bound method _WKBReader.__del__ of <django.contrib.gis.geos.prototypes.io._WKBReader object at 0x6b43690>> ignored

Exception TypeError: "'NoneType' object is not callable" in <bound method WKBWriter.__del__ of <django.contrib.gis.geos.prototypes.io.WKBWriter object at 0x6b1ad50>> ignored

To reproduce, just run a distance function (for example:

airport.objects.distance("POINT (-74.3166750000000036 40.9000249999999994)")

Whent the application exits (or exit from a python prompt), you'll get the
exceptions.

[jbronn]

I can't reproduce. I need details on the platform you're using. What version of Python/Django/GEOS/PostGIS? What OS are you running on (instinct tells me RHEL)?

I don't think memory leakage is occurring. GeoDjango instantiates up to 5 GEOS I/O objects per-thread. These references are cleaned up by the __del__ method on the I/O objects, however in your particular install of Python the GC order has the I/O object destructor wrapper destroyed before the Python object -- hence why it doesn't exist at the time of interpreter exit. The GEOS I/O references aren't being cleaned upon exit of the process, so even though the references aren't released the process memory space should be reclaimed by the OS regardless because the process is terminating.

In other words, it may look bad but it's relatively benign. But again, I need more details so I can confirm what's going on and possibly tweak variable names so it won't occur on platforms of your type.

[jbronn]

This isn't a 1.2 release blocker.

[mroach@…]

We are running:
python: 2.6.5
django: trunk
GEOS: 3.2.1
PostGIS: 3.2.1 (i think)
OS: Arch linux

[Brandon Konkle]

I'm also getting this message upon exit. I'm running Django on an OS X Snow Leopard laptop, using Postgres 8.3, GEOS 3.1.1, Proj 4.7.0, PostGIS 1.3.6, and GDAL 1.6.3. We haven't started running Django 1.2 on our Ubuntu 8.04 dev server yet, but when we do I'll check to see if the message shows up there as well.

[Rozza]

Handle dying threads

[Rozza]

I've added an attachment that handles / catches the dying thread scenario and quietly lets the rest in peace.

I have no idea how I'd write a test to prove it works - but I tested on my machine it stops the warnings.

I based it on how _threading_local.py from in the python library handles destruction.

[Rozza]

Updated after conversation on the geo_django google group.

The root cause for these problems is the same: the module holding the 
destructor is garbage collected in the dying thread before the geometry 
or I/O object is itself garbage collected (and tries to call the now 
non-existent function). 

To compound the issue, it appears that GC order changes depending on the 
version of Python used, as well as the platform it is running on.  In 
other words, Python 2.7 on Snow Leopard won't show these errors while 
Python 2.5 on Arch Linux will. 

However, for the ticket you opened (#13843), the developer may prevent 
these errors by _not_ creating module-level globals of GEOS geometry 
objects.  Here's an example of a module-level global: 


from django.contrib.gis.geos import GEOSGeometry 
geom = GEOSGeometry('POINT(5 23)') 
def my_view(): 
    do_something_with(geom) 


Instead of having `geom` defined in the global scope, it would be better 
to have it instantiated in the local scope of the view function instead. 

As it stands I'm -1 on the #13843 patch.  While it reduces log noise 
in your case, it masks that there could be an underlying problem with 
the developer's code.  Regardless, I'm keeping the ticket open because 
absent a technological fix, there should be at least note in the 
documentation on how to avoid it. 

-Justin 

[jbronn]

I use thread-local module-level global I/O objects that developer has no control over, thus re-opening.

[jbronn]

I got an Arch Linux VM set up and got the problem reproduced using standard packages and with the code from the tutorial. This is what I get in the logs:

[Wed Jul 07 08:26:48 2010] [error] Exception KeyError: KeyError(140158140106560,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Jul 07 08:26:48 2010] [error] Exception KeyError: KeyError(140158140106560,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Jul 07 08:26:48 2010] [error] Exception KeyError: KeyError(140158140106560,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Jul 07 08:26:48 2010] [error] Exception KeyError: KeyError(140158140106560,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Jul 07 08:26:48 2010] [error] Exception KeyError: KeyError(140158140106560,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Jul 07 08:26:48 2010] [error] Exception KeyError: KeyError(140158140106560,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Jul 07 08:26:48 2010] [error] Exception KeyError: KeyError(140158140106560,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Jul 07 08:26:48 2010] [error] Exception KeyError: KeyError(140158140106560,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Jul 07 08:26:48 2010] [error] Exception KeyError: KeyError(140158140106560,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Jul 07 08:26:48 2010] [error] Exception KeyError: KeyError(140158140106560,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Jul 07 08:26:48 2010] [error] Exception KeyError: KeyError(140158140106560,) in <module 'threading' from '/usr/lib/python2.6/threading.pyc'> ignored
[Wed Jul 07 08:26:48 2010] [notice] caught SIGTERM, shutting down
[Wed Jul 07 08:26:48 2010] [error] Exception AttributeError: "'NoneType' object has no attribute 'destroy_geom'" in 
[Wed Jul 07 08:26:48 2010] [error] Exception AttributeError: "'NoneType' object has no attribute 'destroy_geom'" in 

[jbronn]

Create reference to thread_context in GEOSFunc instances.

[jbronn]

Need feedback from users to test this out on their platforms. This patch tries to address root cause of these error messages, rather than masking them in a try/except statement.

[jbronn]

There are a few of you CC'ed on this ticket, anyone tried out the patch? I'd like to get this into 1.3.

[Peter Sheats]

I just tried the patch and it works fine -- error message goes away completely. How refreshing!

[jbronn]

Fixed in r15378.

[jbronn]

In [15379]:

[1.2.X] Fixed #13488 -- No longer generate unhandled exceptions that may occur when destructors of global GEOS I/O objects are called on process termination.

Backport of r15378 from trunk.

[Jacob]

Milestone 1.3 deleted

[matt.rigal@…]

I actually didn't want to reopen the ticket, but I do experience the same issues on Django 1.5.1 with Gdal 1.10 and Geos 3.3.5 on MacOSX 10.8. And I do not have module-level globals.

[Tim Graham]

@matt.rigal -- please open a new ticket with details.

[Abry Maxime]

Duplicate of https://code.djangoproject.com/ticket/13843
Fixed with ​https://github.com/django/django/commit/8f334e55be99ae271d82f9b961a2dc6258487d9c