Opened 13 years ago

Closed 4 years ago

#17943 closed Bug (needsinfo)

Too many open file descriptors while using memcache

Reported by: m.gajda@… Owned by: Przemek Lewandowski
Component: Core (Cache system) Version: 1.4
Severity: Normal Keywords: cache memcache open file descriptors
Cc: Florian Apolloner Triage Stage: Unreviewed
Has patch: yes Needs documentation: no
Needs tests: no Patch needs improvement: yes
Easy pickings: no UI/UX: no

Description

Hi all,

I am using Django 1.3.1 with memcached server under Linux (PLD distro). Django application is run with runfcgi command (maxspare set to 20). After switching to Django 1.3.1 (from line 1.2.x) I encountered very strange behaviour -- the app became crashing after some time.

What I have discovered is very large number of open socket connections made by the Django app to the memcached server (lsof show hundrets of open descriptors). After some time the app process had been killed by the system, due to exceeded number of opened file descriptors.

Look at Google pointed me to the ticket #15324. Since that patch has been incorporated in the Django 1.3.1, I have started digging into memcached Django wrapper over the pyton-memcached module (django.core.cache.backends.memcached). What I found is that Django app run in multithreaded mode creates new cache object for every thread. This is probably good information, since it has allowed avoid difficult race conditions. What is sad - after closing/releasing the thread, the cache object is somehow not deleted properly. I mean it seems that it does not clean up all gathered resources properly. This lasts with large number of opened file descriptors in the memcached case.

I have added simple patch fixing said behaviour. It works quite well for me, thus I assume I could be considered to be improved and somehow merged with the Django sources. I have not created any tests, I am sorry for that.

Attachments (1)

memcached-close-sockets-patch.diff (475 bytes ) - added by m.gajda@… 13 years ago.

Download all attachments as: .zip

Change History (10)

by m.gajda@…, 13 years ago

comment:1 by Jannis Leidel, 12 years ago

Needs documentation: set
Needs tests: set
Triage Stage: UnreviewedAccepted
Version: 1.31.4

comment:2 by m.gajda@…, 12 years ago

I have recently upgraded my Django environment from 1.3.1 to the most recent 1.4.5. The issue described above still persists (dozens of unclosed connections to the memcache). The patch sorts it out perfectly.

comment:3 by Przemek Lewandowski, 12 years ago

Owner: changed from nobody to Przemek Lewandowski
Status: newassigned

comment:4 by Przemek Lewandowski, 12 years ago

Needs documentation: unset
Needs tests: unset

I added test to the patch. Please see: https://github.com/django/django/pull/822

comment:5 by Florian Apolloner, 12 years ago

The connection is closed at the end of the request, can you explain how and why this issue occurs? Using __del__ shouldn't be the solution.

comment:6 by Florian Apolloner, 12 years ago

Cc: Florian Apolloner added

comment:7 by Tim Graham, 11 years ago

Patch needs improvement: set

comment:8 by m.gajda@…, 11 years ago

I have recently updated my Django environment to 1.5.5, and then to 1.6.1. In both cases the above mentioned issues still persists. The patch attached helps.

comment:9 by Mariusz Felisiak, 4 years ago

Resolution: needsinfo
Status: assignedclosed
Triage Stage: AcceptedUnreviewed

Marking as "needsinfo" due to Florian's comment.

Note: See TracTickets for help on using tickets.
Back to Top