PyMemcacheCache backend fails when running as a wsgi application with gevent worker class

[Martijn van der Blom]

In our application we were using the MemcachedCache backend to connect to a memcached server. Since this backend will be removed in Django 4.1 we thought we'd migrate to the alternative PyMemcacheCache backend as suggested in the Django documentation at: ​https://docs.djangoproject.com/en/3.2/topics/cache/

After upgrading we encountered several errors when running the application in gunicorn with the gevent worker class.

Summary of errors:

• gevent._socketcommon.cancel_wait_ex: [Errno 9] File descriptor was closed in another greenlet
• gevent.exceptions.ConcurrentObjectUseError: This socket is already used by another greenlet: <bound method Waiter.switch of <gevent._gevent_c_waiter.Waiter object at 0x7f8e77b00a40>>
• OSError: [Errno 9] Bad file descriptor

These errors seem to be related to either the Django backend implementation or Pymemcache not handling multi-threading/thread-safety properly.
There is a related bug for the Pymemcache library where a member of that team states that is up the application using Pymemcache to handle thread-safety (​https://github.com/pinterest/pymemcache/issues/195#issuecomment-452523524). In this case the Django framework. This comment in Django's BaseMemcachedCache implementation indicates that that was the original intent: ​https://github.com/django/django/blob/main/django/core/cache/backends/memcached.py#L38
so i think PyMemcacheCache should handle this.

Example project that reproduces the error:
​https://github.com/mvanderblom/django-memcached-bugreport

For us, this error prevents us from using the PyMemcacheCache backend and thus from upgrading to Django 4.1 when it gets released.

[Martijn van der Blom]

Log file that contains the log from a run of the example project. The log file contains the full stacktraces for this error.