Add note regarding thread-safety when using PyMemcacheCache.

[Martijn van der Blom]

For thread-safety when using pymemcache the option 'use_pooling': True can be passed via OPTIONS which will make pymemcache.HashClient use pymemcache.PooledClient instead of pymemcache.Client internally.

Some documentation should be added or improved.

---

In our application we were using the MemcachedCache backend to connect to a memcached server. Since this backend will be removed in Django 4.1 we thought we'd migrate to the alternative PyMemcacheCache backend as suggested in the Django documentation at: ​https://docs.djangoproject.com/en/3.2/topics/cache/

After upgrading we encountered several errors when running the application in gunicorn with the gevent worker class.

Summary of errors:

• gevent._socketcommon.cancel_wait_ex: [Errno 9] File descriptor was closed in another greenlet
• gevent.exceptions.ConcurrentObjectUseError: This socket is already used by another greenlet: <bound method Waiter.switch of <gevent._gevent_c_waiter.Waiter object at 0x7f8e77b00a40>>
• OSError: [Errno 9] Bad file descriptor

These errors seem to be related to either the Django backend implementation or Pymemcache not handling multi-threading/thread-safety properly.
There is a related bug for the Pymemcache library where a member of that team states that is up the application using Pymemcache to handle thread-safety (​https://github.com/pinterest/pymemcache/issues/195#issuecomment-452523524). In this case the Django framework. This comment in Django's BaseMemcachedCache implementation indicates that that was the original intent: ​https://github.com/django/django/blob/main/django/core/cache/backends/memcached.py#L38
so i think PyMemcacheCache should handle this.

Example project that reproduces the error:
​https://github.com/mvanderblom/django-memcached-bugreport

For us, this error prevents us from using the PyMemcacheCache backend and thus from upgrading to Django 4.1 when it gets released.

[Martijn van der Blom]

Log file that contains the log from a run of the example project. The log file contains the full stacktraces for this error.

[Mariusz Felisiak]

Thanks for the report. This can be also an issue with asgiref.local.Local. Can you confirm that this issue still exists with asgiref==3.4.1 and on the Django's main branch?

[Nick Pope]

So, yes, pymemcache.Client is not thread-safe. We are using pymemcache.HashClient so that we can support connections to multiple servers.

I note that pymemcache.PooledClient is thread-safe according to the ​documentation. We can pass the use_pooling flag to HashClient. Unfortunately pymemcache's documentation is a little sparse!

If I add 'OPTIONS': {'use_pooling': True} to your CACHES configuration in your reproducer the problem goes away for me.

Would you be prepared to open a PR with a tweak to the documentation? I already mentioned use_pooling in at the end of the ​cache arguments section, so maybe we just need to amend the sentence before?

[Martijn van der Blom]

Logfile of a run with use_pooling set to true