psycopg3 warns about connections not being closed

[HTErik]

When running

• a Django 5.0.6 application
• psycopg 3.1.19
• with db "ENGINE": "django.db.backends.postgresql"
• with Python warnings as errors enabled

This occasionally prints following error from psycopg:

ResourceWarning: connection <psycopg.Connection [IDLE] (REDACTED) at 0x710d29abe120> was deleted while still open. Please use 'with' or '.close()' to close the connection

This warning comes from the __del__ function in psycopg.connection.BaseConnection.

I have no idea how to reproduce this.
I tried patching psycopg to print a stack trace whenever a connection is opened and closed. This tells me that the connection that was deleted before being closed, was opened through the following code path:

  File "blah.py", line 193, in __heartbeat
    MyModel.objects.bulk_update(items_to_run, ["last_update"])
  File "python3.12/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "python3.12/site-packages/django/db/models/query.py", line 922, in bulk_update
    with transaction.atomic(using=self.db, savepoint=False):
  File "python3.12/site-packages/django/db/transaction.py", line 198, in __enter__
    if not connection.get_autocommit():
  File "python3.12/site-packages/django/db/backends/base/base.py", line 450, in get_autocommit
    self.ensure_connection()
  File "python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "python3.12/site-packages/django/db/backends/base/base.py", line 275, in ensure_connection
    self.connect()
  File "python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "python3.12/site-packages/django/db/backends/base/base.py", line 256, in connect
    self.connection = self.get_new_connection(conn_params)

[HTErik]

I have identified the problem now.
It happens when one performs Django queries from a separate thread. The DB connections in Django are thread local inside the ConnectionHandler, so when the thread finishes and is eventually GCd, the thread locals holding the DB connections in Django are also GCd, without any prior close().

To solve it, one must call this in a finally clause of all threads that run Django queries.

for conn in django.db.connections.all():
    conn.close()

I'm not sure if this is expected from users or if this is something Django should handle internally.
The documentation on this topic could use some more details. ​https://docs.djangoproject.com/en/5.0/ref/databases/#connection-management

[Sarah Boyce]

Hi HTErik, I recommend you share more details of this in the async section of the forum: ​https://forum.djangoproject.com/c/internals/async/8, as this is the place for discussion and support.
If there's an agreement that there is a bug in Django or that the docs need updating, come back and I'll be happy to accept improvements 👍