Opened 4 months ago

Closed 4 months ago

Last modified 4 months ago

#35672 closed Bug (invalid)

Using database connection pooling, threads do not return the connection

Reported by: André S. Hansen Owned by:
Component: Database layer (models, ORM) Version: 5.1
Severity: Normal Keywords:
Cc: Florian Apolloner Triage Stage: Unreviewed
Has patch: no Needs documentation: no
Needs tests: no Patch needs improvement: no
Easy pickings: no UI/UX: no

Description (last modified by André S. Hansen)

I am testing the new db connection pooling, which increased the performance greatly, typically lowering api response times from 30ms -> 20ms on my dev env.

However, after a while i started getting 

psycopg_pool.PoolTimeout : couldn't get a connection after 30.00 sec

Further investigation shows the reason was a Threaded Middleware orm interaction, like this simplified code. My theory is that the thread request and receives a connection from the pool, but do not return it.

class TimeMonitorMiddleware(object):
    batched_logs: List[RequestLog] = []

    def __call__(self, request: WSGIRequest) -> HttpResponse:
        # ...
  
        # Save batched logs
        if len(self.batched_logs) >= 50:
            Thread(target=RequestLog.objects.bulk_create, args=(batched_logs,)).start()

        return response

Using settings

{
   "ENGINE": os.environ.get("DB_ENGINE", "django.db.backends.postgresql"),
   "OPTIONS": {"pool": True}
}

Extra info:

  • Python 3.12.1
  • Django 5.1
  • psycopg 3.2.1
  • psycopg-pool 3.2.2
  • Local docker image hosted postgresql database using image "postgres"

Change History (16)

comment:1 by André S. Hansen, 4 months ago

Description: modified (diff)

comment:2 by Sarah Boyce, 4 months ago

Cc: Florian Apolloner added
Resolution: → needsinfo
Status: new → closed

Hi Andre, I think we need more information to understand whether/where Django is at fault here

I believe you have this error because the connection pool is full, this could mean that Django is not closing connections after using them. For us to be able to identify where this is happening, can you share a minimal Django project and steps to reproduce?

comment:3 by Florian Apolloner, 4 months ago

Hi André, it would probably help if you enabled the psycopg.pool logger and then check if the connections are returned to the pool properly. Also what is your request rate and how long are the requests taking? How are you running the django server? What are the postgresql connection limits?

comment:4 by André S. Hansen, 4 months ago

I did some more testing and figured out the source of the problem. When i disabled a Middleware Threaded connection to the database, the bug stopped happening. My current theory is that the thread dont return the connection to the pool. I am investigating now how i should properly do this with Threads.

Simplified version of code

class TimeMonitorMiddleware(object):
    batched_logs: List[RequestLog] = []

    def __call__(self, request: WSGIRequest) -> HttpResponse:
        # ...
  
        # Save batched logs
        if len(self.batched_logs) >= 50:
            Thread(target=RequestLog.objects.bulk_create, args=(batched_logs,)).start()

        return response
Last edited 4 months ago by André S. Hansen (previous) (diff)

comment:5 by André S. Hansen, 4 months ago

Summary: Using database connection pooling raises PoolTimeout after a while → Using database connection pooling with threads raises PoolTimeout after a while

comment:6 by André S. Hansen, 4 months ago

Summary: Using database connection pooling with threads raises PoolTimeout after a while → Using database connection pooling, threads do not return the connection

comment:7 by André S. Hansen, 4 months ago

Description: modified (diff)
Resolution: needsinfo
Status: closed → new

comment:8 by André S. Hansen, 4 months ago

Description: modified (diff)

comment:9 by André S. Hansen, 4 months ago

Description: modified (diff)

Example error logs

Logs

[13/Aug/2024 00:50:41] "POST /api/instances/2/actions/move/ HTTP/1.1" 200 91
couldn't get a connection after 30.00 sec
Traceback (most recent call last):
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\backends\base\base.py", line 279, in ensure_connection
    self.connect()
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\utils\asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\backends\base\base.py", line 256, in connect
    self.connection = self.get_new_connection(conn_params)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\utils\asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\backends\postgresql\base.py", line 348, in get_new_connection
    connection = self.pool.getconn()
                 ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\psycopg_pool\pool.py", line 202, in getconn
    raise PoolTimeout(
psycopg_pool.PoolTimeout: couldn't get a connection after 30.00 sec

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\ninja\operation.py", line 192, in _run_authentication
    result = callback(request)
             ^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\ninja\security\apikey.py", line 23, in __call__
    return self.authenticate(request, key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\ninja\security\session.py", line 17, in authenticate
    if request.user.is_authenticated:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\utils\functional.py", line 251, in inner
    self._setup()
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\utils\functional.py", line 404, in _setup
    self._wrapped = self._setupfunc()
                    ^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\contrib\auth\middleware.py", line 37, in <lambda>
    request.user = SimpleLazyObject(lambda: get_user(request))
                                            ^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\contrib\auth\middleware.py", line 17, in get_user
    request._cached_user = auth.get_user(request)
                           ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\contrib\auth\__init__.py", line 209, in get_user
    user_id = _get_user_session_key(request)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\contrib\auth\__init__.py", line 62, in _get_user_session_key
    return get_user_model()._meta.pk.to_python(request.session[SESSION_KEY])
                                               ~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\contrib\sessions\backends\base.py", line 55, in __getitem__
    return self._session[key]
           ^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\contrib\sessions\backends\base.py", line 249, in _get_session
    self._session_cache = self.load()
                          ^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\contrib\sessions\backends\db.py", line 55, in load
    s = self._get_session_from_db()
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\contrib\sessions\backends\db.py", line 34, in _get_session_from_db
    return self.model.objects.get(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\models\manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\models\query.py", line 645, in get
    num = len(clone)
          ^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\models\query.py", line 382, in __len__
    self._fetch_all()
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\models\query.py", line 1928, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\models\query.py", line 91, in __iter__
    results = compiler.execute_sql(
              ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\cachalot\monkey_patch.py", line 37, in inner
    return original(compiler, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\cachalot\monkey_patch.py", line 94, in inner
    return execute_query_func()
           ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\cachalot\monkey_patch.py", line 80, in <lambda>
    execute_query_func = lambda: original(compiler, *args, **kwargs)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\models\sql\compiler.py", line 1572, in execute_sql
    cursor = self.connection.cursor()
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\utils\asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\backends\base\base.py", line 320, in cursor
    return self._cursor()
           ^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\backends\base\base.py", line 296, in _cursor
    self.ensure_connection()
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\utils\asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\backends\base\base.py", line 278, in ensure_connection
    with self.wrap_database_errors:
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\backends\base\base.py", line 279, in ensure_connection
    self.connect()
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\utils\asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\backends\base\base.py", line 256, in connect
    self.connection = self.get_new_connection(conn_params)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\utils\asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\backends\postgresql\base.py", line 348, in get_new_connection
    connection = self.pool.getconn()
                 ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\psycopg_pool\pool.py", line 202, in getconn
    raise PoolTimeout(
django.db.utils.OperationalError: couldn't get a connection after 30.00 sec
Internal Server Error: /api/instances/2/actions/move/
Traceback (most recent call last):
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\backends\base\base.py", line 279, in ensure_connection
    self.connect()
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\utils\asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\backends\base\base.py", line 256, in connect
    self.connection = self.get_new_connection(conn_params)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\utils\asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\backends\postgresql\base.py", line 348, in get_new_connection
    connection = self.pool.getconn()
                 ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\psycopg_pool\pool.py", line 202, in getconn
    raise PoolTimeout(
psycopg_pool.PoolTimeout: couldn't get a connection after 30.00 sec

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\core\handlers\exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Repos\untitledgame\backend\monitor\middleware.py", line 35, in __call__
    user_id=user_id_or_none(request.user),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Repos\untitledgame\backend\game\api\utils.py", line 9, in user_id_or_none
    if isinstance(user, AnonymousUser):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\utils\functional.py", line 280, in __getattribute__
    value = super().__getattribute__(name)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\utils\functional.py", line 251, in inner
    self._setup()
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\utils\functional.py", line 404, in _setup
    self._wrapped = self._setupfunc()
                    ^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\contrib\auth\middleware.py", line 37, in <lambda>
    request.user = SimpleLazyObject(lambda: get_user(request))
                                            ^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\contrib\auth\middleware.py", line 17, in get_user
    request._cached_user = auth.get_user(request)
                           ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\contrib\auth\__init__.py", line 209, in get_user
    user_id = _get_user_session_key(request)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\contrib\auth\__init__.py", line 62, in _get_user_session_key
    return get_user_model()._meta.pk.to_python(request.session[SESSION_KEY])
                                               ~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\contrib\sessions\backends\base.py", line 55, in __getitem__
    return self._session[key]
           ^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\contrib\sessions\backends\base.py", line 249, in _get_session
    self._session_cache = self.load()
                          ^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\contrib\sessions\backends\db.py", line 55, in load
    s = self._get_session_from_db()
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\contrib\sessions\backends\db.py", line 34, in _get_session_from_db
    return self.model.objects.get(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\models\manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\models\query.py", line 645, in get
    num = len(clone)
          ^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\models\query.py", line 382, in __len__
    self._fetch_all()
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\models\query.py", line 1928, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\models\query.py", line 91, in __iter__
    results = compiler.execute_sql(
              ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\cachalot\monkey_patch.py", line 37, in inner
    return original(compiler, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\cachalot\monkey_patch.py", line 94, in inner
    return execute_query_func()
           ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\cachalot\monkey_patch.py", line 80, in <lambda>
    execute_query_func = lambda: original(compiler, *args, **kwargs)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\models\sql\compiler.py", line 1572, in execute_sql
    cursor = self.connection.cursor()
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\utils\asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\backends\base\base.py", line 320, in cursor
    return self._cursor()
           ^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\backends\base\base.py", line 296, in _cursor
    self.ensure_connection()
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\utils\asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\backends\base\base.py", line 278, in ensure_connection
    with self.wrap_database_errors:
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\backends\base\base.py", line 279, in ensure_connection
    self.connect()
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\utils\asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\backends\base\base.py", line 256, in connect
    self.connection = self.get_new_connection(conn_params)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\utils\asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\django\db\backends\postgresql\base.py", line 348, in get_new_connection
    connection = self.pool.getconn()
                 ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\andre\.virtualenvs\backend-jnn8gBG-\Lib\site-packages\psycopg_pool\pool.py", line 202, in getconn
    raise PoolTimeout(
django.db.utils.OperationalError: couldn't get a connection after 30.00 sec
[13/Aug/2024 00:56:42] "POST /api/instances/2/actions/move/ HTTP/1.1" 500 235606

comment:10 by André S. Hansen, 4 months ago

I edited the ticket with the threaded point, and reopened. I hope this is ok.

Last edited 4 months ago by André S. Hansen (previous) (diff)

comment:11 by André S. Hansen, 4 months ago

Description: modified (diff)

comment:12 by Florian Apolloner, 4 months ago

Hi André, I think this might be intended behavior. You need to close connections. During a request cycle Django does it for you. But stuff like celery etc also have to manually close connections. I assume your code was working before without the pool? What was the configuration then? Especially did you have persistent connections enabled? If yes I think you'd see the same behavior there.

comment:13 by Florian Apolloner, 4 months ago

Addendum to the comment before: The behavior before should have been the same with and without persistent connections. Though I might have an idea why you might not have seen this before (I am just guessing though, I don't know the details of our thread local implementations out of my head). Before thread ids might have gotten reused, so while you were leaking connections as well other requests might have spawned with the same thread id. Now with the pool it is very important that you close the connection at the end of your code to return it.

comment:14 by André S. Hansen, 4 months ago

Thanks for considering my case, and your questions and input, it helped push me toward the solution.

I assume your code was working before without the pool?

Yes

What was the configuration then? Especially did you have persistent connections enabled?

I did not specify CONN_MAX_AGE, meaning i had persistent connections disabled.

---

After an hour of googling and testing I ​found a decent solution:

from django.db import connection

class TestThread(Thread):
    def run(self) -> None:
        super().run()
        connection.close()

TestThread(target=RequestLog.objects.bulk_create, args=(batched_logs,)).start()

The documentation ​https://docs.djangoproject.com/en/5.1/ref/databases/ could do well addressing this case. But i guess you can close this ticket.

Last edited 4 months ago by André S. Hansen (previous) (diff)

comment:15 by Florian Apolloner, 4 months ago

Resolution: → invalid
Status: new → closed

Ok, I will close this as invalid then.

The documentation ​​https://docs.djangoproject.com/en/5.1/ref/databases/ could do well addressing this case. But i guess you can close this ticket.

Don't the caveats in ​https://docs.djangoproject.com/en/5.1/ref/databases/#caveats mention your case exactly:

If a connection is created in a long-running process, outside of Django’s request-response cycle, the connection will remain open until explicitly closed, or timeout occurs. You can use django.db.close_old_connections() to close all old or unusable connections.

If you can think of a better wording please open a PR; we don't need tickets for doc improvements usually.

comment:16 by André S. Hansen, 4 months ago

Hehe, I'd be damned, I scanned that document twice, embarrassing.

Made a little PR here: ​https://github.com/django/django/pull/18495

Note: See TracTickets for help on using tickets.
Back to Top