High CPU/memory consumtion when a 5xx is raised with a large local variable

[Rémi Dupré]

Hi!

In a view with large variable in scope, if an exception is raised, the worker will freeze and its memory usage will raise. Here is a minimal example:

import sys
from django.urls import path

# 300MB
large_data = list(range(40 * 1024 * 1024))

def compute_lines(request):
    local_data = large_data
    raise ValueError

urlpatterns = [
    path("compute-lines/", compute_lines)
]

When calling /compute-lines/ you will notice the issue.

---

After a bit of investigations, it turned out it comes from django.utils.log.AdminEmailHandler in the default logging config. This handler will eventually pretty print the whole context of each trace frame from here : ​https://github.com/django/django/blob/main/django/views/debug.py#L355

This implementation ensures that no more than 4kB are included in the formatted result, but the entire variable will still have to be rendered first. I'm not sure how it could be solved, but maybe reprlib can be a clue?