High CPU/memory consumption when a 5XX is raised with large local variables

[Rémi Dupré]

Hi!

In a view with large variable in its scope, if an exception is raised, the worker will freeze and its memory usage will raise. Here is a minimal example:

import sys
from django.urls import path

# 300MB
large_data = list(range(40 * 1024 * 1024))

def compute_lines(request):
    local_data = large_data
    raise ValueError

urlpatterns = [
    path("compute-lines/", compute_lines)
]

When calling /compute-lines/ you will notice the issue : the server will take a few seconds to responds while its memory raises by ~1GB.

---

After a bit of investigations, it turned out it comes from django.utils.log.AdminEmailHandler in the default logging config. This handler will eventually pretty print the whole context of each trace frame from here : ​https://github.com/django/django/blob/main/django/views/debug.py#L355

This implementation ensures that no more than 4kB are included in the formatted result, but the entire variable will still have to be rendered first. I'm not sure how it could be solved, but maybe ​https://docs.python.org/3/library/reprlib.html can be a clue?

[Natalia Bidart]

I have managed to reproduce the issue, thank you for the example. Indeed if v = pprint(v) is replaced by something that does not render the whole variable, the memory and CPU usage do not spike.

Accepting so we can evaluate whether an alternative could be used instead of pprint that would account for the size limit before printing.

[Natalia Bidart]

After some experimentation following the suggested link to reprlib, perhaps something like this could be a good alternative? (it needs better code formatting, more reprlib limits sets and tests, of course)

django/views/debug.py

@@ -347 +347 @@
             self.template_does_not_exist = True
             self.postmortem = self.exc_value.chain or [self.exc_value]
 
+        import reprlib
+        repr_instance = reprlib.Repr()
+        repr_instance.maxstring = 5000
+        repr_instance.maxlist = 1000
+
         frames = self.get_traceback_frames()
         for i, frame in enumerate(frames):
             if "vars" in frame:
                 frame_vars = []
                 for k, v in frame["vars"]:
-                    v = pprint(v)
+                    v = repr_instance.repr(v)
                     # Trim large blobs of data
                     if len(v) > 4096:
                         v = "%s… <trimmed %d bytes string>" % (v[0:4096], len(v))

[Natalia Bidart]

​PR

[Adam Johnson]

I think using reprlib is acceptable, this is exactly the kind of situation it’s designed for. It will be a shame to lose the pretty-printing aspect, but perhaps reprlib/pprint will grow appropriate functionality in the future.

[Yash Kumar Verma]

Hello folks, I saw the two pull requests that were attached in the ticket are stale, and the original author have both closed. I'm interesting in picking this as my first ticket, can I proceed?

[Yash Kumar Verma]

I was:

• able to verify the following
  • when exception is raised after declaring a large variable, memory usage shoots up.
  • when exception is not raised after declaring a large variable, memory remains in live with normal operations.

Setup I used to reproduce the error and suggested solution

• added a middleware which logs the change in memory after each request.
• created two views, one which throws error and one that does not.
• observed the memory usage in both the cases.

Current

After using reprlib

[Mariusz Felisiak]

You should not mark your own PRs as "ready for checkin".

[Keerthi Vasan S A]

Ah sorry, I wanted to set this ticket as ready to be reviewed as mentioned by a contributor in the PR. How do I do that?

[Natalia Bidart]

Replying to Keerthi Vasan S A:

Ah sorry, I wanted to set this ticket as ready to be reviewed as mentioned by a contributor in the PR. How do I do that?

Hi!

The proper procedure to get a re-review is described in ​the PR review checklist. Basically you need to unset the relevant flags in this ticket so the PR gets added back to the "patches needing review" section of the ​Django Developer Dahsboard.

[Keerthi Vasan S A]

Gotcha, thank you!

[Ahmed Ibrahim]

Is this still being worked on or should I handle t?

[Natalia Bidart]

Replying to Ahmed Ibrahim:

Is this still being worked on or should I handle t?

Hello Ahmed! Thanks for your eagerness to help. As we already discussed in the PR, this is still being work on and I'll provide more feedback in the PR.