#3711 closed (fixed)
document that connection.queries will fill itself until system runs out of memory, and describe work-around
| Reported by: | sdistefano | Owned by: | nobody |
|---|---|---|---|
| Component: | Database layer (models, ORM) | Version: | dev |
| Severity: | Keywords: | connection.queries dict memory | |
| Cc: | Triage Stage: | Accepted | |
| Has patch: | no | Needs documentation: | yes |
| Needs tests: | no | Patch needs improvement: | no |
| Easy pickings: | no | UI/UX: | no |
Description
I found this when running a script which made thousonds of queries looping trough objects (not made to be efficient).
In any case, I think this could also happen just by running the server with DEBUG=True for an extended period of time.
This rendered my computer unusable (reboot required)
Attachments (1)
Change History (12)
comment:1 by , 18 years ago
| Has patch: | set |
|---|---|
| Patch needs improvement: | set |
| Triage Stage: | Unreviewed → Accepted |
by , 18 years ago
| Attachment: | backends.diff added |
|---|
comment:2 by , 18 years ago
The query-logging behavior under DEBUG is documented and intended, and the potential for memory usage is also documented, so I'm not entirely certain there's an issue to be fixed here.
comment:3 by , 18 years ago
This is true, but I don't think Django should take down the computer. However - how *many* queries do you need to do before this happens - we must be talking thousands of them.
comment:4 by , 18 years ago
I agree with Simon. And it his "quick and dirty" patch looks good enough. Perhaps change it to len(self.db.queries) > 1000 just for safety? :D
comment:5 by , 17 years ago
| Needs documentation: | set |
|---|---|
| Triage Stage: | Accepted → Design decision needed |
Some people rely on having all the queries in memory to do some stats (like duplicates and stuff). Maybe we should add a security but hardcoding so low a value seems very restrictive.
comment:6 by , 17 years ago
| Resolution: | → wontfix |
|---|---|
| Status: | new → closed |
I'm going to wontfix per the above comment: we provide clear warnings that queries are logged in memory under DEBUG, and it's easy enough to manually flush it if that's causing you problems.
comment:7 by , 17 years ago
Does anyone know where this is documented? I ran into this issue recently, and my workaround was to set DEBUG to False, but I would love to know how to flush the query log from memory. I couldn't find any clues in the db backend code.
comment:8 by , 17 years ago
| Has patch: | unset |
|---|---|
| Patch needs improvement: | unset |
| Resolution: | wontfix |
| Status: | closed → reopened |
| Summary: | connection.queries will fill itself until system runs out of memory → document that connection.queries will fill itself until system runs out of memory, and describe work-around |
| Triage Stage: | Design decision needed → Accepted |
comment:9 by , 17 years ago
#6734 was opened with another proposal to cap connection.queries. I closed it as a duplicate of this.
comment:10 by , 17 years ago
| Resolution: | → wontfix |
|---|---|
| Status: | reopened → closed |
Guys, this is documented in the FAQ, and both the official docs and several people referencing them show up on the very first page of results for a Google search on "django memory leak"; if people already aren't doing the basic legwork to find the answer, it's unlikely they're looking at the docs at all, and so adding more documentation won't help those people in the least.
Marking wontfix again. Take it to django-dev if you have a convincing argument for why this should change, since ticket trackers suck as a discussion medium.
comment:11 by , 10 years ago
| Resolution: | wontfix → fixed |
|---|
Quick and dirty patch for this to delete older queries once it goes above (an arbitrarily chosen) number of queries, but I'm not sure that this is the best approach.