djangoadmin loaddata gives odd error if it runs out of memory

[chrisw]

Hi All,

I tried to import a 3.2GB xml file, partly to see if Django loads the whole file into memory when it's importing and partly to see how large pg client transactions work. Whichever caused the problem, I have a fair idea that even thought the machine has 4GB of memory, the process ran out of memory.

I'd expect a MemoryError if that were the case, what I actually got was:

  File "django/core/management/__init__.py", line 362, in execute_manager
    utility.execute()
  File "/django/core/management/__init__.py", line 303, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "django/core/management/base.py", line 195, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "django/core/management/base.py", line 222, in execute
    output = self.handle(*args, **options)
  File "django/core/management/commands/loaddata.py", line 200, in handle
    transaction.leave_transaction_management()
  File "django/db/transaction.py", line 74, in leave_transaction_management
    raise TransactionManagementError("This code isn't under transaction management")
django.db.transaction.TransactionManagementError: This code isn't under transaction management

cheers,

Chris

[Russell Keith-Magee]

For those looking to dig into this, it's important to identify whether this is a database issue (i.e., the database runs out of memory if you try to do a 3GB transaction) or a Django issue (i.e., a problem with our wrapper loading the data file into memory).

Its also important to note the analog with #5423 - that ticket deals with making dumpdata behave as an iterator. I'm not marking this as a dupe because the solution won't necessarily be the same, but it's worth noting if someone wants to tackle both problems.

[Alex Gaynor]

Please use preview