Geodjango running through memory

[Taylor]

I have been trying to upload a raster to my postgis backend and keep running into

django.db.utils.OperationalError: cannot allocate memory for output buffer

I try to upload the file in the django shell using the following:

raster = GDALRaster('rasters/data/my_image.tif', write=True) 
r = RastModel(name="Foo", image=raster)
r.save()

The image is only about a gb and I have plenty of ram to handle it. When I save the raster it runs through almost all 32gb of ram and then throws the error. From all the docs and posts I have seen it appears that I am uploading the raster correctly so I am assuming there is a bug somewhere.

>>> rast = GDALRaster('rasters/data/raw_tempe.tif', write=True)
>>> r = Rast(name='Test', image=rast)
>>> r.save()
Traceback (most recent call last):
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
psycopg2.OperationalError: cannot allocate memory for output buffer


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/models/base.py", line 718, in save
    force_update=force_update, update_fields=update_fields)
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/models/base.py", line 748, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/models/base.py", line 831, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/models/base.py", line 869, in _do_insert
    using=using, raw=raw)
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/models/query.py", line 1136, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1289, in execute_sql
    cursor.execute(sql, params)
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/backends/utils.py", line 100, in execute
    return super().execute(sql, params)
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/backends/utils.py", line 68, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/mnt/f/Projects/gis322/env/lib/python3.6/site-packages/django/db/backends/utils.py", line 85, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.OperationalError: cannot allocate memory for output buffer

[Simon Charette]

Please provide a full traceback so we can determine in which phase the process runs out of memory. I assume it's happening at on the save() call?

Also while I'm not that familiar with GDAL rasters I know that TIFF images can be heavily compressed so it's possible that even if the compressed file is only 1GB it could be way larger uncompressed.

[Taylor]

Replying to Simon Charette:

Please provide a full traceback so we can determine in which phase the process runs out of memory. I assume it's happening at on the save() call?

Also while I'm not that familiar with GDAL rasters I know that TIFF images can be heavily compressed so it's possible that even if the compressed file is only 1GB it could be way larger uncompressed.

I added the traceback above let me know if I can supply anything else to help resolve the issue. If wanted I could supply the image as well since it is freely available landsat imagery.