Add a RasterField to GeoDjango

[Daniel Wiesmann]

Raster integrations is becoming stronger for database back-ends such as PostGIS, so an integration of raster data into Django would open new possibilities for online mapping. Handling raster data in online environments is currently hard to accomplish and integration into a web framework would make this easier in many cases.

Adding a RasterField makes sense as a first step to handling raster data input and output. Spatial querying and other operations could then be built on top of that.

[Claude Paroz]

(Daniel is author of ​https://github.com/geodesign/django-raster/)

[Daniel Wiesmann]

I have made good process in creating some basic raster functionality, including gdal bindings, a GDALRaster object, a RasterField and a Tiler engine to create tiles from GDALRasters, and tests for most of it.

For now I have kept all new code in a submodule of contrib/gis/gdal, see

​https://github.com/geodesign/django/tree/gdalraster/django/contrib/gis/gdal/raster

Now I was thinking of moving the module to master and starting to write the documentation for it. But I am not sure if it would be better to integrate it directly into the contrib.gis.gdal module. In the gdal.raster module, I have mimicked the structure of that module (for instance by creating a prototypes/ds.py file with the ctypes gdal bindings).

Does anyone have time to give me some advice or discuss how to proceed from here?

Note: I branched off the stable/1.7.x branch because like this I am able to use the code in one of my projects and play with it in a full project development environment.

[Claude Paroz]

Nice to see quick progress!

I think that we could aim for more integration with main gdal module. For example, when I compare driver.py and raster/driver.py, I see much code duplication. The RasterField model should live somewhere in gis/db/models (maybe by creating a fields module). I also would like to see all prototypes in gdal/prototypes and tests in gdal/tests. Maybe the other files could be left in a raster module. These are just quick thoughts, without deep looking in the code.

[Claude Paroz]

Daniel, look at this pull request: ​https://github.com/django/django/pull/3648

Could you review/comment on it, and tell if you think it's going in a good direction?

[Claude Paroz <claude@…>]

In 4df3a3e0e9bfcd64a3e2eb85e0d4449d7b8f4f6e:

Added GDAL prototypes for raster support

Refs #23804.

[Claude Paroz <claude@…>]

In 00fa1474d7d2dbd5530a650ea9fe09f36bff77c7:

Added raster support for GDAL Driver class

Based on Daniel Wiesmann's work. Refs #23804.

[Claude Paroz]

The base is now in. I'll continue to work on integrating Daniel's branch.

[Daniel Wiesmann]

I branched off master and started integrating the raster files into the structure with the new driver. I followed your suggestions above. The new branch is this one:

​https://github.com/geodesign/django/tree/raster/django/contrib/gis/gdal

• Moved all tests into gdal.tests folder
• Separated constants from utils functions
• Moved RasterField into django.contrib.gis.db.models.fields module

Most of the changes are file-rearangements.

The raster field can leverage some of the features around the GeometryFields, but here some of the code that can be re-used is named Geometry* for instance the GeometryProxy and the geom_type property. Both are related to lazy-instanciation of field target instances (Rasters or Geometries).

See:

​https://github.com/geodesign/django/blob/raster/django/contrib/gis/db/models/fields.py#L356
​https://github.com/geodesign/django/blob/raster/django/contrib/gis/db/models/fields.py#L390

[Daniel Wiesmann]

The RasterField is still very minimalistic, here are some thoughts of what is missing/could be improved:

• Integration with different db backends. Currently the raster IO is tailored towrards PostGIS. The value passed to the db is the output of the wkb property of the GDALRaster, which converts between PostGIS WKB and GDALRaster. This seems to work in the test-databases. The db_type is fixed to 'raster', which is also PostGIS specific.

• A deeper integration with the GeometryField would be beneficial. For instance the SRID property could be shared in a master class. But this is currently integrated into GeometryField directly. So integrating the two would probably require a GeoField master class or so.

[Claude Paroz]

I've prepared a new (small) step, with the following patch. I've deliberately chosen some different paths from your branch. It's not to contradict you by pleasure :-), I may totally be wrong or naive with certain choices, but I'm trying to create the most Pythonesque API as possible. Feel free to criticize and contradict me in return!

​https://github.com/django/django/pull/3694

[Daniel Wiesmann]

Just had a quick look, this is awsome, its very clean like this. I can't resist to add a quote of a movie I saw the other day:

"You've got to understand your limitations. It's your limitations that make you the wonderful disaster that you most probably are. For me that is where collaboration comes in. To take an idea that is blind and unformed and that has been hatched largely in solitude and allow these strange collaborator creatures to morph it into something else, something that is better, that's really something to see."

-- Nick Cave on collaboration from 20,000 Days on Earth

My main concern for now on this is that it limits the data source to be only from a file. You can't instantiate a source from memory, which is what I had intended to use in the raster field. And in-memory rasters work just the same way as the file based ones.

Were you thinking of having a separate GDALRaster object that can be created from memory and used in a field? If you have time, maybe you could briefly summarize how you would see the rest of the code to look like.

Will have a closer look over the next few days.

[Claude Paroz]

Thanks for the quote :-)

My main concern for now on this is that it limits the data source to be only from a file.

Don't fear, it's just a technique, going step after step. Of course, the current functionality is almost unusable for anything serious. And like you I'm looking forward to new possibilities. Don't stop to write code, I wouldn't be able to write anything without taking inspiration from your work, seriously.
Now I may at some point question the extent of the code needed to be in the framework and the code which could be better in an external project. For me the RasterField code is clearly an objective at short term. But the tiling functionality, for example, I'm still not completely convinced it belongs to Django core. We'll see.

[Daniel Wiesmann]

I am aware that some of the code in the raster branch might be too high level for Django core and am happy to discuss this. Here is a start:

For the img property
I kept the img property as part of it as this might be needed for showing the raster in the admin interface.

For the Tiler egine/utility
I envisioned the Tiler engine to be the raster version of the LayerMapping utility (see link below). For larger rasters, it is unpractical to load them as one single raster, so in many use cases some sort of tiling will be necessary. The tiler I wrote is very focused on creating tiles for TMS, which is probably not general enough as a tiling utitlity. Still I think something along those lines is necessary for loading large rasters from files into a db.

​https://github.com/django/django/blob/master/django/contrib/gis/utils/layermapping.py

[Tim Graham <timograham@…>]

In f3eed95175f8a8a365218a09fdc2dc864ff75d0c:

Removed netCDF from GIS driver testing; refs #23804.

It may not be installed on all systems.

[Claude Paroz <claude@…>]

In 6e08bde8c4525dda7d82bbf55b4b45a6e16213da:

Added RasterSource/GDALBand GDAL objects

Based on Daniel Wiesmann's raster branch. Thanks Daniel Wiesmann
and Tim Graham for the reviews. Refs #23804.

[Claude Paroz]

Note that to add the RasterField, we'll probably be facing the same issue as #23879 (skipping model creation for non PostGIS).

[Daniel Wiesmann]

I was working on integrating pixel value read and write support. When you have time, please have a look at the following pull request:

​https://github.com/django/django/pull/3909

[Claude Paroz]

Thanks for the great work, I will definitely put that in my review list.
Unfortunately, this will not be included in Django 1.8 (currently feature frozen), but let's aim for a bright raster support in Django 1.9 (including the raster db field, hopefully)!

[Daniel Wiesmann]

I added a HEX representation for the GDAL Rasters to the above pull request. Rasters can be instantiated from HEX data and converted into that format too.

I was trying to find a standard for Raster data in binary/hex but unfortunately such a standard does not seem to exist.

​http://gis.stackexchange.com/questions/130209/is-there-an-open-standard-for-raster-information
​http://lists.osgeo.org/pipermail/gdal-dev/2015-January/040807.html

So I guess to create a RasterField, the conversion to the different databases will be quite different (I think for geometries, the OpenGIS specification is used in most spatial databases). I would propose to focusing on PostGIS support for a RasterField first and add sqlite and oracle afterwards.

[Daniel Wiesmann]

I added a first take at a RasterField for PostGIS backends to the pull request.

The testing is very basic and I have not yet written any documentation for it and it lacks proper Index creation. However, I wanted to get feedback on the basic implementation before workin on it more.

[Claude Paroz]

Sorry, I was busy with #24214 which is no1 priority for GIS in 1.9. I should have time to review part of your work soon.

[Claude Paroz <claude@…>]

In f269c1d6:

Added write support for GDALRaster

• Instantiation of GDALRaster instances from dict or json data.
• Retrieve and write pixel values in GDALBand objects.
• Support for the GDALFlushCache in gdal C prototypes
• Added private flush method to GDALRaster to make sure all data is written to files when file-based rasters are changed.
• Replaced ptr with _ptr for internal ptr variable

Refs #23804. Thanks Claude Paroz and Tim Graham for the reviews.

[Tim Graham <timograham@…>]

In b9cb815:

Made SRID a required parameter for GDALRaster instantiation; refs #23804.

Earlier versions of GDAL do not allow the srid to be set to 0,
so it should be a required parameter to ensure compatibility.

[Daniel Wiesmann]

Here is a first draft of a Raster field for PostGIS >= 2.0 backends. I am not sure if I covered all the important aspects of creating a field, but it should serve as a starting point at least.

​https://github.com/django/django/pull/4339

[Tim Graham <timograham@…>]

In 0d9b018e:

Fixed gis test failures when numpy isn't installed.

Thanks to Bas Peschier for pointing this out. Refs #23804.

[Daniel Wiesmann]

I updated the pull request for the first RasterField implementation. The raster field should now be fairly complete, but it became quite large due to that. Its hard to split things at this point though, as the different elements are quite interrelated. I hope its still an acceptable size.

Here is a description of the current state:

• The field is implemented for PostGIS backends only.

• Similar to GeometryFields, the field srid can be specified and the raster will automatically be transformed to the field srid upon saving.

• A spatial index for the raster is built by default, and can be disabled with spatial_index=False.

• The null and blank flags can be specified.

• Lazy instantiation works for the raster field as well.

• Tests are in place for the RasterField and corresponding migration commands and operations

For this, I made some changes on the existing codebase:

• Generalized the GeometryProxy and renamed to SpatialProxy. This is used for lazy-instantiation of objects related to spatial fields.

• Override the get_indexes introspection function for postgis, as the spatial indices are based on expressions and are not straight indices (using ST_ConvexHull). So the sql code retrieving the list of indices has to be adopted to that.

• Update the get_placeholder function in the PostGISOperator to implicitly reproject rasters that are passed to a field with a different srid than the field srid.

• Change in PostGISSchemaEditor to create spatial indices on raster (wrapping the index creation sql with the ST_ConvexHull function).

• Split parts of the GeometryField into a generalized BaseSpatialField class. The GeometryField and the RasterField subclass the BaseSpatialField.

• Updated the gis_migrations test module to also include the RasterField into the tests.

[Tim Graham <timograham@…>]

In b769bbd4:

Fixed #23804 -- Added RasterField for PostGIS.

Thanks to Tim Graham and Claude Paroz for the reviews and patches.

[Tim Graham <timograham@…>]

In c0fff64:

Fixed #25011, Refs #23804 -- Added check for GDAL on RasterField initialization

[Tim Graham <timograham@…>]

In 9509268:

Refs #23804 -- Improved value validation in GDALRaster.geotransform setter.