Opened 11 years ago
Last modified 11 years ago
#21273 new Cleanup/optimization
Add read only support for Oracle XE to django.contrib.gis
Reported by: | vinhussey | Owned by: | nobody |
---|---|---|---|
Component: | GIS | Version: | dev |
Severity: | Normal | Keywords: | |
Cc: | vinhussey | Triage Stage: | Accepted |
Has patch: | no | Needs documentation: | yes |
Needs tests: | yes | Patch needs improvement: | no |
Easy pickings: | no | UI/UX: | no |
Description
django.contrib.gis.backends supports Oracle but does not support the spatial component of Oracle XE. Oracle XE has a more lenient licence than Oracle and may be useful for accessing legacy spatial data.
Oracle XE may be accessed using cx_Oracle, but spatial data is not converted into a usable form. The spatial data stored in oracle is documented (search for 'oracle sdo_geometry').
The following is a short description of the point attribute - I will expand with other types asap.
Example - Oracle spatial table named Floods with a spatial field geoloc:
>>> from import_data.models import * >>> floods = Floods.objects.all() >>> f = floods[99] >>> g = f.geoloc >>> g <cx_Oracle.OBJECT object at 0xa02a3e0> >>> dir(g) ['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'type'] >>> gtype = str(int(g.__getattribute__('SDO_GTYPE'))) >>> print gtype 2001 >>> gdims = gtype[0] >>> gdims '2' >>> gtopo = gtype[1] >>> gtopo '0' >>> ggeomtype = gtype[2:] >>> ggeomtype '01' >>> gsrid = g.__getattribute__('SDO_SRID') >>> gsrid 82086.0 >>> gpoint_x = g.__getattribute__('SDO_POINT').X >>> gpoint_y = g.__getattribute__('SDO_POINT').Y >>> gpoint_z = g.__getattribute__('SDO_POINT').Z >>> print gpoint_x, gpoint_y, gpoint_z 162913.389524 340748.357977 None >>> g_ewkt = "SRID:%s:POINT(%f %f)" % (str(int(g.__getattribute__('SDO_SRID'))), g.__getattribute__('SDO_POINT').X, g.__getattribute__('SDO_POINT').Y) >>> g_ewkt 'SRID:82086:POINT(162913.389524 340748.357977)'
From the Oracle docs, this is a 2 dimensional (gdims), null topology (gtopo), POINT geometry, srid = 82086 (Irish Grid), with x and y co-ordinates and null z coordinates.
Further information is stored in other attributes for more complex geometries - a very wide range is available. For a point, these may not necessarily be used.
>>> print g.__getattribute__('SDO_ELEM_INFO') None >>> print g.__getattribute__('SDO_ORDINATES') None
Attachments (7)
Change History (17)
comment:1 by , 11 years ago
Cc: | added |
---|---|
Needs documentation: | set |
Needs tests: | set |
Version: | 1.4 → master |
comment:2 by , 11 years ago
comment:3 by , 11 years ago
Polygon example
>>> from import_data.models import * >>> f = Floods.objects.get(flood_id=305) >>> g = f.geoloc >>> g <cx_Oracle.OBJECT object at 0xaee2b40> >>> dir(g) ['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'type'] >>> gtype = str(int(g.__getattribute__('SDO_GTYPE'))) >>> print gtype 2003 >>> gdims = gtype[0] >>> gdims '2' >>> gtopo = gtype[1] >>> gtopo '0' >>> ggeomtype = gtype[2:] >>> ggeomtype '03' >>> gsrid = g.__getattribute__('SDO_SRID') >>> gsrid 82086.0 >>> gpoint_x = g.__getattribute__('SDO_POINT').X >>> gpoint_y = g.__getattribute__('SDO_POINT').Y >>> gpoint_z = g.__getattribute__('SDO_POINT').Z >>> print gpoint_x, gpoint_y, gpoint_z 158775.736744 155058.070105 None >>> print g.__getattribute__('SDO_ELEM_INFO') [1.0, 1003.0, 1.0] >>> print g.__getattribute__('SDO_ORDINATES') [158851.770034052, 155240.090599927, 158814.553479071, 155238.39040849303, 158783.91910735102, 155236.430187781, 158758.984345447, 155234.259943421, 158738.883116615, 155231.929681045, 158722.73284741, 155229.469404029, 158709.667461087, 155226.92911800402, 158698.804384201, 155224.358828602, 158689.277540007, 155221.79854032502, 158680.352825359, 155219.268255426, 158671.914763359, 155216.767973906, 158663.96335400498, 155214.267692386, 158656.539839049, 155211.747408614, 158649.652466839, 155209.197121463, 158643.32598242702, 155206.57682643, 158637.58513086, 155203.866521262, 158632.438160491, 155201.056204833, 158627.918064717, 155198.11587376602, 158624.04958859, 155195.035526933, 158620.865725509, 155191.805163209, 158618.39946887302, 155188.424782593, 158616.692060433, 155184.874382835, 158615.768245238, 155181.153963933, 158615.652768339, 155177.253524761, 158616.395119835, 155173.17306532, 158618.003548075, 155168.882582232, 158620.436811311, 155164.382075495, 158623.645419442, 155159.631540607, 158627.55513731902, 155154.61097531402, 158632.12472319198, 155149.29037623902, 158637.28819026198, 155143.659742256, 158642.98780007902, 155137.689069986, 158649.174062542, 155131.33835492402, 158655.789239203, 155124.62759932398, 158662.800336662, 155117.58680656302, 158670.166113169, 155110.32598902902, 158677.853575324, 155102.895152351, 158685.846226427, 155095.384306664, 158694.094576378, 155087.863459851, 158702.573880129, 155080.42262204702, 158711.24289592798, 155073.131801134, 158720.07687872602, 155066.061004995, 158728.98509667302, 155059.20023250402, 158737.893314621, 155052.539482534, 158746.727297419, 155046.078755086, 158755.388064868, 155039.788046781, 158763.809630169, 155033.677358746, 158771.901261471, 155027.716687602, 158779.58872362602, 155021.896032223, 158786.80602983403, 155016.205391483, 158793.569676794, 155010.604760877, 158799.912657907, 155005.064137029, 158805.876214922, 154999.54351543201, 158811.493341239, 154994.002891583, 158816.805278607, 154988.41226210402, 158821.853268778, 154982.73162249, 158826.66205679902, 154976.930969363, 158831.289381122, 154970.970298219, 158835.77648349598, 154964.859610184, 158840.19759907, 154958.60890638301, 158844.626962994, 154952.228187944, 158849.11406536802, 154945.737457118, 158853.73314134, 154939.136713904, 158858.55017771202, 154932.45596168202, 158863.631161282, 154925.695200452, 158869.009085451, 154918.884433591, 158874.650956818, 154912.14367461202, 158880.474291883, 154905.612939281, 158886.42135219803, 154899.42224223702, 158892.417902613, 154893.70159811902, 158898.397956328, 154888.581021566, 158904.279029843, 154884.220530595, 158910.00338471, 154880.730137593, 158915.488537428, 154878.219854946, 158920.61076274802, 154876.67968153, 158925.22159037, 154876.05961171302, 158929.180798347, 154876.329642117, 158932.36466142803, 154877.41976486, 158934.624709315, 154879.30997768903, 158935.812471708, 154881.930272722, 158935.812471708, 154885.260647707, 158934.525729116, 154889.23109476102, 158932.11721093, 154893.79160825402, 158928.80962259902, 154898.86217917703, 158924.825669573, 154904.382800774, 158920.379808949, 154910.283465161, 158915.70299452601, 154916.484163331, 158911.009683404, 154922.92488852702, 158906.52258103, 154929.52563174098, 158902.414902754, 154936.23638734102, 158898.703145276, 154943.01715082402, 158895.379060246, 154949.857921063, 158892.39315756303, 154956.70869242802, 158889.745437229, 154963.56946492, 158887.39465749302, 154970.410235159, 158885.324321655, 154977.190998642, 158883.501436316, 154983.901754242, 158881.91775312502, 154990.532500833, 158880.581520433, 154997.11324179402, 158879.492738239, 155003.71398500798, 158878.667903244, 155010.39473723, 158878.123512147, 155017.195502965, 158877.867813299, 155024.19629122101, 158877.909055048, 155031.437106504, 158878.271982446, 155038.977955568, 158878.940098792, 155046.86884404602, 158879.80617553703, 155055.11977306302, 158880.754735782, 155063.74074374398, 158881.653805926, 155072.721754965, 158882.37966072198, 155082.07280785, 158882.80857492, 155091.793902401, 158882.81682327, 155101.885038616, 158882.280680523, 155112.34621649698, 158881.09291813, 155123.24744392498, 158879.204045991, 155134.878753557, 158876.597567406, 155147.600185932, 158873.24048897598, 155161.771781588, 158869.0998173, 155177.753581065, 158864.167304029, 155195.895623776, 158858.39345906302, 155216.557950259, 158851.770034052, 155240.090599927] >>> e_wkt = 'SRID=%s;POLYGON((' % (str(int(g.__getattribute__('SDO_SRID')))) >>> for i in range(len(ords)/2): ... e_wkt = e_wkt + '%f %f' %(ords[2*(i-1)+2], ords[2*(i-1)+3]) ... if i != len(ords)/2 - 1: ... e_wkt = e_wkt + ',' ... else: ... e_wkt = e_wkt + '))' ... >>> print e_wkt SRID=82086;POLYGON((158851.770034 155240.090600,158814.553479 155238.390408,158783.919107 155236.430188,158758.984345 155234.259943,158738.883117 155231.929681,158722.732847 155229.469404,158709.667461 155226.929118,158698.804384 155224.358829,158689.277540 155221.798540,158680.352825 155219.268255,158671.914763 155216.767974,158663.963354 155214.267692,158656.539839 155211.747409,158649.652467 155209.197121,158643.325982 155206.576826,158637.585131 155203.866521,158632.438160 155201.056205,158627.918065 155198.115874,158624.049589 155195.035527,158620.865726 155191.805163,158618.399469 155188.424783,158616.692060 155184.874383,158615.768245 155181.153964,158615.652768 155177.253525,158616.395120 155173.173065,158618.003548 155168.882582,158620.436811 155164.382075,158623.645419 155159.631541,158627.555137 155154.610975,158632.124723 155149.290376,158637.288190 155143.659742,158642.987800 155137.689070,158649.174063 155131.338355,158655.789239 155124.627599,158662.800337 155117.586807,158670.166113 155110.325989,158677.853575 155102.895152,158685.846226 155095.384307,158694.094576 155087.863460,158702.573880 155080.422622,158711.242896 155073.131801,158720.076879 155066.061005,158728.985097 155059.200233,158737.893315 155052.539483,158746.727297 155046.078755,158755.388065 155039.788047,158763.809630 155033.677359,158771.901261 155027.716688,158779.588724 155021.896032,158786.806030 155016.205391,158793.569677 155010.604761,158799.912658 155005.064137,158805.876215 154999.543515,158811.493341 154994.002892,158816.805279 154988.412262,158821.853269 154982.731622,158826.662057 154976.930969,158831.289381 154970.970298,158835.776483 154964.859610,158840.197599 154958.608906,158844.626963 154952.228188,158849.114065 154945.737457,158853.733141 154939.136714,158858.550178 154932.455962,158863.631161 154925.695200,158869.009085 154918.884434,158874.650957 154912.143675,158880.474292 154905.612939,158886.421352 154899.422242,158892.417903 154893.701598,158898.397956 154888.581022,158904.279030 154884.220531,158910.003385 154880.730138,158915.488537 154878.219855,158920.610763 154876.679682,158925.221590 154876.059612,158929.180798 154876.329642,158932.364661 154877.419765,158934.624709 154879.309978,158935.812472 154881.930273,158935.812472 154885.260648,158934.525729 154889.231095,158932.117211 154893.791608,158928.809623 154898.862179,158924.825670 154904.382801,158920.379809 154910.283465,158915.702995 154916.484163,158911.009683 154922.924889,158906.522581 154929.525632,158902.414903 154936.236387,158898.703145 154943.017151,158895.379060 154949.857921,158892.393158 154956.708692,158889.745437 154963.569465,158887.394657 154970.410235,158885.324322 154977.190999,158883.501436 154983.901754,158881.917753 154990.532501,158880.581520 154997.113242,158879.492738 155003.713985,158878.667903 155010.394737,158878.123512 155017.195503,158877.867813 155024.196291,158877.909055 155031.437107,158878.271982 155038.977956,158878.940099 155046.868844,158879.806176 155055.119773,158880.754736 155063.740744,158881.653806 155072.721755,158882.379661 155082.072808,158882.808575 155091.793902,158882.816823 155101.885039,158882.280681 155112.346216,158881.092918 155123.247444,158879.204046 155134.878754,158876.597567 155147.600186,158873.240489 155161.771782,158869.099817 155177.753581,158864.167304 155195.895624,158858.393459 155216.557950,158851.770034 155240.090600))
Note we are losing precision with the %f (6 decimal places, down from 12).
comment:4 by , 11 years ago
Example model
models.py
from django.db import models class Floods(models.Model): flood_id = models.IntegerField(primary_key=True) ... geoloc = models.TextField(blank=True)
Settings
This example used django core, not django.contrib.gis
settings.py
DATABASES = { 'remote': { 'ENGINE': 'django.db.backends.oracle', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'. 'NAME': 'XE', # Or path to database file if using sqlite3. ... },
comment:5 by , 11 years ago
Triage Stage: | Unreviewed → Accepted |
---|
by , 11 years ago
Attachment: | oracle_xe_sdo_geometry.py added |
---|
Initial class for handling SDO_GEOMETRY
comment:6 by , 11 years ago
Added an initial class that generates wkt for POINT (tested), LINESTRING (not tested yet), POLYGON (tested).
by , 11 years ago
Attachment: | oracle_xe_sdo_geometry.2.py added |
---|
Unpdated (incomplete) class for handling SDO_GEOMETRY
comment:7 by , 11 years ago
Attached 2 files:
- Sample data
- Updated code (version 4)
To test this file:
import sys from oracle_xe_geom.models import Sdo_Test from oracle_xe_geom.test_oracle_xe_class import sdo_geometry def main(argv=None): if argv is None: argv = sys.argv usage = "usage: %prog [options] arg" data = [] geoms = [] sd = Sdo_Test.objects.all() for s in sd: data.append(s) g = sdo_geometry(s.geom) geoms.append(g) print s.id, g.wkt print data if __name__ == "__main__": sys.exit(main()) main()
Result:
1 POINT(200000.000000000000 200000.000000000000) 2 LINESTRING(200000.000000000000 200000.000000000000,220000.000000000000 200000.000000000000,220000.000000000000 220000.000000000000) 0 [1.0, 1003.0, 1.0] 3 POLYGON((200000.000000000000 200000.000000000000,220000.000000000000 200000.000000000000,220000.000000000000 220000.000000000000,200000.000000000000 220000.000000000000,200000.000000000000 200000.000000000000)) 0 [1.0, 1003.0, 1.0] 1 [11.0, 2003.0, 1.0] 4 POLYGON((200000.000000000000 200000.000000000000,220000.000000000000 200000.000000000000,220000.000000000000 220000.000000000000,200000.000000000000 220000.000000000000,200000.000000000000 200000.000000000000)(202000.000000000000 202000.000000000000,208000.000000000000 208000.000000000000,208000.000000000000 202000.000000000000,202000.000000000000 202000.000000000000)) 5 MULTIPOINT(200000.000000000000 200000.000000000000,220000.000000000000 200000.000000000000,220000.000000000000 220000.000000000000,200000.000000000000 220000.000000000000) 0 [1.0, 2.0, 1.0] 1 [9.0, 2.0, 1.0] 6 MULTILINESTRING((200000.000000000000 200000.000000000000,220000.000000000000 200000.000000000000,220000.000000000000 220000.000000000000,200000.000000000000 220000.000000000000)(202000.000000000000 202000.000000000000,208000.000000000000 208000.000000000000,208000.000000000000 202000.000000000000)) 0 [1.0, 1003.0, 1.0] 7 MULTIPOLYGON(((200000.000000000000 200000.000000000000,220000.000000000000 200000.000000000000,220000.000000000000 220000.000000000000,200000.000000000000 220000.000000000000,200000.000000000000 200000.000000000000))) 0 [1.0, 1003.0, 1.0] 1 [11.0, 2.0, 1.0] 2 [17.0, 1.0, 1.0] [[1003], 2, 1] [['200000.000000000000 200000.000000000000,220000.000000000000 200000.000000000000,220000.000000000000 220000.000000000000,200000.000000000000 220000.000000000000,200000.000000000000 200000.000000000000'], '200000.000000000000 200000.000000000000,220000.000000000000 200000.000000000000,220000.000000000000 220000.000000000000', '210000.000000000000 210000.000000000000'] 8 GEOMETRY(POLYGON((200000.000000000000 200000.000000000000,220000.000000000000 200000.000000000000,220000.000000000000 220000.000000000000,200000.000000000000 220000.000000000000,200000.000000000000 200000.000000000000)),LINESTRING(200000.000000000000 200000.000000000000,220000.000000000000 200000.000000000000,220000.000000000000 220000.000000000000),POINT(210000.000000000000 210000.000000000000)) 9 POLYGON(([[200000.0, 200000.0], [200000.0, 200000.0], [200000.0, 200000.0], [200000.0, 200000.0], [200000.0, 200000.0]])) 10 POLYGON(([[200000.0, 200000.0], [220000.0, 200000.0], [220000.0, 220000.0]])) [<Sdo_Test: point>, <Sdo_Test: linestring>, <Sdo_Test: polygon>, <Sdo_Test: polygon with inner ring>, <Sdo_Test: multi point>, <Sdo_Test: multi line>, <Sdo_Test: multi polygon with 1 ring>, <Sdo_Test: geometry collection - polygon and point>, <Sdo_Test: rectangle - 2 points>, <Sdo_Test: circle - 3 points>]
follow-up: 9 comment:8 by , 11 years ago
Hi Vincent,
I am far from being a GeoDjango expert; in fact, I only started looking into it recently. I usually handle Oracle issues, but I don't use Oracle XE currently, and am not sure exactly how much spatial functionality is available on XE. I took a look at your latest code, and I have some comments.
First of all, the general scope of the work: As far as I could see, what you have added is a class for interpreting the SDO_GEOMETRY type; do you intend to also support other operations, such as geometric queries? Are they supported on XE? If not, the best integration into Django seems to be a custom field type.
Second, technical details:
The most glaring point in your code is the explicit use of __getattribute__
: This is a "magic method" in Python, it implements attribute access. You should be able to replace every occurrence of self.geometry.__getattribute__('NAME')
with self.geometry.NAME
, for every NAME
.
The second is -- instead of trying to generate the WKT yourself, it seems better to use GEOS objects -- WKT generation is already implemented there, as long as a ton of other functionality. The GEOS objects are explained here.
So -- the right thing to achieve, apparently, is a custom field, which inherits TextField
and interprets the object it gets as a GEOS object. I haven't looked into the GeoDjango (Oracle backend) code, but perhaps you can extract some code from there which will help you; I'd be a little surprised if SDO_GEOMETRY objects themselves are much different between Oracle XE and Standard.
Hope this helps,
Shai.
comment:9 by , 11 years ago
Hi Shai,
Thanks for the constructive comments. I have changed from the explicit use of __getattribute__
as suggested.
The reason for taking this approach is that Oracle XE does not provide a WKT function. See these comments in https://github.com/django/django/blob/master/django/contrib/gis/db/backends/oracle/operations.py
""" This module contains the spatial lookup types, and the `get_geo_where_clause` routine for Oracle Spatial. Please note that WKT support is broken on the XE version, and thus this backend will not work on such platforms. Specifically, XE lacks support for an internal JVM, and Java libraries are required to use the WKT constructors. """ ... class OracleOperations(DatabaseOperations, BaseSpatialOperations): compiler_module = "django.contrib.gis.db.backends.oracle.compiler" name = 'oracle' oracle = True valid_aggregates = {'Union', 'Extent'} Adapter = OracleSpatialAdapter Adaptor = Adapter # Backwards-compatibility alias. area = 'SDO_GEOM.SDO_AREA' gml = 'SDO_UTIL.TO_GMLGEOMETRY' centroid = 'SDO_GEOM.SDO_CENTROID' difference = 'SDO_GEOM.SDO_DIFFERENCE' distance = 'SDO_GEOM.SDO_DISTANCE' extent = 'SDO_AGGR_MBR' intersection = 'SDO_GEOM.SDO_INTERSECTION' length = 'SDO_GEOM.SDO_LENGTH' num_geom = 'SDO_UTIL.GETNUMELEM' num_points = 'SDO_UTIL.GETNUMVERTICES' perimeter = length point_on_surface = 'SDO_GEOM.SDO_POINTONSURFACE' reverse = 'SDO_UTIL.REVERSE_LINESTRING' sym_difference = 'SDO_GEOM.SDO_XOR' transform = 'SDO_CS.TRANSFORM' union = 'SDO_GEOM.SDO_UNION' unionagg = 'SDO_AGGR_UNION' # We want to get SDO Geometries as WKT because it is much easier to # instantiate GEOS proxies from WKT than SDO_GEOMETRY(...) strings. # However, this adversely affects performance (i.e., Java is called # to convert to WKT on every query). If someone wishes to write a # SDO_GEOMETRY(...) parser in Python, let me know =)
Once we have WKT, we can use geos, so thanks for that tip.
Vincent
Replying to shai:
Hi Vincent,
I am far from being a GeoDjango expert; in fact, I only started looking into it recently. I usually handle Oracle issues, but I don't use Oracle XE currently, and am not sure exactly how much spatial functionality is available on XE. I took a look at your latest code, and I have some comments.
First of all, the general scope of the work: As far as I could see, what you have added is a class for interpreting the SDO_GEOMETRY type; do you intend to also support other operations, such as geometric queries? Are they supported on XE? If not, the best integration into Django seems to be a custom field type.
Second, technical details:
The most glaring point in your code is the explicit use of
__getattribute__
: This is a "magic method" in Python, it implements attribute access. You should be able to replace every occurrence ofself.geometry.__getattribute__('NAME')
withself.geometry.NAME
, for everyNAME
.
The second is -- instead of trying to generate the WKT yourself, it seems better to use GEOS objects -- WKT generation is already implemented there, as long as a ton of other functionality. The GEOS objects are explained here.
So -- the right thing to achieve, apparently, is a custom field, which inherits
TextField
and interprets the object it gets as a GEOS object. I haven't looked into the GeoDjango (Oracle backend) code, but perhaps you can extract some code from there which will help you; I'd be a little surprised if SDO_GEOMETRY objects themselves are much different between Oracle XE and Standard.
Hope this helps,
Shai.
by , 11 years ago
Attachment: | oracle_xe_sdo_geometry_4.py added |
---|
Updated class for Oracle SDO_GEOMETRY
comment:10 by , 11 years ago
SDO_GEOMETRY itself has been same for a good while. I do have almost working version of read/write support for Oracle XE, as well as 3D support for Oracle geometries in general.
There exists few peculiarities between versions, specially between 10g and 11g and 3d geometries.
Currently I think that backend I've been working on is only failing is some coordinate transformations.
Correction to WKT in example
Note on SRID
Before version 10, Oracle used it's own SRIDs. The sample is Irish Grid 1965, this corresponds to EPSG:29903.