[GO-ESSP] gridded data management systems

[Jon Blower]

Hi all,

As some of you may know, we at the Reading e-Science Centre have been
investigating some new ways to store and manage data from models of the
oceans and atmosphere.  We have been looking at storing data in databases,
rather than standard flat-file systems, and have over the last few months
been evaluating IBM's Informix database with Barrodale Computing Services'
Grid DataBlade plug-in (see http://www.resc.rdg.ac.uk/projects.php for more
details).  Eventually this might form the back-end to our own data portal
page (http://www.nerc-essc.ac.uk/godiva).

We have found good and bad points about this system and are now wondering
how to take things forward.  I have been considering the feasibility of
writing (essentially from scratch) an intelligent storage/management
application for gridded geospatial data.  The key features of this system
would include:

1) Data would be stored in a single format but can be extracted in a variety
of formats
2) Data could be sliced and subsetted in all possible ways (e.g. extraction
of 1-D timeseries, 2-D areas, 3-D volumes/animations, 4-D data blocks) and
extracted at different spatial and temporal resolutions
3) Data could be stored on the original grid (including rotated grids) but
extracted on the grid of the user's choice
4) The necessary projection and interpolation would happen on the fly
5) The system would allow complex queries to be made (e.g. "Give me all the
times and locations at which the sea surface temperature was greater than 20
degC in the North Atlantic in June 2003")

The systems we have looked at so far get us part, but not all, of the way
there.  Furthermore, the system currently under evaluation (Informix/Grid
DataBlade) is closed-source, commercial software so we can't modify it
ourselves.  However, such database-based systems have some key advantages
over standard flat files, notably intelligent tiling and caching, giving
very fast retrieval of data.

I was wondering whether this community would welcome an effort to create an
open-source data management/storage system for geospatial data, perhaps as a
plug-in to an open-source DBMS such as PostgreSQL.  I haven't found an
existing project that answers our requirements, but please let me know if
you know of anything (some packages seem to deal with geospatial data, but
are not designed for _gridded_ data).  It seems that this could be of
benefit to a to the GO-ESSP community, considering that any Earth System
Portal must be backed by some kind of data store! ;-)

This has been rather a long post, sorry!  Any suggestions or feedback would
be very much appreciated.

Best wishes,
Jon

--------------------------------------------------------------
Dr Jon Blower              Tel: +44 118 378 5213 (direct line)
Technical Director         Tel: +44 118 378 8741 (ESSC)
Reading e-Science Centre   Fax: +44 118 378 6413
ESSC                       Email: jdb at mail.nerc-essc.ac.uk
University of Reading
3 Earley Gate
Reading RG6 6AL, UK
--------------------------------------------------------------