<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
<html>
Hi Jon,
<p>When you refer to "standard flat files" are you including formats like
netCDF and HDF5 under that title? This is often a source of terminology
confusion as "flat files" sometimes refers to "anything but a database".
Others regard n-dimensional, multi-variate data standards like netCDF and
HDF to be alternatives to "flat" IEEE files.
<p>The question that you pose is essentially to weigh the pros and cons
of managing your data with a commercial database that has been enhanced
to handle grids, or to handle your data with netCDF (which in the next
version will merge with HDF5 to handle compression, tiles, etc.) and the
free netCDF utilities. (Presumably from-scratch development with
IEEE binary files is not the way to go.) You mentioned some down-sides
to the commercial software route (cost, "proprietariness" of software,
dependence on a single supplier,...). Are the advantages of
the database approach sufficient to outweigh these costs? You
have also not mentioned network access to the data. Is it a requirement
is for the data to be OPeNDAP accessible? Or alternatively, is access
from enterprise GIS systems at the center of your bullseye?
<ul>
<li>
Items 1-2 are trivial for either system. Comparative performance
... do you have any data? The Barrodale product is new and one-of-a-kind.
It would be interesting to see some benchmarks comparing it to netCDF and
HDF5.</li>
<li>
Items 3-4 can be handled with the new FDS (Ferret Data Server) and probably
the GDS server, as well. Custom code may be required depending upon
the list of projections that is desired, but these are open environments,
where this can be added. Item 3-4 capabilities are also available
and presumably well supported if your database is embedded in an enterprise
GIS framework.</li>
<li>
Item 5 is probably better handled in a database environment, though it
can also be handled (with some effort -- in various ways) in a Web service
environment based on OPeNDAP.</li>
</ul>
Just bouncing around the ideas. This community will be interested
to hear what further you learn.
<p> - steve
<p>====================================
<p>Jon Blower wrote:
<blockquote TYPE=CITE>Hi all,
<p>As some of you may know, we at the Reading e-Science Centre have been
<br>investigating some new ways to store and manage data from models of
the
<br>oceans and atmosphere. We have been looking at storing data in
databases,
<br>rather than standard flat-file systems, and have over the last few
months
<br>been evaluating IBM's Informix database with Barrodale Computing Services'
<br>Grid DataBlade plug-in (see <a href="http://www.resc.rdg.ac.uk/projects.php">http://www.resc.rdg.ac.uk/projects.php</a>
for more
<br>details). Eventually this might form the back-end to our own
data portal
<br>page (<a href="http://www.nerc-essc.ac.uk/godiva">http://www.nerc-essc.ac.uk/godiva</a>).
<p>We have found good and bad points about this system and are now wondering
<br>how to take things forward. I have been considering the feasibility
of
<br>writing (essentially from scratch) an intelligent storage/management
<br>application for gridded geospatial data. The key features of
this system
<br>would include:
<p>1) Data would be stored in a single format but can be extracted in a
variety
<br>of formats
<br>2) Data could be sliced and subsetted in all possible ways (e.g. extraction
<br>of 1-D timeseries, 2-D areas, 3-D volumes/animations, 4-D data blocks)
and
<br>extracted at different spatial and temporal resolutions
<br>3) Data could be stored on the original grid (including rotated grids)
but
<br>extracted on the grid of the user's choice
<br>4) The necessary projection and interpolation would happen on the fly
<br>5) The system would allow complex queries to be made (e.g. "Give me
all the
<br>times and locations at which the sea surface temperature was greater
than 20
<br>degC in the North Atlantic in June 2003")
<p>The systems we have looked at so far get us part, but not all, of the
way
<br>there. Furthermore, the system currently under evaluation (Informix/Grid
<br>DataBlade) is closed-source, commercial software so we can't modify
it
<br>ourselves. However, such database-based systems have some key
advantages
<br>over standard flat files, notably intelligent tiling and caching, giving
<br>very fast retrieval of data.
<p>I was wondering whether this community would welcome an effort to create
an
<br>open-source data management/storage system for geospatial data, perhaps
as a
<br>plug-in to an open-source DBMS such as PostgreSQL. I haven't
found an
<br>existing project that answers our requirements, but please let me know
if
<br>you know of anything (some packages seem to deal with geospatial data,
but
<br>are not designed for _gridded_ data). It seems that this could
be of
<br>benefit to a to the GO-ESSP community, considering that any Earth System
<br>Portal must be backed by some kind of data store! ;-)
<p>This has been rather a long post, sorry! Any suggestions or feedback
would
<br>be very much appreciated.
<p>Best wishes,
<br>Jon
<p>--------------------------------------------------------------
<br>Dr Jon Blower
Tel: +44 118 378 5213 (direct line)
<br>Technical Director
Tel: +44 118 378 8741 (ESSC)
<br>Reading e-Science Centre Fax: +44 118 378 6413
<br>ESSC
Email: jdb@mail.nerc-essc.ac.uk
<br>University of Reading
<br>3 Earley Gate
<br>Reading RG6 6AL, UK
<br>--------------------------------------------------------------
<p>_______________________________________________
<br>GO-ESSP mailing list
<br>GO-ESSP@ucar.edu
<br><a href="http://mailman.ucar.edu/mailman/listinfo/go-essp">http://mailman.ucar.edu/mailman/listinfo/go-essp</a></blockquote>
<p>--
<p>Steve Hankin, NOAA/PMEL -- Steven.C.Hankin@noaa.gov
<br>7600 Sand Point Way NE, Seattle, WA 98115-0070
<br>ph. (206) 526-6080, FAX (206) 526-6744
<br> </html>