[Go-essp-tech] Interfacing to esgcet database (was Minutes: 1/6/10 Output Variable Telecon)

Mon Jan 11 06:35:26 MST 2010

Hi Bob,

> However there is no guarantee that the schema will not change in such
a way that your extra 
> ORM classes break. It might be worth considering specification of an
API that would be stable > across future schema migrations ...

If you are willing to do (or help us do it) this that would be very
helpful.  I am quite relaxed about tracking updates to your source as
they arise but I am more familiar with SQLAlchemy than Hans.

I have put up a quick recipe for accessing the database through
SQLALchemy at
https://is-enes-wiki.dkrz.de/farm/SA2/JRA4%20Collaboration/ESGDataNode/S
QLAlchemyHowTo.  I need to complete our datanode upgrade before I go
into more detail but if anyone from IS-ENES wants to add detail just
email me or (if you are part of IS-ENES) edit the wiki directly.

An informal API could evolve there or on the NCAR wiki.  A place to
start might be the Dataset class.  It looks like this class is different
to the other ORM classes in that as well as ORM-instrumented properties
it has a set of methods that take a session argument.  I presume you did
this so that code can use Dataset instances without being aware of
SQLAlchemy session semantics.

S.

---
Stephen Pascoe  +44 (0)1235 445980
British Atmospheric Data Centre
Rutherford Appleton Laboratory

-----Original Message-----
From: Bob Drach [mailto:drach at llnl.gov] 
Sent: 08 January 2010 21:09
To: Pascoe, Stephen (STFC,RAL,SSTD)
Cc: hans.ramthun at zmaw.de; momipsl at ipsl.jussieu.fr;
metafor at lists.enes.org; go-essp-tech at ucar.edu
Subject: Re: [Go-essp-tech] Interfacing to esgcet database (was Minutes:
1/6/10 Output Variable Telecon)

Hi Stephen,

I agree that top-down coordination probably isn't necessary at this
point, especially given that the database access is read-only
(correct?). However there is no guarantee that the schema will not
change in such a way that your extra ORM classes break. It might be
worth considering specification of an API that would be stable across
future schema migrations, and could be shared among  METAFOR
applications.

Bob

On Jan 8, 2010, at 1:24 AM, <stephen.pascoe at stfc.ac.uk> wrote:

>
> Is either Hans' or Mark's work intended to be part of the METAFOR 
> tools or are they primarily internal development for DKRZ & IPSL?  I 
> am assuming we will be deploying METAFOR tools to generate CIM from 
> the NetCDF.
>
> For our part we are designing an interface to the esgcet database as 
> part of our CMIP5 data ingest system.  This is a different use case to

> producing CIM.  Our plan is to use the esgcet.model package's 
> SQLAlchemy interface and add extra ORM classes that reference our own 
> schema, thus using SQLALchemy as a schema integration tool.
>
> At this stage top-down coordination probably isn't benificial but I 
> suggest we should keep on our separate tracks but keep our code in the

> open so that any obvious overlaps are seen early.  I will post our SVN

> URL once we start coding.
>
> Cheers,
> Stephen.
>
> ---
> Stephen Pascoe  +44 (0)1235 445980
> British Atmospheric Data Centre
> Rutherford Appleton Laboratory
>
> -----Original Message-----
> From: metafor-bounces at lists.enes.org
> [mailto:metafor-bounces at lists.enes.org] On Behalf Of Hans Ramthun
> Sent: 07 January 2010 15:09
> To: Mark Morgan
> Cc: go-essp-tech at ucar.edu; curator at ucar.edu; Sylvia Murphy; Metafor 
> List; Karl Taylor
> Subject: Re: [metafor] [Go-essp-tech] Minutes: 1/6/10 Output Variable 
> Telecon
>
> Hallo Mark,
>
> Yes of course there are several points of collaboration. I'm happy not

> to dig alone through the mountain...
>
> I published my very first steps in the Metafor svn:
> http://*metaforclimate.eu/trac/browser/cimTOOLS/branches/ESG-DB-
> access-TE
> ST
> The first test was with the python library 'psycopg2'.
>
> Cheers
> Hans
>
>
> Mark Morgan wrote:
>> Hans
>>
>> Here at IPSL we are in the process of beginning to do exactly the 
>> same
>> thing: coding a python library to interface with the ESG publisher 
>> postgres database in order to act as a prelimnary step in the Metafor

>> CIM publishing process.
>>
>> Perhaps this can be a point of collaboration either in the 
>> development
>
>> or testing (or both)?
>>
>> Regards
>>
>> Mark
>>
>> Hans Ramthun wrote:
>>> Hallo,
>>>
>>> Bryan Lawrence wrote:
>>>
>>>> ...
>>>>
>>>>> Right now, we would have to parse the thredds catalog as you say 
>>>>> to
>
>>>>> get the list of variables.  No code exists to convert the data 
>>>>> collection names into a list of variable names.  Code does exist 
>>>>> to
>
>>>>> create the list of data collections.
>>>>>
>>>> I think we all need a bit of code for that. We should write it 
>>>> standalone so we can all use the logic if not the code ...
>>>> hopefully this is on the agenda for Hans and/or Stephen.
>>>>
>>> On my agenda is to access the ESG publisher postgres database with 
>>> SQLAlchemy and to retrieve the appropriate data to create CIMRecords

>>> (dataObject). This should led to a stand alone python program. I 
>>> think Stephen wants to go the same way.
>>>
>>>
>>>> Cheers
>>>> Bryan
>>>>
>>> Cheers
>>> Hans
>>>
>>>
>>
>
>
> --
> Hans Ramthun
> Tel.: +49.40.460.094.112
> Email: hans.ramthun at zmaw.de
>
> DKRZ-Hamburg / Max-Planck-Institut fuer Meteorologie
> Abteilung Modelle und Daten     http://*www.*mad.zmaw.de/
> Bundesstr. 45a
> D-20146 Hamburg                 Germany
>
> _______________________________________________
> metafor mailing list
> metafor at lists.enes.org
> https://*lists.enes.org/mailman/listinfo/metafor
> --
> Scanned by iCritical.
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://*mailman.ucar.edu/mailman/listinfo/go-essp-tech
>

-- 
Scanned by iCritical.