[Go-essp-tech] Interfacing to esgcet database (was Minutes: 1/6/10 Output Variable Telecon)

Mon Jan 11 11:47:04 MST 2010

Hi Stephen,

On Jan 11, 2010, at 5:35 AM, <stephen.pascoe at stfc.ac.uk> wrote:

>
> Hi Bob,
>
>> However there is no guarantee that the schema will not change in such
> a way that your extra
>> ORM classes break. It might be worth considering specification of an
> API that would be stable > across future schema migrations ...
>
> If you are willing to do (or help us do it) this that would be very
> helpful.  I am quite relaxed about tracking updates to your source as
> they arise but I am more familiar with SQLAlchemy than Hans.
>
> I have put up a quick recipe for accessing the database through
> SQLALchemy at
> https://*is-enes-wiki.dkrz.de/farm/SA2/JRA4%20Collaboration/ 
> ESGDataNode/S
> QLAlchemyHowTo.  I need to complete our datanode upgrade before I go
> into more detail but if anyone from IS-ENES wants to add detail just
> email me or (if you are part of IS-ENES) edit the wiki directly.
>
> An informal API could evolve there or on the NCAR wiki.  A place to
> start might be the Dataset class.  It looks like this class is  
> different
> to the other ORM classes in that as well as ORM-instrumented  
> properties
> it has a set of methods that take a session argument.  I presume  
> you did
> this so that code can use Dataset instances without being aware of
> SQLAlchemy session semantics.

Yes, the getXXX methods on Dataset have a Session argument to allow  
the getter methods to add the instance to a session if necessary.

Bob

>
> S.
>
>
> ---
> Stephen Pascoe  +44 (0)1235 445980
> British Atmospheric Data Centre
> Rutherford Appleton Laboratory
>
> -----Original Message-----
> From: Bob Drach [mailto:drach at llnl.gov]
> Sent: 08 January 2010 21:09
> To: Pascoe, Stephen (STFC,RAL,SSTD)
> Cc: hans.ramthun at zmaw.de; momipsl at ipsl.jussieu.fr;
> metafor at lists.enes.org; go-essp-tech at ucar.edu
> Subject: Re: [Go-essp-tech] Interfacing to esgcet database (was  
> Minutes:
> 1/6/10 Output Variable Telecon)
>
> Hi Stephen,
>
> I agree that top-down coordination probably isn't necessary at this
> point, especially given that the database access is read-only
> (correct?). However there is no guarantee that the schema will not
> change in such a way that your extra ORM classes break. It might be
> worth considering specification of an API that would be stable across
> future schema migrations, and could be shared among  METAFOR
> applications.
>
> Bob
>
> On Jan 8, 2010, at 1:24 AM, <stephen.pascoe at stfc.ac.uk> wrote:
>
>>
>> Is either Hans' or Mark's work intended to be part of the METAFOR
>> tools or are they primarily internal development for DKRZ & IPSL?  I
>> am assuming we will be deploying METAFOR tools to generate CIM from
>> the NetCDF.
>>
>> For our part we are designing an interface to the esgcet database as
>> part of our CMIP5 data ingest system.  This is a different use  
>> case to
>
>> producing CIM.  Our plan is to use the esgcet.model package's
>> SQLAlchemy interface and add extra ORM classes that reference our own
>> schema, thus using SQLALchemy as a schema integration tool.
>>
>> At this stage top-down coordination probably isn't benificial but I
>> suggest we should keep on our separate tracks but keep our code in  
>> the
>
>> open so that any obvious overlaps are seen early.  I will post our  
>> SVN
>
>> URL once we start coding.
>>
>> Cheers,
>> Stephen.
>>
>> ---
>> Stephen Pascoe  +44 (0)1235 445980
>> British Atmospheric Data Centre
>> Rutherford Appleton Laboratory
>>
>> -----Original Message-----
>> From: metafor-bounces at lists.enes.org
>> [mailto:metafor-bounces at lists.enes.org] On Behalf Of Hans Ramthun
>> Sent: 07 January 2010 15:09
>> To: Mark Morgan
>> Cc: go-essp-tech at ucar.edu; curator at ucar.edu; Sylvia Murphy; Metafor
>> List; Karl Taylor
>> Subject: Re: [metafor] [Go-essp-tech] Minutes: 1/6/10 Output Variable
>> Telecon
>>
>> Hallo Mark,
>>
>> Yes of course there are several points of collaboration. I'm happy  
>> not
>
>> to dig alone through the mountain...
>>
>> I published my very first steps in the Metafor svn:
>> http://**metaforclimate.eu/trac/browser/cimTOOLS/branches/ESG-DB-
>> access-TE
>> ST
>> The first test was with the python library 'psycopg2'.
>>
>> Cheers
>> Hans
>>
>>
>> Mark Morgan wrote:
>>> Hans
>>>
>>> Here at IPSL we are in the process of beginning to do exactly the
>>> same
>>> thing: coding a python library to interface with the ESG publisher
>>> postgres database in order to act as a prelimnary step in the  
>>> Metafor
>
>>> CIM publishing process.
>>>
>>> Perhaps this can be a point of collaboration either in the
>>> development
>>
>>> or testing (or both)?
>>>
>>> Regards
>>>
>>> Mark
>>>
>>> Hans Ramthun wrote:
>>>> Hallo,
>>>>
>>>> Bryan Lawrence wrote:
>>>>
>>>>> ...
>>>>>
>>>>>> Right now, we would have to parse the thredds catalog as you say
>>>>>> to
>>
>>>>>> get the list of variables.  No code exists to convert the data
>>>>>> collection names into a list of variable names.  Code does exist
>>>>>> to
>>
>>>>>> create the list of data collections.
>>>>>>
>>>>> I think we all need a bit of code for that. We should write it
>>>>> standalone so we can all use the logic if not the code ...
>>>>> hopefully this is on the agenda for Hans and/or Stephen.
>>>>>
>>>> On my agenda is to access the ESG publisher postgres database with
>>>> SQLAlchemy and to retrieve the appropriate data to create  
>>>> CIMRecords
>
>>>> (dataObject). This should led to a stand alone python program. I
>>>> think Stephen wants to go the same way.
>>>>
>>>>
>>>>> Cheers
>>>>> Bryan
>>>>>
>>>> Cheers
>>>> Hans
>>>>
>>>>
>>>
>>
>>
>> --
>> Hans Ramthun
>> Tel.: +49.40.460.094.112
>> Email: hans.ramthun at zmaw.de
>>
>> DKRZ-Hamburg / Max-Planck-Institut fuer Meteorologie
>> Abteilung Modelle und Daten     http://**www.**mad.zmaw.de/
>> Bundesstr. 45a
>> D-20146 Hamburg                 Germany
>>
>> _______________________________________________
>> metafor mailing list
>> metafor at lists.enes.org
>> https://**lists.enes.org/mailman/listinfo/metafor
>> --
>> Scanned by iCritical.
>> _______________________________________________
>> GO-ESSP-TECH mailing list
>> GO-ESSP-TECH at ucar.edu
>> http://**mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>
>
> -- 
> Scanned by iCritical.
>