[Go-essp-tech] Aggregations and LAS in THREDDS catalogs

Cinquini, Luca (3880) Luca.Cinquini at jpl.nasa.gov
Tue Mar 20 06:46:28 MDT 2012 

On Mar 20, 2012, at 6:29 AM, <stephen.pascoe at stfc.ac.uk<mailto:stephen.pascoe at stfc.ac.uk>> wrote:

No, it's on my perpetual TODO list.  I'd be interested to know whether the WMS is configured right in the default P2P node configuration.

No, it's not - but we should....
thanks, Luca


Stephen.

---
Stephen Pascoe  +44 (0)1235 445980
Centre of Environmental Data Archival
STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK

From: Estanislao Gonzalez [mailto:gonzalez at dkrz.de]
Sent: 20 March 2012 12:27
To: Pascoe, Stephen (STFC,RAL,RALSP)
Cc: roland.schweitzer at noaa.gov<mailto:roland.schweitzer at noaa.gov>; plieger at knmi.nl<mailto:plieger at knmi.nl>; drach1 at llnl.gov<mailto:drach1 at llnl.gov>; go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>
Subject: Re: [Go-essp-tech] Aggregations and LAS in THREDDS catalogs

Hi Stephen,

I've never tried it but the TDS offers apparently WMS services just as OpenDAP AFAICT (that info is in the web.xml)
Have you tried it?

Thanks,
Estani

Am 20.03.2012 12:47, schrieb stephen.pascoe at stfc.ac.uk:<mailto:stephen.pascoe at stfc.ac.uk:>
Hi Roland,

Thanks, that's very helpful.  Separating aggregations into contiguous portions is completely sensible.  I'll roll this info into the THREDDS spec.

Incidentally, Marten and I recently prototyped bringing a CMIP5 dataset into KNMI's WMS visualisation portal that we are using in our IS-ENES project.  This picks up the OPeNDAP aggregation endpoints and res-serves them as WMS.  We managed to get this working through the ESGF security layer.

The repercussions of this are:
 1) down the road there will be visualisation options that don't rely on explicit declarations in THREDDS ( the visualisation will read the THREDDS XML and know what to do)
2) Marten needs to know how the aggregations are laid-out in THREDDS and what the subsets are for.

Cheers,
Stephen.

---
Stephen Pascoe  +44 (0)1235 445980
Centre of Environmental Data Archival
STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK

From: Roland Schweitzer [mailto:roland.schweitzer at noaa.gov]
Sent: 19 March 2012 14:40
To: Pascoe, Stephen (STFC,RAL,RALSP)
Cc: go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>; drach1 at llnl.gov<mailto:drach1 at llnl.gov>; plieger at knmi.nl<mailto:plieger at knmi.nl>
Subject: Re: Aggregations and LAS in THREDDS catalogs

Stephan,

Sorry for the delay.  I've been away from the office.  I'll do my best here to fill you in.

On Fri, Mar 9, 2012 at 3:14 AM, <stephen.pascoe at stfc.ac.uk<mailto:stephen.pascoe at stfc.ac.uk>> wrote:
Hi,

I'd like to understand better why aggregations are described as they are in ESGF THREDDS catalogs and what LAS expects from them.

Modern datanodes have 2 levels of nested dataset elements describing aggregations.  E.g. [1].  In some cases there are multiple leaf datasets "subset 1" "subset 2" etc.  What's the reason for this?  Is it to split the field into manageable chunks for TDS or LAS?

I believe the reasoning is as follows.  Some data sets have large gaps. For example, a run might write out data for a period of 100 years, skip 100 years and then write additional data for 100 years.  This makes for challenging user interface issues in LAS since it expects data to be more or less regular and contiguous.  So, I suggested that the publisher create separate aggregations where there are these know large gaps in the data. This will allow LAS to think of these things as two separate data sets in the UI and thus guarantee that the UI is presenting only time ranges for which data exists.


There are LAS access elements for the dataset at 3 levels of the dataset hierarchy: top-level dataset, first-level aggregation and "subset" aggregation.  Are all three of these used by LAS?

In the end, no they are not.

Originally LAS would only load the single variable when that URL of the "subset" was selected and would load all the variables in the data set when the top-level URL was selected.  I subsequently changed how the LAS used in the ESGF organized the data sets internally, so that the "top-level" url is the only one that matters and it loads all of the variables in the data set into the interface.

Luca and I were working on this recently and concluded that the publisher should only insert the "top-level" LAS URLs.  We could in a subsequent release create URLs at the "subset" level that would load all of the variable into the interface and then select that variable as the active variable.  The "first-level" URL seems less useful since we can't determine which subset the user wants.  However, neither of these last two options seems pressing or necessary since the search UI does not expose the links to individual variables.

Roland

Thanks,
Stephen.

[1] -----------

   <dataset name="cmip5.output1.INM.inmcm4.1pctCO2.mon.landIce.LImon.r1i1p1.sbl.20110323.aggregation" ID="cmip5.output1.INM.inmcm4.1pctCO2.mon.landIce.LImon.r1i1p1.sbl.20110323.aggregation" restrictAccess="esg-user">
     <property name="aggregation_id" value="cmip5.output1.INM.inmcm4.1pctCO2.mon.landIce.LImon.r1i1p1.sbl.20110323.aggregation"/>
     <!-- ... -->
     <access urlPath="?catid=D9C519D5A310E197819B7197215FD574_ns_cmip5.output1.INM.inmcm4.1pctCO2.mon.landIce.LImon.r1i1p1.sbl.20110323.aggregation" serviceName="LASatPCMDI9" dataFormat="NetCDF"/>
     <dataset name="cmip5.output1.INM.inmcm4.1pctCO2.mon.landIce.LImon.r1i1p1.sbl.20110323.aggregation - Subset 1" ID="cmip5.output1.INM.inmcm4.1pctCO2.mon.landIce.LImon.r1i1p1.sbl.20110323.aggregation.1" urlPath="cmip5.output1.INM.inmcm4.1pctCO2.mon.landIce.LImon.r1i1p1.sbl.20110323.aggregation.1" restrictAccess="esg-user">
       <serviceName>gridded</serviceName>
       <property name="aggregation_id" value="cmip5.output1.INM.inmcm4.1pctCO2.mon.landIce.LImon.r1i1p1.sbl.20110323.aggregation.1"/>
       <property name="time_delta" value="1 month"/>
       <property name="calendar" value="365_day"/>
       <property name="start" value="2090-1-1 0:0:0.0"/>
       <property name="time_length" value="1680"/>
       <access urlPath="?catid=D9C519D5A310E197819B7197215FD574_ns_cmip5.output1.INM.inmcm4.1pctCO2.mon.landIce.LImon.r1i1p1.sbl.20110323.aggregation.1" serviceName="LASatPCMDI9" dataFormat="NetCDF"/>
     </dataset>
   </dataset>--
Scanned by iCritical.



--
Scanned by iCritical.





_______________________________________________

GO-ESSP-TECH mailing list

GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>

http://mailman.ucar.edu/mailman/listinfo/go-essp-tech




--

Estanislao Gonzalez



Max-Planck-Institut für Meteorologie (MPI-M)

Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre

Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany



Phone:   +49 (40) 46 00 94-126

E-Mail:  gonzalez at dkrz.de<mailto:gonzalez at dkrz.de>


--
Scanned by iCritical.

_______________________________________________
GO-ESSP-TECH mailing list
GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20120320/812c88a3/attachment-0001.html

More information about the GO-ESSP-TECH mailing list