[Go-essp-tech] Aggregations and LAS in THREDDS catalogs

[Roland Schweitzer]

Stephan,

Sorry for the delay.  I've been away from the office.  I'll do my best here
to fill you in.

On Fri, Mar 9, 2012 at 3:14 AM, <stephen.pascoe at stfc.ac.uk> wrote:

> Hi,
>
> I'd like to understand better why aggregations are described as they are
> in ESGF THREDDS catalogs and what LAS expects from them.
>
> Modern datanodes have 2 levels of nested dataset elements describing
> aggregations.  E.g. [1].  In some cases there are multiple leaf datasets
> "subset 1" "subset 2" etc.  What's the reason for this?  Is it to split the
> field into manageable chunks for TDS or LAS?
>

I believe the reasoning is as follows.  Some data sets have large gaps. For
example, a run might write out data for a period of 100 years, skip 100
years and then write additional data for 100 years.  This makes for
challenging user interface issues in LAS since it expects data to be more
or less regular and contiguous.  So, I suggested that the publisher create
separate aggregations where there are these know large gaps in the data.
This will allow LAS to think of these things as two separate data sets in
the UI and thus guarantee that the UI is presenting only time ranges for
which data exists.


>
> There are LAS access elements for the dataset at 3 levels of the dataset
> hierarchy: top-level dataset, first-level aggregation and "subset"
> aggregation.  Are all three of these used by LAS?
>

In the end, no they are not.

Originally LAS would only load the single variable when that URL of the
"subset" was selected and would load all the variables in the data set when
the top-level URL was selected.  I subsequently changed how the LAS used in
the ESGF organized the data sets internally, so that the "top-level" url is
the only one that matters and it loads all of the variables in the data set
into the interface.

Luca and I were working on this recently and concluded that the publisher
should only insert the "top-level" LAS URLs.  We could in a subsequent
release create URLs at the "subset" level that would load all of the
variable into the interface and then select that variable as the active
variable.  The "first-level" URL seems less useful since we can't determine
which subset the user wants.  However, neither of these last two options
seems pressing or necessary since the search UI does not expose the links
to individual variables.

Roland

Thanks,
> Stephen.
>
> [1] -----------
>
>    <dataset
> name="cmip5.output1.INM.inmcm4.1pctCO2.mon.landIce.LImon.r1i1p1.sbl.20110323.aggregation"
> ID="cmip5.output1.INM.inmcm4.1pctCO2.mon.landIce.LImon.r1i1p1.sbl.20110323.aggregation"
> restrictAccess="esg-user">
>      <property name="aggregation_id"
> value="cmip5.output1.INM.inmcm4.1pctCO2.mon.landIce.LImon.r1i1p1.sbl.20110323.aggregation"/>
>      <!-- ... -->
>      <access
> urlPath="?catid=D9C519D5A310E197819B7197215FD574_ns_cmip5.output1.INM.inmcm4.1pctCO2.mon.landIce.LImon.r1i1p1.sbl.20110323.aggregation"
> serviceName="LASatPCMDI9" dataFormat="NetCDF"/>
>      <dataset
> name="cmip5.output1.INM.inmcm4.1pctCO2.mon.landIce.LImon.r1i1p1.sbl.20110323.aggregation
> - Subset 1"
> ID="cmip5.output1.INM.inmcm4.1pctCO2.mon.landIce.LImon.r1i1p1.sbl.20110323.aggregation.1"
> urlPath="cmip5.output1.INM.inmcm4.1pctCO2.mon.landIce.LImon.r1i1p1.sbl.20110323.aggregation.1"
> restrictAccess="esg-user">
>        <serviceName>gridded</serviceName>
>        <property name="aggregation_id"
> value="cmip5.output1.INM.inmcm4.1pctCO2.mon.landIce.LImon.r1i1p1.sbl.20110323.aggregation.1"/>
>        <property name="time_delta" value="1 month"/>
>        <property name="calendar" value="365_day"/>
>        <property name="start" value="2090-1-1 0:0:0.0"/>
>        <property name="time_length" value="1680"/>
>        <access
> urlPath="?catid=D9C519D5A310E197819B7197215FD574_ns_cmip5.output1.INM.inmcm4.1pctCO2.mon.landIce.LImon.r1i1p1.sbl.20110323.aggregation.1"
> serviceName="LASatPCMDI9" dataFormat="NetCDF"/>
>      </dataset>
>    </dataset>--
> Scanned by iCritical.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20120319/3dd795fc/attachment.html