[Go-essp-tech] DRS syntax and TDS identifiers

Thu Sep 2 05:11:44 MDT 2010

Hello Phil,

I think we can safely isolate the discussion below from CMOR2.

Firstly, CMOR2 is clearly not in a position to consider version control.
Version control has deliberately been postponed to a later stage. If
CMOR2 is used to analyse output for the same experiment twice it will
produce the same file names and directory structure. In order to support
version control, the files will have to be moved away from the directory
structure created by CMOR2. The directory structure created by CMOR2
does, however, use the DRS vocabulary, so the rearrangement to support
version control is a reasonably simple post-processing step. Jamie might
want to comment on this, but I believe that in the UKMO-BADC publishing
stream, this rearrangement will be done at BADC. The discussion below is
really about how this re-arrangement will be done, and how it will be
represented in the THREDDS catalogue, so it is a concern to groups
running the ESG data nodes and seeking (as Martina is) to develop
software which will interact with the data nodes. The data node
configuration has not yet been finalised.

What CMOR2 needs from the DRS are the controlled vocabularies, and I
can't see any changes happening there (except that there are rumours of
additional experiments by some groups).

Regards,
Martin

> -----Original Message-----
> From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-
> bounces at ucar.edu] On Behalf Of Bentley, Philip
> Sent: 02 September 2010 11:19
> To: Martina Stockhause; GO-ESSP
> Cc: Elkington, Mark; Kettleborough,Jamie
> Subject: Re: [Go-essp-tech] DRS syntax and TDS identifiers
> 
> Hi folks,
> 
> Just for the record we (here at MOHC) have this week started producing
> our first batch of CMOR-netCDF files for the CMIP5 archive using the
> latest (and, we understand, final) version of the CMOR-2 library.
> (Charles Doutriaux at PCMDI has indicated recently that, with the
> exception of essential bug fixes, the CMOR-2 library is in effect now
> frozen.)
> 
> It's not entirely clear to me if the changes to the DRS
> syntax/structure
> being suggested at this *very* late stage in the game will impact on
i)
> the validity of the CMIP5 netCDF data files that we have just
produced,
> of which there are many GB, or ii) if this will require corresponding
> updates to the CMOR-2 library.
> 
> I *think* the answer is 'no' on both counts, i.e. our data will still
> be
> valid and the CMOR-2 library will not need to change. But it would
> definitely be reassuring to us if someone on the mailing list could
> corroborate this view!
> 
> Thanks,
> Phil
> 
> > -----Original Message-----
> > From: go-essp-tech-bounces at ucar.edu
> > [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of Martina
> Stockhause
> > Sent: 02 September 2010 09:54
> > To: Bob Drach; GO-ESSP
> > Subject: [Go-essp-tech] DRS syntax and TDS identifiers
> >
> > Hi Bob, dear all,
> >
> > for the QC a defined and stable DRS syntax and a clear
> > mapping of other IDs (TDS and metafor) to the DRS are required.
> >
> > The granularity of the quality checks is the atomic dataset.
> > QC level 3
> > (STD-DOI) is assigned on the experiment level but includes
> > references to every netcdf file (or chunk) of this
> > experiment. For a correct identification of the files, which
> > belong to the DOI experiment, the DRS definition has to
> > include DRS levels: 'experiment, atomic dataset, netcdf
> > file'. We extract the metadata of the data from the TDS XML.
> > So we have to identify the names of experiment, atomic
> > dataset (aggregation of netcdf files) and netcdf file by the
> > names or identifiers used by the TDS (e.g. with field
> > dataset_ID). Presently, there are differences between them.
> >
> > I attached a figure visualizing the differences in DRS and
> > TDS field 'dataset_ID'.
> >
> > ----------
> > DRS-Syntax: We agreed on moving the <version> from the
> > position behind the atomic dataset to the position behind the
> > realm, i.e.
> >
> > cmip5.<product>.<institute>.<model>.<experiment>.<frequency>.<
> > realm>.<version>.<ensemble>.<variable>.<netcdf>
> >
> > Karl, could you please update the DRS document?
> >
> > Bob, why does the TDS realm version include the ensemble member?
> >
> > cmip5.<product>.<institute>.<model>.<experiment>.<frequency>.<
> > realm>.<ensemble>.<version>
> >
> > So, in the DRS a realm like atmos of an experiment performed
> > with a specific Earth System model has a version, but in the
> > TDS every realization of this experiment (ensemble member)
> > has its own version.
> >
> > ----------
> > Mapping TDS dataset ID to DRS syntax:
> > Down to the realm the TDS ID is identical with the DRS
> > syntax, beneath it not.
> > Bob, I need a clear mapping direction for the atomic dataset
> > and chunk levels.
> > We figured out the following mapping from our example
> > publication. Can you verify that?
> >
> > atomic dataset ID:
> >
> > cmip5.<product>.<institute>.<model>.<experiment>.<frequency>.<
> > realm>.<ensemble>.<variable>.1.aggregation
> >
> > Question: Is the '1' identical with the realm.ensemble
> > version 'v1'? If not, what does it mean and where do we find
> > the <version>?
> >
> > chunk dataset ID:
> >
> > cmip5.<product>.<institute>.<model>.<experiment>.<frequency>.<
> > realm>.<ensemble>.<version>.<netcdf>
> >
> > Question: The <variable> is left out there and can be
> > extracted as first part of the chunk name split by '_'?
> >
> > By the way, could you explain, why the TDS dataset IDs are
> > not identical with the DRS syntax any longer?
> >
> > If we do not have a clear definition of the DRS syntax and
> > mapping directions between TDS IDs and DRS syntax, we cannot
> > assign DOIs to a distinct set of chunks organized according
> > to the DRS syntax (for which we have the required quality
> > information).
> >
> > Alternatively, we would have to scan the netcdf dataheaders
> > and build aggregations for the atomic dataset level again. We
> > would like to avoid that additional effort.
> >
> > I need answers and stability in the DRS syntax and TDS IDs
> > soon. I suggest we speak about this on the telco 14th or 21st
> > September.
> >
> >
> > Best wishes,
> > Martina
> >
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
-- 
Scanned by iCritical.