<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi, Bob,<br>
<br>
what I do not understand is that you have the right version syntax
on the TDS (<a class="moz-txt-link-freetext" href="http://pcmdi3.llnl.gov/thredds/esgcet/catalog.html">http://pcmdi3.llnl.gov/thredds/esgcet/catalog.html</a>):<br>
<a
href="http://pcmdi3.llnl.gov/thredds/esgcet/18/cmip5.output2.INM.inmcm4.1pctCO2.mon.seaIce.OImon.r1i1p1.v20110323.html"><tt>cmip5.output2.INM.inmcm4.1pctCO2.mon.seaIce.OImon.r1i1p1.v20110323/</tt></a><br>
but the wrong in the QC DB<br>
test1=> select * from v_last_version where
exp_name='cmip5/output1/INM/inmcm4/1pctCO2';<br>
exp_id | exp_name | exp_type | ads_id
| ads_name
| ads_type<br>
--------+----------------------------------+----------+--------+--------------------------------------------------------------------+-----------<br>
2738 | cmip5/output1/INM/inmcm4/1pctCO2 | DOI_EXP | 2623 |
cmip5/output1/INM/inmcm4/1pctCO2/mon/seaIce/OImon/r1i1p1/snomelt/1 |
ATOMIC_DS<br>
<br>
Moreover even the position of old-style version and variable are
switched in the QCDB.<br>
<br>
If you run the versioning tool before ESG publication and then the
QC tool on the "versionened" data, you should have no problem with
QC tool application.<br>
<br>
Best wishes,<br>
Martina<br>
<br>
<br>
On 04/18/2011 09:44 PM, Drach, Bob wrote:
<blockquote cite="mid:C9D1E294.1FEED%25drach1@llnl.gov" type="cite">
<pre wrap="">I'm also in agreement on the salient points:
- QC is an essential part of CMIP5
- CMIP5 is using date-style versioning. (Yes, I'll remind the data node
publishers).
I don't think the implementation issues are (or should be) showstoppers.
We'll find a way to make the QC tool work in our environment.
Regards,
Bob
On 4/18/11 1:21 AM, <a class="moz-txt-link-rfc2396E" href="mailto:stephen.pascoe@stfc.ac.uk">"stephen.pascoe@stfc.ac.uk"</a> <a class="moz-txt-link-rfc2396E" href="mailto:stephen.pascoe@stfc.ac.uk"><stephen.pascoe@stfc.ac.uk></a>
wrote:
</pre>
<blockquote type="cite">
<pre wrap="">I completely agree with Martin. For us the key event that dictates the
dataset version is when the data is placed in our archive -- the date of that
event is what becomes the version number.
I also notice that neither the CCCMA, NASA-GISS or BCC datanodes are using
date-versioning!
Stephen.
---
Stephen Pascoe +44 (0)1235 445980
Centre of Environmental Data Archival
STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
-----Original Message-----
From: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a> [<a class="moz-txt-link-freetext" href="mailto:go-essp-tech-bounces@ucar.edu">mailto:go-essp-tech-bounces@ucar.edu</a>] On
Behalf Of <a class="moz-txt-link-abbreviated" href="mailto:martin.juckes@stfc.ac.uk">martin.juckes@stfc.ac.uk</a>
Sent: 18 April 2011 08:48
To: <a class="moz-txt-link-abbreviated" href="mailto:martina.stockhause@zmaw.de">martina.stockhause@zmaw.de</a>; <a class="moz-txt-link-abbreviated" href="mailto:drach1@llnl.gov">drach1@llnl.gov</a>; <a class="moz-txt-link-abbreviated" href="mailto:taylor13@llnl.gov">taylor13@llnl.gov</a>; Lawrence,
Bryan (STFC,RAL,RALSP)
Cc: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a>; <a class="moz-txt-link-abbreviated" href="mailto:michael.lautenschlager@zmaw.de">michael.lautenschlager@zmaw.de</a>; <a class="moz-txt-link-abbreviated" href="mailto:painter1@llnl.gov">painter1@llnl.gov</a>
Subject: Re: [Go-essp-tech] Versioning in CMIP5 including QC procedure
Hello All,
I'd like to back up Martina on this. We do have an agreed version control
system defined in the DRS document (or, more precisely, the DRS document
defines an aspect of the version control which should be implemented -- namely
a subdirectory level which indicates the version).
We agreed that the version of a publication dataset should be reflected in the
directory structure, and it is clear that this requires that the version of
the publication dataset be determined before running the ESG publisher, not by
the ESG publisher. The QC software runs on the file system rather than
accessing data through the data node software, so consistency between the
file-system layout and the publication units is essential. While it is
possible to imagine archives in which all access to data is through the data
node and the file-system layout is of no interest anyone but the data node
development team, that is clearly not the situation here -- no matter how
desirable it might be.
QC is a very important part of ensuring consistency of quality in the archive,
we really need to make it work.
At BADC, we started doing the layout on the same machine as the publishing,
but are now moving it to a different machine -- there is no fundamental
problem with this (the machine on which the layout is done can, of course,
communicate with the machine on which the publishing is done).
regards,
Martin
From: <a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a> [<a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a>] on behalf
of Martina Stockhause [<a class="moz-txt-link-abbreviated" href="mailto:martina.stockhause@zmaw.de">martina.stockhause@zmaw.de</a>]
Sent: 18 April 2011 06:57
To: Drach, Bob
Cc: GO-ESSP; Painter, Jeff; michael.lautenschlager
Subject: Re: [Go-essp-tech] Versioning in CMIP5 including QC procedure
Hi, Bob,
since not every published dataset is part of the DOI (on the level of
experiment), I have to keep track of versions as well, on the dataset and on
the experiment (DOI) level. The inhomogeneity of the dataset version syntax is
more a problem of version control within the QC than one of the QC L2 checker,
the QC L2 analyzer, or the QC L2 result export for QC L3.
I do not care if the homogeneous version syntax is yours or that of BADC and
DKRZ, though the latter would save me adaptation effort, but *that* it is
homogeneous. Maybe you could talk to Stephen to find an agreement on the
version syntax / version handling.
I am sorry that I have to insist.
Best wishes,
Martina
On 04/15/2011 11:18 PM, Drach, Bob wrote:
Hi Martina,
There are a lot of things I like about the layout tool, but one aspect I'm not
happy with is that it chooses a dataset version. IMO that logic should reside
in the publisher, which has access to the history of dataset publication and
dataset definitions. In our environment the layout is done on a different
machine than publication, and does not have access to that history.
Consequently we support the DRS file layout with the exception of dataset
version numbers, which are defined later in the processing stream.
Would it be difficult to provide an option for the QC tool to ignore
extraneous directories (not defined by DRS)?
Best regards,
Bob
On 4/15/11 4:52 AM, "Martina Stockhause"
<<a class="moz-txt-link-abbreviated" href="mailto:martina.stockhause@zmaw.de">martina.stockhause@zmaw.de</a><UrlBlockedError.aspx>> wrote:
Hi, Dean, Karl, and Bob,
there was a discussion started about different types of versioning inside
ESGF for CMIP5 data on the QC request tracker (see:
<a class="moz-txt-link-freetext" href="http://redmine.dkrz.de/collaboration/issues/321">http://redmine.dkrz.de/collaboration/issues/321</a>). Jeff wrote: "
Bob Drach corrected me on one issue: our PCMDI version numbers are not DRS
version numbers, they are just a tool for keeping track of the data received
at PCMDI. Thus these version numbers are generated at PCMDI, while DRS version
numbers are generated by the data producer. PCMDI does not use Stephen's
versioning tool, or the DRS-style version numbers.
"
Is that right? I thought that we agreed on a versioning procedure using
Stephen's tool.
And I do have a problem with different ESG publication procedures (QC level 1
checks), i.e. different QC procedures at the three partners. Additionally, the
inconsistent naming conventions between WDCC / BADC on one side and PCMDI on
the other side cannot be handled by the QC Workflow. Since we do a federated
QC in three locations we need to use not only the same tools with the same
configurations for a comparability of QC results, but we need to use the same
naming conventions to grant a continuation of the overall QC process with QC
L3 / DOI publication.
Thus the question:
Could PCMDI use Stephen's tool for CMIP5 data versioning as well?
Best wishes,
Martina
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
----------- DKRZ / Data Management -----------
Martina Stockhause
Deutsches Klimarechenzentrum
Bundesstr. 45a
D-20146 Hamburg
Germany
phone: +49-40-460094-122
FAX: +49-40-460094-106
e-mail: <a class="moz-txt-link-abbreviated" href="mailto:martina.stockhause@zmaw.de">martina.stockhause@zmaw.de</a>
----------------------------------------------
</pre>
</body>
</html>