<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
</head>
<body text="#000000" bgcolor="#ffffff">
<br>
Hi Ag,<br>
<br>
we classified a missing time step in the exception category "Error",
which means that the QC manager can decide to assign QC level 2 for
the data. For QC L3 / DOI data publication we rely on the results of
QC L2, thus we would follow the decision of the QC manager. <br>
<br>
The assignment procedure is documented in:
<a class="moz-txt-link-freetext"
href="https://redmine.dkrz.de/collaboration/projects/cmip5-qc/wiki/Qcl2_criteria">https://redmine.dkrz.de/collaboration/projects/cmip5-qc/wiki/Qcl2_criteria</a><br>
The exceptions and their categories are documented in: <a
class="moz-txt-link-freetext"
href="http://www.leuchtturm-atlas.de/SCR/qc2list.html">http://www.leuchtturm-atlas.de/SCR/qc2list.html</a><br>
<br>
Our view is nearly the same as yours, just a few precisions:<br>
*1. ok<br>
*2a1. If the QC manager decides to reject this data in total and not
assign QC L2. -> stop; data remains on QC level 1.<br>
*2a2. For other cases: Only part of the data, a few variables, do
not meet the QC L2 requirements: Publish a new dataset version in
the ESG publisher excluding these variables and rerun the QC L2<br>
*2b. ok<br>
*2c. If the QC manager decides to accept the data then it is
assigned QC level 2 and can get a DOI. The QC manager should add a
comment during QC L2 assignment for that, which is stored in the
metadata and is needed by the WDCC for the DOI process.<br>
*3. We do not see a difference between 2b and 3. If the data
provider decides to create new data than this is published as a new
version and the QC L2 tool is restarted. <br>
<br>
*A provider probably will not be able to only create the missing
months, as the last time step of them should fit to the following
step (which is the first step after the gap). This normally is not
the case as most progtrammes do not run two times the same way in
parameter space.<br>
So to avoid discontinuities at end of gap after this has been
filled: <br>
*the data after the gap will be dumped,<br>
*the programme restart will be initialised with the last step before
the gap and has to run until normal ending,<br>
*old data before the gap and new data (from start of gap on) may be
glued - new version... and so on!<br>
<br>
The assignment of a new version in the ESG publisher for newly
delivered data is important.<br>
<br>
Thanks for bringing up this discussion.<br>
<br>
Best wishes,<br>
Martina and Frank<br>
<br>
<br>
-------- Original Message --------
<table class="moz-email-headers-table" border="0" cellpadding="0"
cellspacing="0">
<tbody>
<tr>
<th valign="BASELINE" align="RIGHT" nowrap="nowrap">Subject: </th>
<td>[Go-essp-tech] Handling missing data in the CMIP5 archive</td>
</tr>
<tr>
<th valign="BASELINE" align="RIGHT" nowrap="nowrap">Date: </th>
<td>Thu, 28 Apr 2011 10:12:51 +0000</td>
</tr>
<tr>
<th valign="BASELINE" align="RIGHT" nowrap="nowrap">From: </th>
<td><a class="moz-txt-link-rfc2396E"
href="mailto:ag.stephens@stfc.ac.uk"><ag.stephens@stfc.ac.uk></a></td>
</tr>
<tr>
<th valign="BASELINE" align="RIGHT" nowrap="nowrap">To: </th>
<td><a class="moz-txt-link-rfc2396E"
href="mailto:go-essp-tech@ucar.edu"><go-essp-tech@ucar.edu></a>,
<a class="moz-txt-link-rfc2396E"
href="mailto:taylor13@llnl.gov"><taylor13@llnl.gov></a></td>
</tr>
</tbody>
</table>
<br>
<br>
<pre>Dear all,
At BADC we have come across our first "missing data" issue in the CMIP5 datasets we are ingesting. We have an example of some missing months for a particular set of variables that was revealed when running the QC code from DKRZ.
It would be very useful for the CMIP5 archive managers to make an authoritative statement about how we should handle missing data time steps in the archive.
I propose the following response when a Data Node receives a dataset in which time steps are missing:
1. QC manager (i.e. whoever runs the QC code) informs Data Provider that there is missing data in a dataset (specifying full DRS structure and date range missing).
2a. If Data Provider says "no, cannot provide this data" then the affected datasets cannot get a DOI and cannot be part of the "crystallised archive". STOP
2b. Data Provider re-generates files, data is re-ingested, new version is generated, QC is re-run, all is good. STOP
2c. Data Provider cannot re-generate but wants to pass QC - so needs to create the required files full of missing data.
3. Data Provider creates missing data files and sends, data re-ingested, new version is generated, QC re-run, all good. STOP
In cases 2a and 2c it would also be very useful if the dataset is annotated to inform the user which dates have been FILLED with missing data. This would, I believe, be in the QC logs but we might want a more prominent record of this if possible.
Cheers,
Ag
BADC--
Scanned by iCritical.
_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</body>
</html>