<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
<font face="Times New Roman">Dear Ag,<br>
<br>
There is another possible way of handling the "missing data"
issue. I'm not sure that a dataset should be be required to be
complete (i.e., required to include all time slices) to be
considered eligible for DOI assignment. That is, we could relax
the criteria. Note that I don't think we require *all* variables
requested within a single dataset to be present, so some datasets
will indeed be incomplete but be eligible for a DOI. I think the
QC procedure should be to check with the modeling group, and if
they can't supply the missing time-slices, then we somehow note
this flaw in the dataset documentation and if other QC checks are
passed, assign it a DOI. <br>
<br>
The criteria for getting a DOI should be that there are no known
errors in the data itself, and that there are no major problems
with the metadata. In this case the data will be reliable, and
analysts will be welcome to use it and publish results, so I think
it should be assigned a DOI. <br>
<br>
What do others think?<br>
<br>
Best regards,<br>
Karl <br>
<br>
</font><br>
On 4/28/11 3:12 AM, <a class="moz-txt-link-abbreviated" href="mailto:ag.stephens@stfc.ac.uk">ag.stephens@stfc.ac.uk</a> wrote:
<blockquote
cite="mid:E29AD60EA782034B956B91A481410FD30FC77F@EXCHMBX01.fed.cclrc.ac.uk"
type="cite">
<pre wrap="">Dear all,
At BADC we have come across our first "missing data" issue in the CMIP5 datasets we are ingesting. We have an example of some missing months for a particular set of variables that was revealed when running the QC code from DKRZ.
It would be very useful for the CMIP5 archive managers to make an authoritative statement about how we should handle missing data time steps in the archive.
I propose the following response when a Data Node receives a dataset in which time steps are missing:
1. QC manager (i.e. whoever runs the QC code) informs Data Provider that there is missing data in a dataset (specifying full DRS structure and date range missing).
2a. If Data Provider says "no, cannot provide this data" then the affected datasets cannot get a DOI and cannot be part of the "crystallised archive". STOP
2b. Data Provider re-generates files, data is re-ingested, new version is generated, QC is re-run, all is good. STOP
2c. Data Provider cannot re-generate but wants to pass QC - so needs to create the required files full of missing data.
3. Data Provider creates missing data files and sends, data re-ingested, new version is generated, QC re-run, all good. STOP
In cases 2a and 2c it would also be very useful if the dataset is annotated to inform the user which dates have been FILLED with missing data. This would, I believe, be in the QC logs but we might want a more prominent record of this if possible.
Cheers,
Ag
BADC--
Scanned by iCritical.
</pre>
</blockquote>
</body>
</html>