<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Hi, Sylvia,<br>
<br>
thanks for the definition of terms. Yes, I talk of simulations. <br>
<br>
We have QC data and CIM quality documents on<br>
- datasets (every displayed entry in your screenshot)<br>
- simulations<br>
<br>
Not all datasets and not all latest versions of datasets belong to
the quality level assigned to the simulation. How is that made
obvious to the user? E.g, we assign QC L2 to a set of datasets and
display that information for the simulation. Then we find an error
in QC L3 checks and new data is delivered by the modeling center and
published with QC L1. Then the QC L2 of the simulation refers to old
datasets but the displayed output (latest dataset versions) would be
QC L1. <br>
<br>
We store the whole DRS syntax in the QCDB, based on the data
directory structure underneath the TDS. In the examples: <br>
cmip5.output1.IPSL.IPSL-CM5A-LR.amip4K, <br>
cmip5.output1.IPSL.IPSL-CM5A-LR.amip4K.3hr.atmos.3hr.r1i1p1.v20110429<br>
The keys, like "experiment", are not present in the directory
structure and therefore not in the QCDB. If there is need, we can
provide that using the DRS syntax definition. For our examples:
experiment=amip4K,model=IPSL-CM5A-LR. The present full DRS syntax is
more complete, though.<br>
If the metafor people come up with a tag to carry this information,
I will use it. Until then I can only provide it as part of the
"title" and "evaluationMethodDescription".<br>
<br>
Do we have to change our CIM quality documents?<br>
<br>
Thanks a lot,<br>
cheers,<br>
Martina<br>
<br>
On 08.07.2011 18:21, sylvia murphy wrote:
<blockquote cite="mid:AAD10AA4-8C82-431D-B81C-7FFB54C3B84F@noaa.gov"
type="cite">Hi Martina,
<br>
<br>
Let me explain what I mean by the data and model-metadata side of
ESG. Some other comments are inline below.
<br>
<br>
There are two places where metadata is displayed within ESG and
they come from totally different sources and are being developed
by two slightly different groups of people.
<br>
<br>
Data:
<br>
Metadata is being harvested from the netCDF files and placed
within a TDS catalog. Some of this metadata is being displayed on
pages that serve the datasets themselves. This is being
developed at PCMDI (and perhaps others) through CMOR and the
Gateway publishing software etc.
<br>
<br>
Model-metadata:
<br>
This is the metadata coming in from the Metafor questionnaire via
XML files that are converted to RDF and displayed in the ESG
trackback pages. This is being developed in a collaboration
between the Curator project and the NCAR ESG team.
<br>
<br>
I am attaching a screen shot showing how the model metadata side
lists the data collections associated with the rendered
simulation:
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
Perhaps the above image will help inform my question. Is there
going to be a QC value for each of the links above, for all the
links as a set (representing all the data that the simulation
produces), or at the level of individual files within each link
listed above?
<br>
<br>
more comments inline below...
<br>
<br>
On Jul 8, 2011, at 7:23 AM, Martina Stockhause wrote:
<br>
<br>
<blockquote type="cite"> Hi, Sylvia,
<br>
<br>
I add the answers for 2) and 3)
<br>
<br>
<blockquote type="cite">2) I am trying to understand what you
mean by the granularity of an ESG dataset / publication unit
and experiment. Again I just do see this side of things....
<br>
The current query from the model metadata side displays all
the datasets that have model=x and experiment=y in their TDS.
Is there one QC value for this entire set or are they on the
level of the individual datasets in that set (e.g. from a data
search..... project=CMIP5 / IPCC Fifth Assessment Report,
model=HadGEM2-ES, Met Office Hadley Centre,
experiment=historical, time_frequency=6hr, modeling
realm=atmos, ensemble=r1i1p1, version=20101208)
<br>
<br>
The reason I ask is that if there is one QC value for the
entire set, then displaying this information with the model
metadata makes sense. If it is at the level of the search
listing above, then it may best be displayed with the data
page itself since there is a one to one correspondence.
<br>
<br>
</blockquote>
I am not sure what you mean by model metadata side. If I speak
of
<br>
experiment I mean the TDS sense of experiment (in metafor that
would be
<br>
a simulation) and I mean the full DRS name of the experiment,
e.g.
<br>
cmip5.output1.IPSL.IPSL-CM5A-LR.amip4K. For me the model is part
of this
<br>
DRS experiment name "IPSL-CM5A-LR".
<br>
If your model metadata is the metadata of the realization of one
<br>
experiment performed by one model, than this should be my
"experiment".
<br>
</blockquote>
<br>
Just to restate how we think of this....to us an experiment is
something like CMIP5 rcp45. It is a list of requirements. Every
model participating in CMIP5 will run a simulation against this
experiment. Therefore, a simulation is a model run that for
CMIP5 conforms to a particular experiment. It produces a bunch of
files. I think this is your "experiment".
<br>
<br>
<br>
<blockquote type="cite">
<br>
The information for the QC status of an experiment is not
sufficient,
<br>
esp. if it comes to DOIs. DOIs are related to the latest version
(in our
<br>
example: version=20101208 ) of all datasets belonging to an
experiment
<br>
at the time of DOI assignment. Afterwards that version(s) of the
<br>
individual datasets belonging to the DOI are fixed. If there is
data
<br>
published afterwards under a new version, it is not part of the
DOI. So,
<br>
if you have a quality information/status for an experiment, not
all of
<br>
the datasets and not always the latest versions of every dataset
are
<br>
connected to it. For the time of QC level assignment for the
experiment
<br>
and CIM experiment document publication, the CIM dataset
documents tell
<br>
you which versions of the datasets are part of that assignment.
<br>
</blockquote>
<br>
Not sure what you mean by a CIM experiment document and a CIM
dataset document here. To my knowledge the gateways are receiving
one CIM document set, the content of the questionnaire. Not sure
what these others are. Of course your new QC XML is another
document but that is new.
<br>
<br>
<blockquote type="cite">
<br>
Difficult to explain, I hope you get it nevertheless.
<br>
<blockquote type="cite">3) The primary suggestion I would make
is that if there is key information that we need to extract
and display or extract to connect to something else, that this
should not be buried in a long string. It should be a stand
alone entity.
<br>
</blockquote>
I agree. I just did not find a proper tag. I would be grateful
for any
<br>
suggestions. I hoped that the metafor people would make assist
me, but
<br>
they seem to be occupied by other topics. Besides, I still have
not
<br>
understood there concept how to map metafor UUIDs to
CMIP5/Thredds DRS_IDs.
<br>
</blockquote>
<br>
Alas, I am not the person to assist with this. I can tell you
that ESG has gotten away from using the DRS ID in its entirety for
connecting model metadata and datasets and is now just focusing on
two parts of it, the model name + the experiment (or if that does
not exist) the model name + the simulation.
<br>
<br>
If those two pieces of information were to exist in the XML file,
we could connect the QC value to simulation metadata (again
assuming it exists on that granularity).
<br>
<br>
<br>
<blockquote type="cite">
<br>
Have a nice weekend,
<br>
Martina
<br>
<br>
<blockquote type="cite">
<br>
On Jul 7, 2011, at 4:52 AM, Martina Stockhause wrote:
<br>
<br>
<blockquote type="cite"> Hi, Sylvia,
<br>
<br>
we offer two preliminary CIM Quality examples to give you a
first
<br>
impression of the quality documents to be harvested by the
gateways:
<br>
<br>
<a class="moz-txt-link-freetext" href="http://anticyclone.dkrz.de:8088/geonetwork/srv/en/atom.latestCIM">http://anticyclone.dkrz.de:8088/geonetwork/srv/en/atom.latestCIM</a>
<br>
internal_ids=1896,1897
<br>
<br>
A few explanations where to find the core information:
<br>
<br>
- Quality information will be provided on ESG dataset /
publication unit
<br>
level and experiment level. The example with
internal_id=1897 is a
<br>
dataset and the internal_id=1896 is the related experiment.
<br>
<br>
- pass/fail information: "pass" with pass=0 (failed) pass=1
(passed).
<br>
The examples haven't completed QC L2 and therefore are
pass=0.
<br>
Interesting are more the pass=1 documents. Therefore I
suggest to
<br>
publish them by AtomFeed only for QC status changes and
therefore a pass=1.
<br>
<br>
- QC Level: "nameOfMeasure" or "measureIdentification/code"
(string
<br>
includes "2" or "3"). These are examples for QC L2.
<br>
<br>
- DRS_ID: I did not find an exclusive tag for that,
therefore it is part
<br>
of "evaluationMethodDescription", e.g.
<br>
dataset_id=cmip5.output1.IPSL.IPSL-CM5A-LR.amip4K.3hr.atmos.3hr.r1i1p1.v20110429
<br>
or part of the "title". I would not suggest to use the
"title". We keep
<br>
the version because for QC L3 it is important to assign only
a specific
<br>
version of a dataset the QC level 3.
<br>
<br>
<br>
Status and expected changes:
<br>
- Hans and I were handed over Bryan's QC tool for CIM
quality document
<br>
creation. We still need a bit of development and discussion
about the
<br>
application with Bryan, before we can use it. Therefore the
CIM document
<br>
might change (CIM schema v1.5 should be stable, but
semantics might change).
<br>
- The field for the DRS_ID might change.
"measureIdentification/code"
<br>
and "pass" will remain. We can give you distinct
<br>
"measureIdentification/code" values for QC L2 and QC L3 when
we are ready.
<br>
- Address of AtomFeed might change.
<br>
- Schema location for validation will be added, document_id
etc. with
<br>
the use of Bryan's QC tool.
<br>
<br>
If you need more information or have suggestions for
changes, please let
<br>
us know.
<br>
<br>
Best wishes,
<br>
Martina and Hans
<br>
<br>
_______________________________________________
<br>
GO-ESSP-TECH mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<br>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
<br>
</blockquote>
***********************************
<br>
Sylvia Murphy
<br>
NESII/CIRES/NOAA Earth System Research Laboratory
<br>
325 Broadway, Boulder CO 80305
<br>
Email: <a class="moz-txt-link-abbreviated" href="mailto:sylvia.murphy@noaa.gov">sylvia.murphy@noaa.gov</a>
<br>
Phone: 303-497-7753
<br>
Fax: 303-497-7649
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
</blockquote>
<br>
<br>
-- <br>
------------------ DKRZ / Data Management ------------------
<br>
Martina Stockhause <br>
Deutsches Klimarechenzentrum phone: +49-40-460094-122
<br>
Bundesstr. 45a FAX: +49-40-460094-106
<br>
D-20146 Hamburg, Germany e-mail: <a class="moz-txt-link-abbreviated" href="mailto:stockhause@dkrz.de">stockhause@dkrz.de</a>
<br>
------------------------------------------------------------
<br>
<br>
_______________________________________________
<br>
GO-ESSP-TECH mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<br>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
<br>
</blockquote>
<br>
<pre wrap="">
<fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<br>
</body>
</html>