[Go-essp-tech] Fwd: Re: status metadata from/to CIM for quality control in CMIP5
Martina Stockhause
martina.stockhause at zmaw.de
Thu Nov 4 03:05:40 MDT 2010
Hi, Bryan, hi Gerry,
thanks to Sylvia and Hannah who provided me with the address of the test
AtomFeed, I would like to make a suggestion for point 1 of my open issue
list,
get CIM simulationRun for CMIP5 experiment:
The test AtomFeed provides a list of links to the documents available.
To find a specific simulation document in a simulation AtomFeed the
individual entries in the list need a DRS syntax tag. In your test
example the "title" tag is unused. Could we agree on using that for the
DRS experiment name of the simulation document?
E.g. cmip5/output/MOHC/HadGEM2-ES/piControl ?
QC L3 cross-checks and contact extraction:
I have to examples: one MPI-M amip XML exported out of the questionnaire
and the test example out of the AtomFeed Mark_MO_Sim_exampleCIM.xml
The responsible parties in the AtomFeed example don't distinguish
between organisations and individuals. All responsible parties are
described as individualNames. At the moment, I extract the institute_id
from the abbreviation tag of the responsible party section for
cross-checks of QC L3 if the contact is an organisationName. And the
information I need, an email address in any of the contacts, is missing.
The location of the start and end times for the simulation in the two
documents differ as well. The options for start time locations are
//xmlns:simulationRun/xmlns:startPoint and
//xmlns:simulationRun/xmlns:duration/xmlns:startDate. In both cases the
value is in ISO datetime format.
In the questionnaire example the start time is found at
//xmlns:simulationRun/xmlns:startPoint and
//xmlns:simulationRun/xmlns:duration/xmlns:startDate is not present in
the xml schema.
In the test example Mark_MO_Sim_exampleCIM.xml
//xmlns:simulationRun/xmlns:startPoint is present in the schema but not
filled, start time is found at
//xmlns:simulationRun/xmlns:duration/xmlns:startDate.
Well, developing cross-checks on non-existing metadata is not easy, but
to have different xml schemas in examples is like reading tea leaves.
I hope we can clearify that soon.
Best wishes,
Martina
-------- Original Message --------
Subject: Re: [Go-essp-tech] status metadata from/to CIM for quality
control in CMIP5
Date: Wed, 3 Nov 2010 11:00:50 -0600
From: Hannah Wilcox <hannah at ucar.edu>
To: martina.stockhause at zmaw.de
CC: Sylvia Murphy <Sylvia.Murphy at noaa.gov>
Hi Martina,
Here's the atom feed URL that we are using: http://q.cmip5.ceda.ac.uk/feeds/cmip5/test/
Hannah
On Nov 3, 2010, at 10:54 AM, Sylvia Murphy wrote:
> Hi Hannah,
> Can you provide Martina with the URL of the feed we are playing with
> Sylvia
>
> Begin forwarded message:
>
>> From: Martina Stockhause<martina.stockhause at zmaw.de>
>> Date: November 2, 2010 8:08:49 AM MDT
>> To: Sylvia Murphy<Sylvia.Murphy at noaa.gov>
>> Subject: Re: [Go-essp-tech] status metadata from/to CIM for quality control in CMIP5
>>
>> Hi, Sylvia,
>>
>> thanks a lot for the example.
>>
>> Every incomplete example is welcome to extract some knowledge about CIM out of it.
>>
>> Which is the atom feed address you use?
>>
>> Cheers,
>> Martina
>>
>>
>> On 11/02/2010 03:29 PM, Sylvia Murphy wrote:
>>>
>>> Hi Martina,
>>> This is the latest sample XML file that has been made available to the community. It is out in the Atom feed at the moment but due to some ID confusion we have not been able to download it from there. I am sending it to you in case you have the same issue.
>>>
>>> The<authorship> tag currently does not exist in the output. The output format is far from complete at this point.
>>>
>>> Cheers,
>>>
>>> Sylvia
>>> On Nov 1, 2010, at 7:27 AM, Martina Stockhause wrote:
>>>
>>>> Hi Bryan, hi Rupert,
>>>>
>>>>
>>>>
>>>> On 11/01/2010 11:01 AM, Bryan Lawrence wrote:
>>>>> Hi Martina
>>>>>
>>>>> I'm glad you're keeping a list of live issues. Ideally we should have
>>>>> this a list of trac tickets, so I can tick things off as I answer them.
>>>>> Can you, would you, either use the metafor or go-essp trac to do this
>>>>> (or even the dkrz/is-enes issue trackers, don't mind which).
>>>>>
>>>>>> 1. get CIM simulationRun for CMIP5 experiment:
>>>>>>
>>>>>> I have still not understood how this will be provided: as AtomFeed
>>>>>> including all described simulationRuns or by a tool underlying the
>>>>>> CIM portal (ask for simulationRun of DRS experiment) providing the
>>>>>> needed simulationRun? I could need an example or at least the
>>>>>> description of my interface.
>>>>> There is an atom feed of simulation instances ... so that's the place to
>>>>> get it. Gerry has an example instance hooked up right now, he can give
>>>>> you the exact URL. (I expect some slight changes to the details of how
>>>>> things are laid out in that instance, but we'll synchronise any changes
>>>>> with those of you dependent downstream: ie you and ESG)>
>>>> Yes, please send the URL to the AtomFeed example to me.
>>>> Then I can check
>>>> - whether I am able to identify the simulationRun document belonging to
>>>> my DRS experiment and
>>>> - if and where I find the field "authorship" in the XML.
>>>>
>>>> During production the AtomFeed address will be
>>>> http://q.cmip5.ceda.ac.uk/feeds/cmip5/simulation/ ?
>>>>>> 2. get contact out of CIM simulationRun:
>>>>> So (I think) there are two things one needs: you need a contact person
>>>>> (wand that's the role=contact)_ person. There is also a box for someone
>>>>> to put a complete list of authors suitable for direct transposition into
>>>>> a formatted reference. (The box called authorship, e.g.
>>>>> http://q.cmip5.ceda.ac.uk/cmip5/15/simulation/29/edit/ )
>>>>>
>>>>>> the list of authors of the DOI during QC L3. And for the contact I
>>>>>> asked a couple of times to make the "email" field in the parties
>>>>>> form mandatory. Unfortunately, it is not.
>>>>> We should make that part of our qc-2m for the contact author. (There are
>>>>> good reasons why an emali address for all parties is not desirable, but
>>>>> obviously we need one here).
>>>>>
>>>> Answering to you and Rupert here. I suggest that the simulation contact
>>>> and the email address for this specific contact are mandatory. Or if
>>>> not, that qc-2m takes care that there will be one email address
>>>> available for qc-2d and qc-3. That would enable me to extract all
>>>> necessary information out of the simulationRun document.
>>>>>> Where do I get the email address for my contact? Will you check and
>>>>>> if necessary add that during QC L2 for metadata? If not, is there a
>>>>>> list of contacts including email addresses available that I could
>>>>>> use instead of CIM metadata?
>>>>> So, we need them to give us a contact email. I think we would use the
>>>>> email that Charlotte will have for support to get them to tell us one in
>>>>> qc-2m.
>>>>>
>>>>>> 3. add quality information to CIM
>>>>>>
>>>>>> qctool: Bryan, I am not allowed to open requests in the metafor
>>>>>> request tracker.
>>>>> Ok, sorry, so please follow the instructions on the front page to get an
>>>>> access (or log it on your redmine with a specfic keyword).
>>>>>
>>>>>> So, I report it here. A report is stored without
>>>>>> the sections conformance result and quantitative result. These
>>>>>> information is simply lost.
>>>>> Can you confirm that happened with the 0.21 version?
>>>> Yes, I can. I decided to report in our redmine:
>>>> https://redmine.dkrz.de/collaboration/projects/cmip5-qc/issues #83,
>>>> since we both have access to it.
>>>>>> I am prepared to export the QC L2 result out of the QC DB. One for
>>>>>> exporting the QC L3 result from CERA will follow. I could need the
>>>>>> XML scheme for the report, since I cannot export it from the qctool
>>>>>> web site (see above).
>>>>> I'm not sure what you are saying here. Do you need a command line tool
>>>>> or not? If not, the (current) schema is available on the repository (and
>>>>> the current qc tool output validates against it).
>>>> Yes, I need the command line tool. I understood that I add the report by
>>>> the tool and an xml snippet for the report. I guess I need a sample call
>>>> of that to be provided tool.
>>>>>> 4. update CIM information for QC L3
>>>>>>
>>>>>> It is still not clear to me, how metadata changed by the data author
>>>>>> during the QC L3 process, which is changed in CERA, will be reported
>>>>>> to CIM.
>>>>> They change it via the questionnaire, thus generating a new version of a
>>>>> document with the same uri (but not therefore, the same url).
>>>>>
>>>>> You see the new version in the feed ... and when we're happy with it, we
>>>>> fix the DOI to point at that specific version.
>>>> The author approval step for QC L3 is supported by the WDCC and
>>>> therefore the CERA DB. Changes during this step directly change data in
>>>> CERA but not in CIM. We ask the author to confirm that CIM metadata is
>>>> checked, but we cannot guarantee that the metadata in CIM is the same as
>>>> in the metadata of CERA, which is the metadata we send to the DOI
>>>> registration agency. That difference will be relevant if the contact
>>>> email is changed, which appears on the DOI target page you host.
>>>>
>>>>> I'll look at your example below in a few days. (I'm travelling to
>>>>> Australia tomorrow and may be even more quiet than usual for about ten
>>>>> days ... or maybe I wont ... if I'm awake in hotel rooms, I intend to
>>>>> work on the atom feed and qc tool issues).
>>>> Then have a nice and safe journey!
>>>> Best wishes,
>>>> Martina
>>>>
>>>>
>>>>
>>>>> Meanwhile, I have three talks to give in those ten days ...
>>>>>
>>>>> Cheers
>>>>> Bryan
>>>>>
>>>>>> Meantime we set up a view in CERA providing information for quality
>>>>>> status and DOI, e.g. for our test case:
>>>>>>
>>>>>> http://cera-www.dkrz.de/ceraMeta/SelectGeneric?&q=*%20from%20cera2.v_
>>>>>> qc_Status%20where%20entry_name%3D%27CMIP5%20output%20MPI-M%20ECHAM6-M
>>>>>> PIOM-LR%20rcp45%27&sep=,&fo=x
>>>>>>
>>>>>> (select * from cera2.v_qc_Status where entry_name='CMIP5 output MPI-M
>>>>>> ECHAM6-MPIOM-LR rcp45';)
>>>>>>
>>>>>> ENTRY_NAME = DRS experiment name (separated by blanks)
>>>>>> ACCESS_SPEC = DOI link if available
>>>>>> SPECIFICATION = quality flags
>>>>>>
>>>>>> We can change that if necessary.
>>>>>>
>>>>>> That's our status for the QC-CIM metadata exchanges.
>>>>>>
>>>>>> Best wishes,
>>>>>> Martina
>>>>>>
>>>>>> On 10/08/2010 05:11 PM, Bryan Lawrence wrote:
>>>>>>> hi Folks
>>>>>>>
>>>>>>> Dominic has deployed the qctool for entering quality information at
>>>>>>> http://qc.cmip5.ceda.ac.uk (just one more "c" than the
>>>>>>> questionnaire).
>>>>>>>
>>>>>>> The front page is open, to get further, you use the same access
>>>>>>> criteria you use for access to the questionnaire.
>>>>>>>
>>>>>>> This tool is for *recording* information. It has an atom feed, so
>>>>>>> other tools can use the information, and we expect humans and
>>>>>>> (other) tools to *generate* the information. Currently one would
>>>>>>> have to enter such information manually (although it does allow
>>>>>>> upload of or links to plots and logfiles). A future version will
>>>>>>> allow one to post an xml upload. Future versions will also allow
>>>>>>> qctool instances deployed to federate with each other.
>>>>>>>
>>>>>>> (On access control: in the next few weeks I am hopeful that the ESG
>>>>>>> openid federation will "go-live", which means that an ESG access
>>>>>>> control token should allow access to either the questionnaire or
>>>>>>> the qctool).
>>>>>>>
>>>>>>> The qctool has had *no* user testing. It might break horribly. It
>>>>>>> is being backed up daily ... but ... It's much simpler than the
>>>>>>> questionnaire, so hopefully we will fix things in much more real
>>>>>>> time, and have little problem persisting information (and, in the
>>>>>>> short term, should we fail, it's less likely to be a problem to
>>>>>>> re-enter ... particularly given it is trivial to get xml versions
>>>>>>> of the material entered from the tool).
>>>>>>>
>>>>>>> (I expect it to change even over the next few days, but these
>>>>>>> changes should just be adding functionality and improving
>>>>>>> look-n-feel, things like a title etc :-) ... but we wanted it it
>>>>>>> out NOW, since we said we would :-).
>>>>>>>
>>>>>>> Feedback on usage gratefully accepted (to the metafor list and/or
>>>>>>> metafor trac site using keyword qctool).
>>>>>>>
>>>>>>> Building this has exposed some interesting issues, both with the
>>>>>>> CIM UML (ConCIM) with the CIM serialisation (ApCIM) that we will
>>>>>>> explore with Allyn at the upcoming metafor services meeting.
>>>>>>>
>>>>>>> Cheers
>>>>>>> Bryan
>>>>>>>
>>>>>>>
>>>>>>> Bryan Lawrence
>>>>>>> Director of Environmental Archival and Associated Research
>>>>>>> (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
>>>>>>> STFC, Rutherford Appleton Laboratory
>>>>>>> Phone +44 1235 445012; Fax ... 5848;
>>>>>>> Web: home.badc.rl.ac.uk/lawrence
>>>>>>> _______________________________________________
>>>>>>> GO-ESSP-TECH mailing list
>>>>>>> GO-ESSP-TECH at ucar.edu
>>>>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>> Bryan Lawrence
>>>>> Director of Environmental Archival and Associated Research
>>>>> (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
>>>>> STFC, Rutherford Appleton Laboratory
>>>>> Phone +44 1235 445012; Fax ... 5848;
>>>>> Web: home.badc.rl.ac.uk/lawrence
>>>> _______________________________________________
>>>> GO-ESSP-TECH mailing list
>>>> GO-ESSP-TECH at ucar.edu
>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>> ***********************************
>>> Sylvia Murphy
>>> NESII/CIRES/NOAA Earth System Research Laboratory
>>> 325 Broadway, Boulder CO 80305
>>> Email: sylvia.murphy at noaa.gov
>>> Phone: 303-497-7753
>>> Fax: 303-497-7649
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>> --
>> ----------- DKRZ / Data Management -----------
>>
>> Martina Stockhause
>> Deutsches Klimarechenzentrum
>> Bundesstr. 45a
>> D-20146 Hamburg
>> Germany
>>
>> phone: +49-40-460094-122
>> FAX: +49-40-460094-106
>> e-mail: martina.stockhause at zmaw.de
>>
>> ----------------------------------------------
>>
>
> ***********************************
> Sylvia Murphy
> NESII/CIRES/NOAA Earth System Research Laboratory
> 325 Broadway, Boulder CO 80305
> Email: sylvia.murphy at noaa.gov
> Phone: 303-497-7753
> Fax: 303-497-7649
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20101104/1de1fb31/attachment-0001.html
More information about the GO-ESSP-TECH
mailing list