[Go-essp-tech] status metadata from/to CIM for quality control in CMIP5

[Martina Stockhause]

  Hi Bryan, hi Rupert,



On 11/01/2010 11:01 AM, Bryan Lawrence wrote:
> Hi Martina
>
> I'm glad you're keeping a list of live issues. Ideally we should have
> this a list of trac tickets, so I can tick things off as I answer them.
> Can you, would you, either use the metafor or go-essp trac to do this
> (or even the dkrz/is-enes issue trackers, don't mind which).
>
>> 1. get CIM simulationRun for CMIP5 experiment:
>>
>> I have still not understood how this will be provided: as AtomFeed
>> including all described simulationRuns or by a tool underlying the
>> CIM portal (ask for simulationRun of DRS experiment) providing the
>> needed simulationRun? I could need an example or at least the
>> description of my interface.
> There is an atom feed of simulation instances ... so that's the place to
> get it.  Gerry has an example instance hooked up right now, he can give
> you the exact URL. (I expect some slight changes to the details of how
> things are laid out in that instance, but we'll synchronise any changes
> with those of you dependent downstream: ie you and ESG)>
Yes, please send the URL to the AtomFeed example to me.
Then I can check
- whether I am able to identify the simulationRun document belonging to 
my DRS experiment and
- if and where I find the field "authorship" in the XML.

During production the AtomFeed address will be 
http://q.cmip5.ceda.ac.uk/feeds/cmip5/simulation/ ?
>
>> 2. get contact out of CIM simulationRun:
> So (I think) there are two things one needs: you need a contact person
> (wand that's the role=contact)_ person.  There is also a box for someone
> to put a complete list of authors suitable for direct transposition into
> a formatted reference.  (The  box called authorship, e.g.
> http://q.cmip5.ceda.ac.uk/cmip5/15/simulation/29/edit/ )
>
>> the list of authors of the DOI during QC L3. And for the contact I
>> asked a couple of times to make the "email" field in the parties
>> form mandatory. Unfortunately, it is not.
> We should make that part of our qc-2m for the contact author. (There are
> good reasons why an emali address for all parties is not desirable, but
> obviously we need one here).
>
Answering to you and Rupert here. I suggest that the simulation contact 
and the email address for this specific contact are mandatory. Or if 
not, that qc-2m takes care that there will be one email address 
available for qc-2d and qc-3. That would enable me to extract all 
necessary information out of the simulationRun document.
>> Where do I get the email address for my contact? Will you check and
>> if necessary add that during QC L2 for metadata? If not, is there a
>> list of contacts including email addresses available that I could
>> use instead of CIM metadata?
> So, we need them to give us a contact email. I think we would use the
> email that Charlotte will have for support to get them to tell us one in
> qc-2m.
>
>> 3. add quality information to CIM
>>
>> qctool: Bryan, I am not allowed to open requests in the metafor
>> request tracker.
> Ok, sorry, so please follow the instructions on the front page to get an
> access (or log it on your redmine with a specfic keyword).
>
>> So, I report it here. A report is stored without
>> the sections conformance result and quantitative result. These
>> information is simply lost.
> Can you confirm that happened with the 0.21 version?
Yes, I can. I decided to report in our redmine: 
https://redmine.dkrz.de/collaboration/projects/cmip5-qc/issues #83, 
since we both have access to it.
>> I am prepared to export the QC L2 result out of the QC DB. One for
>> exporting the QC L3 result from CERA will follow. I could need the
>> XML scheme for the report, since I cannot export it from the qctool
>> web site (see above).
> I'm not sure what you are saying here. Do you need a command line tool
> or not? If not, the (current) schema is available on the repository (and
> the current qc tool output validates against it).
Yes, I need the command line tool. I understood that I add the report by 
the tool and an xml snippet for the report. I guess I need a sample call 
of that to be provided tool.
>> 4. update CIM information for QC L3
>>
>> It is still not clear to me, how metadata changed by the data author
>> during the QC L3 process, which is changed in CERA, will be reported
>> to CIM.
> They change it via the questionnaire, thus generating a new version of a
> document with the same uri (but not therefore, the same url).
>
> You see the new version in the feed ... and when we're happy with it, we
> fix the DOI to point at that specific version.
The author approval step for QC L3 is supported by the WDCC and 
therefore the CERA DB. Changes during this step directly change data in 
CERA but not in CIM. We ask the author to confirm that CIM metadata is 
checked, but we cannot guarantee that the metadata in CIM is the same as 
in the metadata of CERA, which is the metadata we send to the DOI 
registration agency. That difference will be relevant if the contact 
email is changed, which appears on the DOI target page you host.

> I'll look at your example below in a few days. (I'm travelling to
> Australia tomorrow and may be even more quiet than usual for about ten
> days ... or maybe I wont ... if I'm awake in hotel rooms, I intend to
> work on the atom feed and qc tool issues).
Then have a nice and safe journey!
Best wishes,
Martina



> Meanwhile, I have three talks to give in those ten days ...
>
> Cheers
> Bryan
>
>> Meantime we set up a view in CERA providing information for quality
>> status and DOI, e.g. for our test case:
>>
>> http://cera-www.dkrz.de/ceraMeta/SelectGeneric?&q=*%20from%20cera2.v_
>> qc_Status%20where%20entry_name%3D%27CMIP5%20output%20MPI-M%20ECHAM6-M
>> PIOM-LR%20rcp45%27&sep=,&fo=x
>>
>> (select * from cera2.v_qc_Status where entry_name='CMIP5 output MPI-M
>> ECHAM6-MPIOM-LR rcp45';)
>>
>> ENTRY_NAME = DRS experiment name (separated by blanks)
>> ACCESS_SPEC = DOI link if available
>> SPECIFICATION = quality flags
>>
>> We can change that if necessary.
>>
>> That's our status for the QC-CIM metadata exchanges.
>>
>> Best wishes,
>> Martina
>>
>> On 10/08/2010 05:11 PM, Bryan Lawrence wrote:
>>> hi Folks
>>>
>>> Dominic has deployed the qctool for entering quality information at
>>> http://qc.cmip5.ceda.ac.uk  (just one more "c" than the
>>> questionnaire).
>>>
>>> The front page is open, to get further, you use the same access
>>> criteria you use for access to the questionnaire.
>>>
>>> This tool is for *recording* information. It has an atom feed, so
>>> other tools can use the information, and we expect humans and
>>> (other) tools to *generate* the information. Currently one would
>>> have to enter such information manually (although it does allow
>>> upload of or links to plots and logfiles). A future version will
>>> allow one to post an xml upload. Future versions will also allow
>>> qctool instances deployed to federate with each other.
>>>
>>> (On access control: in the next few weeks I am hopeful that the ESG
>>> openid federation will "go-live", which means that an ESG access
>>> control token should allow access to either the questionnaire or
>>> the qctool).
>>>
>>> The qctool has had *no* user testing. It might break horribly.  It
>>> is being backed up daily ... but ...  It's much simpler than the
>>> questionnaire, so hopefully we will fix things in much more real
>>> time, and have little problem persisting information (and, in the
>>> short term, should we fail, it's less likely to be a problem to
>>> re-enter ... particularly given it is trivial to get xml versions
>>> of the material entered from the tool).
>>>
>>> (I expect it to change even over the next few days, but these
>>> changes should just be adding functionality and improving
>>> look-n-feel, things like a title etc :-) ... but we wanted it it
>>> out NOW, since we said we would :-).
>>>
>>> Feedback on usage gratefully accepted (to the metafor list and/or
>>> metafor trac site using keyword qctool).
>>>
>>> Building this has exposed some interesting issues, both with the
>>> CIM UML (ConCIM) with the CIM serialisation (ApCIM) that we will
>>> explore with Allyn at the upcoming metafor services meeting.
>>>
>>> Cheers
>>> Bryan
>>>
>>>
>>> Bryan Lawrence
>>> Director of Environmental Archival and Associated Research
>>> (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
>>> STFC, Rutherford Appleton Laboratory
>>> Phone +44 1235 445012; Fax ... 5848;
>>> Web: home.badc.rl.ac.uk/lawrence
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> Bryan Lawrence
> Director of Environmental Archival and Associated Research
> (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
> STFC, Rutherford Appleton Laboratory
> Phone +44 1235 445012; Fax ... 5848;
> Web: home.badc.rl.ac.uk/lawrence