[Go-essp-tech] Updated Metadata Pipeline

[Sylvia Murphy]

Hi Everyone,

In May we finished a three week Curator sprint (25 April-13 May).  Development from the sprint will end up in version 1.3.1.  Below is an update on the status.  We will be starting a new 2 week sprint shortly (13 June-24 June).

***STATUS***
* The metadata pipeline has been tested with 1.3.1, with metadata linked to data and vice versa, for an instance where the model metadata is co-located with the data.   We have not confirmed that metadata and data can be linked when they are on different gateways and will need a test plan for establishing this.  This is the top priority for metadata services at this point and will be the focal point of the upcoming sprint. 

For the 1.3.1 release, a gateway can download files from the Metafor Atom feed and harvest and display the results in the ESG gateway trackback.  The items we can  currently harvest and display include basic properties, technical properties, conformances, platforms, citations, grids, inputs, experiments, and scientific properties. We have tested this capability with 3 CIM samples from MOHC (the Hadgem2-es model) that are currently sitting in METAFOR's live simulation feed.

* Specific accomplishments in the 1.3.1 release include:

** Fixed the harvesting of inputs, citations, platforms, and conformances that broke as a result of CIM changes.
** Fixed the ingestation of CIM XML files from METAFOR that broke once the software switched from the test to live simulation feed.
** Added the ability to harvest the following items and display them on the trackback:
    *** Metadata source
    *** Experiment ID and connection to the CMIP5 experiment information
    *** Ensemble ID
    *** Simulation properties
    *** Conformance Types and Conformance Notes
    *** Project
    *** CIM Grids
** Adjusting the display for component instances with no description provided
** Adjusting the ontology to deal with platform controlled vocabularies called "other"
** Created a file containing all experiment information 
** Received and uploaded new METAFOR scientific property mapping files, which fixed the display of some scientific properties
** Added some explanatory text to the grid page

Caveats:
    ** While we have three nearly complete instances from MOHC to examine, errors in the CIM XML output are still being identified and will have to be adjusted for.
  
*** STILL TO DO ***
Below is the current list of priorities for the upcoming sprint:

* Address and test metadata-data link when information is distributed
* Harvest coupling sources that are files
* Harvest and display the metadata version
* Harvest and display field component properties
* Harvest and display the model modifications section of the CIM XML
* Implement requested component tree features:
   ** Make the tree collapsed by default
   ** Retain a user's changes as they use the tree for navigation
   ** Change the color of the default location from black to something more noticeable
* Improve the tree robustness throughout the software stack.  Currently, attempted changes to the tree to 
fix the above items have caused the tree to break in other parts of the stack.  
* Indicate realms for lower level components in the label for the component

DEMONSTRATIONS
* The Curator Trackback was demonstrated in Brussels (by METAFOR) on May 6th, at the GO-ESSP 
meeting in North Carolina on May 11, during the 5/23/11 Curator Call, and to METAFOR  on 5/24/11.   
Demonstration of the work to be completed in the upcoming sprint is tentatively scheduled for 6/27 and 7/5.
 
***********************************
Sylvia Murphy
NESII/CIRES/NOAA Earth System Research Laboratory
325 Broadway, Boulder CO 80305
Email: sylvia.murphy at noaa.gov
Phone: 303-497-7753
Fax: 303-497-7649







-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20110606/fec6e6aa/attachment.html