[Go-essp-tech] Agenda: 6/14/10 go-essp-tech metadata pipeline call

[Sylvia Murphy]

Hi Everyone,

On Tuesday 15 June the go-essp-tech call will cover metadata.  Below is the metadata pipeline as I understand it.   Given the fact that we only have an hour, I have listed the items in order of priority. I have placed content of this email on etherpad (http://openetherpad.org/L8xHFG9ywd). I will be taking live notes at this URL during the call.  Participants are welcome join me as I do this.

Since I last sent out this email, both the ESG's and METAFOR's questionnaire schedules have shifted significantly. I have adjusted the target due dates accordingly. 

SCHEDULES:

Please note that Curator's timelines are synced with those of ESG. Below is ESG's current release schedule, which can also be viewed at https://wiki.ucar.edu/display/esgcet/ESG+Gateway+Release+Roadmap.

Please note that code freezes occur TWO WEEKS before these dates. Only bug fixes are allowed after a freeze.  Also note that new features should be demonstrated AT LEAST THREE WEEKS before any release so that changes can be made before the freeze.  I have put these dates into the list below. 

1.1 July 15th (Freeze: July 1st, Demontration Deadline: June 24th)
1.2 August 15th (Freeze: August 2nd, Demonstration Deadline: July 26th)
1.3 September 16th (Freeze: September 2nd, Demonstration Deadline: August 26th)

Resource Constraints: 
 
* Curator's developer Julien will be on travel starting 21 August.  He will be gone one month. His last days to get anything done will be August 19th, 20th, and 23rd. Because of this, Curator's primary contributions to CMIP need to be finalized by ESG's 1.2 release (Freeze 2 August). 

At the bottom of the email there is a draft outline of capabilities by release.

Curator also syncs its efforts to those of METAFOR.  Below I am quoting Charlotte's latest questionnaire release schedule (dated 3 June).

"Here is the release schedule showing just the current version of the timeline – to avoid confusion.
14th June – Beta testing starts
13th-20th July – Release of the questionnaire."
*Middle of August: First CMIP5 files received from METAFOR (Target based upon conversations with METAFOR)

Demonstration Schedule:

a) XML Harvest demonstration.  XML Harvest demonstrations have been occurring on a regular basis.   They are primarily for the METAFOR and go-essp-tech communities.  They show a sample questionnaire output (given to us by METAFOR) and are meant to primarily QC the harvesting process but also give folks an idea of what the questionnaire output looks like so that changes in the questionnaire and/or the ESG display can be identified. The next demonstration is targeted for the week of 28 June.  This date is dependent upon receipt of an updated XML file from Gerry and upon Julien who will need to add code to harvest the improved inputs and conformances.  See 4) below.

b) Scientist demonstration.  To my knowledge, the ESG has been primarily demonstrated to technical personnel.  The entire gateway (data browse, data search, model search, metadata content) needs to be demonstrated to scientists.  Curator/ESG would like to conduct this joint demonstration the week of 19 July in order to make any recommended changes in time for ESG 1.2 release, which is frozen on 2 August. 

METADATA PATHS:

There are three primary paths:  the gridspec path, the netCDF path, and the questionnaire path. Tasks to be completed are embedded under the specific metadata pathway step they refer to.  This is a summary that includes portions of the ESG/CMIP5 effort outside of the Curator project scope and may have gaps and errors. 

Gridspec Path:
   * What is the status of Gridspec command line program?
   * Do we anticipate getting gridspec files now?
   * How do we balence gridspec metadata from the new grid info coming from the Questionnaire?

netCDF Path: 
* Is the DRS final?
* What is the status of CMOR2?
* What is the status of the netCDF harvesting software?
* Are there any missing pieces with respect to netCDF metadata harvesting to ESG display?

Questionnaire Path:  

1) Modelling centers fill out online questionnaire (being developed by METAFOR for 12 July release).

2) METAFOR converts output from the questionnaire into a CIM compliant XML file.

3) METAFOR sends the XML output from the modelling centers to ESG via Atom.
   *In Progress: ESG to write software to periodically query METAFOR's Atom server for new files and to download those files (Target: 1 July)

4) ESG uses software (to be developed by ESG) to convert the XML into an OWL file for ingestation into the Sesame Triple Store.
    * Needed:                  
         a) Better sample XML file from METAFOR: Required from METAFOR (Target: 17 June).  METAFOR has been sending us periodic XML updates.  We need a sample that uses the current version of the mindmaps in order to finish the ingestion of the scientific properties  
         b) Conformance, ensembles, numerical requirements, genealogy, and inputs finalized in both the CIM and XML output (Target: 17 June) so ESG can write harvesting code for those pieces.  Conformance and inputs are partially complete in the output and we have code to harvest those pieces
         c) COMPLETE SAMPLE XML from METAFOR (Target: 23 June)            
         d) Curator/ESG to complete the XML to OWL software (Target: 14 July)   
         e) ESG to create a means of automatically running Julien's harvesting code in an operational manner (Target: 14 July)

5) ESG continuity of operations
    * Don identified the need to develop a plan to reinitialize the sesame triple store with the CMIP5 metadata given a catastrophic failure (Target: 12 July)
 
6) The XML instance is displayed on the web. The XML instances will contain all the information about each CMIP5 model and simulation.  This will include information about the platform the simulation was run on, the descriptive scientific properties of each component, etc. 
   * Needed: Next XML Harvesting demo (Target: week of 28 June)
   * Needed: All of the CMIP5 experiment information needs to be represented in the system.  A separate XML conversion program needs to be written to convert METAFOR's experiment XML files to OWL.  (Target: 14 July)
  * Needed: Formal demonstration to scientists of a complete questionnaire XML file to determine needed changes to the display (Target: week of 19 July)

7) Users can use the ESG search page to find model metadata. My last converstation with Don indicated that ESG was scheduling this task for September.  Is that too late?
    * Needed: Community-wide consensus on the facets displayed for the search
    * Needed: ESG modify the search pages to conform to community desires 
    * Needed: ESG finalize the look and feel of the search page and fix any identified bugs
    
SUMMARY OF METADATA-RELATED CAPABILITIES BY ESG RELEASE

Note:   Here is a current list of baseline model metadata capabilities.  Only future capabilities  are listed under the releases.  Please assume that the baseline carries forward:

Baseline capabilities: 
* Component navigation
* Technical properties displayed
* Basic properties (e.g. institution, contacts etc) displayed
* Pop-up definitions of attributes
* Associated grids displayed
* Datahook
* Initial conditions/boundary conditions displayed
* Conformance displayed
* Scientific properties displayed
* Experiment information displayed

ESG Release 1.2 (Freeze: 2 August)
* XML Harvest complete software complete and made operational
* Experiment information harvested and host files reconciled
* Trackback page display adjusted for users coming via the data browse
* Citation formatting improved
* Genealogy displayed
* Simulation to data connection and publishing process made more robust
* Ensemble information displayed

ESG Release 1.3 (Freeze: 2 September)
* Finalize search interface
* User's changes to the component navigation retained in the session
* Link behavior throughout the site made more consistent
* Loading of the component tree made more efficient

***********************************
Sylvia Murphy
NESII/CIRES/NOAA Earth System Research Laboratory
325 Broadway, Boulder CO 80305
Email: sylvia.murphy at noaa.gov
Phone: 303-497-7753





-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20100614/81860e1f/attachment-0001.html