<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.6000.16981" name=GENERATOR></HEAD>
<BODY text=#000000 bgColor=#ffffff>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=782111312-23032010>Hi Karl,</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=782111312-23032010></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=782111312-23032010>There is lots to consider in this email. I'll
leave the policy stuff to Bryan but I want to focus on your description of
how versions will work. I think the definition of versions below is
confused. Just to enumerate what I understand you are
saying:</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=782111312-23032010> 1. ESG datanode assigns versions to files and
datasets<BR> 2. A dataset is a collection of variables from a particular
experiment (& realisation) and model (presumably these collections are
realms)<BR> 3. A version subdirectory is inserted after
<ensemble-member> in the DRS hierarchy<BR> 4. variables may be stored
in more than one file.</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=782111312-23032010>There are 3 different version concepts here: file
versions, dataset versions and the version subdirectory (aka DRS-version) and
they all apply to different levels of granularity. This has arisen because
"dataset" is no longer at the same level as DRS-version if we follow Bob's
advice and publish at the realm level. The DRS-version no longer matches
either types of version managed by esg publisher. I think I suggested in a
previous email that this could be solved by moving the version
subdirectory to be directly below realm. </SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2></FONT> </DIV>
<DIV><FONT face=Arial><FONT size=2><SPAN class=782111312-23032010><FONT
color=#0000ff>However, if we stick with the DRS structure as-is there is a
non-trivial relationship between these 3 version concepts. To try and
understand your proposal I've sketched it out at </FONT></SPAN><A
href="http://proj.badc.rl.ac.uk/go-essp/wiki/CMIP5/VersionStructure">http://proj.badc.rl.ac.uk/go-essp/wiki/CMIP5/VersionStructure</A><SPAN
class=782111312-23032010>. I've given 2 scenarios there on how files will
be moved into version folders. How these relate to "dataset version" is
TBD.</SPAN></FONT></FONT></DIV>
<DIV><FONT face=Arial><FONT size=2><SPAN
class=782111312-23032010></SPAN></FONT></FONT><FONT face=Arial color=#0000ff
size=2></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=782111312-23032010>You say version consistency is not essential across
datanodes (#7) and that the version directory should contain a "latest"
directory(#14). In this case how do we know whether 2 datanodes have the
same "latest" data? It would seem obvious that datanodes need consistent
versioning. Even if version numbers are consistent I'm not convinced
"latest" is a good idea. One one datanode "latest" could mean v2 and on
another it could mean v3.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=782111312-23032010></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=782111312-23032010>If a dataset's version can't be kept consistent
across all datanodes we will need another means of determining whether 2
datasets are the same. One possibility is a combined hash of all the
files' tracking_ids, or even combined checksum.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=782111312-23032010></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT><SPAN class=782111312-23032010></SPAN></FONT><FONT
face=Arial color=#0000ff size=2><SPAN
class=782111312-23032010>Cheers,</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=782111312-23032010>Stephen.</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2></FONT> </DIV>
<DIV align=left><FONT face=Arial size=2>---</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>Stephen Pascoe +44 (0)1235
445980</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>British Atmospheric Data
Centre</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>Rutherford Appleton
Laboratory</FONT></DIV>
<DIV> </DIV><BR>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> go-essp-tech-bounces@ucar.edu
[mailto:go-essp-tech-bounces@ucar.edu] <B>On Behalf Of </B>Karl
Taylor<BR><B>Sent:</B> 22 March 2010 09:30<BR><B>To:</B>
GO-ESSP<BR><B>Subject:</B> [Go-essp-tech] How will it all
work?<BR></FONT><BR></DIV>
<DIV></DIV>Dear all,<BR><BR>Here is an attempt to write down how CMIP5 data
might be served by ESG. Perhaps someone can find a better way to do this,
but if not, perhaps this will be acceptable. {I apologize if my limited
understanding of ESG means that either this is impractical or stupid. It
is meant to inspire others to come up with a better approach, but I would like
to see a very explicit written description of any proposed alternative.)
Perhaps some of you will have a chance to study this before our next
teleconference.<BR><BR>Procedure for putting in place the CMIP5
archive:<BR>1. A modeling group generates model output in native format
and file structure.<BR>2. The modeling group rewrites data consistent with
CMIP5 requirements (see attached document) using either CMOR2 or an
equivalent post-processing coding. Data is placed in a directory structure
specified by the
<META content=Word.Document name=ProgId>
<META content="Microsoft Word 12" name=Generator>
<META content="Microsoft Word 12" name=Originator><LINK
href="file:///C:%5CDOCUME%7E1%5CTAYLOR%7E1%5CLOCALS%7E1%5CTemp%5Cmsohtmlclip1%5C01%5Cclip_filelist.xml"
rel=File-List><LINK
href="file:///C:%5CDOCUME%7E1%5CTAYLOR%7E1%5CLOCALS%7E1%5CTemp%5Cmsohtmlclip1%5C01%5Cclip_themedata.thmx"
rel=themeData><LINK
href="file:///C:%5CDOCUME%7E1%5CTAYLOR%7E1%5CLOCALS%7E1%5CTemp%5Cmsohtmlclip1%5C01%5Cclip_colorschememapping.xml"
rel=colorSchemeMapping><!--[if gte mso 9]><xml>
<w:WordDocument>
<w:View>Normal</w:View>
<w:Zoom>0</w:Zoom>
<w:TrackMoves/>
<w:TrackFormatting/>
<w:DoNotShowComments/>
<w:PunctuationKerning/>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:DoNotPromoteQF/>
<w:LidThemeOther>EN-US</w:LidThemeOther>
<w:LidThemeAsian>X-NONE</w:LidThemeAsian>
<w:LidThemeComplexScript>X-NONE</w:LidThemeComplexScript>
<w:Compatibility>
<w:BreakWrappedTables/>
<w:SnapToGridInCell/>
<w:WrapTextWithPunct/>
<w:UseAsianBreakRules/>
<w:DontGrowAutofit/>
<w:SplitPgBreakAndParaMark/>
<w:DontVertAlignCellWithSp/>
<w:DontBreakConstrainedForcedTables/>
<w:DontVertAlignInTxbx/>
<w:Word11KerningPairs/>
<w:CachedColBalance/>
</w:Compatibility>
<w:BrowserLevel>MicrosoftInternetExplorer4</w:BrowserLevel>
<m:mathPr>
<m:mathFont m:val="Cambria Math"/>
<m:brkBin m:val="before"/>
<m:brkBinSub m:val="--"/>
<m:smallFrac m:val="off"/>
<m:dispDef/>
<m:lMargin m:val="0"/>
<m:rMargin m:val="0"/>
<m:defJc m:val="centerGroup"/>
<m:wrapIndent m:val="1440"/>
<m:intLim m:val="subSup"/>
<m:naryLim m:val="undOvr"/>
</m:mathPr></w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="true"
DefSemiHidden="true" DefQFormat="false" DefPriority="99"
LatentStyleCount="267">
<w:LsdException Locked="false" Priority="0" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Normal"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="heading 1"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 2"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 3"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 4"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 5"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 6"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 7"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 8"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 9"/>
<w:LsdException Locked="false" Priority="39" Name="toc 1"/>
<w:LsdException Locked="false" Priority="39" Name="toc 2"/>
<w:LsdException Locked="false" Priority="39" Name="toc 3"/>
<w:LsdException Locked="false" Priority="39" Name="toc 4"/>
<w:LsdException Locked="false" Priority="39" Name="toc 5"/>
<w:LsdException Locked="false" Priority="39" Name="toc 6"/>
<w:LsdException Locked="false" Priority="39" Name="toc 7"/>
<w:LsdException Locked="false" Priority="39" Name="toc 8"/>
<w:LsdException Locked="false" Priority="39" Name="toc 9"/>
<w:LsdException Locked="false" Priority="35" QFormat="true" Name="caption"/>
<w:LsdException Locked="false" Priority="10" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Title"/>
<w:LsdException Locked="false" Priority="1" Name="Default Paragraph Font"/>
<w:LsdException Locked="false" Priority="11" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtitle"/>
<w:LsdException Locked="false" Priority="22" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Strong"/>
<w:LsdException Locked="false" Priority="20" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Emphasis"/>
<w:LsdException Locked="false" Priority="59" SemiHidden="false"
UnhideWhenUsed="false" Name="Table Grid"/>
<w:LsdException Locked="false" UnhideWhenUsed="false" Name="Placeholder Text"/>
<w:LsdException Locked="false" Priority="1" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="No Spacing"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 1"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 1"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 1"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 1"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 1"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 1"/>
<w:LsdException Locked="false" UnhideWhenUsed="false" Name="Revision"/>
<w:LsdException Locked="false" Priority="34" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="List Paragraph"/>
<w:LsdException Locked="false" Priority="29" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Quote"/>
<w:LsdException Locked="false" Priority="30" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Quote"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 1"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 1"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 1"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 1"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 1"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 1"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 1"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 1"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 2"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 2"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 2"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 2"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 2"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 2"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 2"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 2"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 2"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 2"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 2"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 2"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 2"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 2"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 3"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 3"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 3"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 3"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 3"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 3"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 3"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 3"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 3"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 3"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 3"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 3"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 3"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 3"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 4"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 4"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 4"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 4"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 4"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 4"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 4"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 4"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 4"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 4"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 4"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 4"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 4"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 4"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 5"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 5"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 5"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 5"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 5"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 5"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 5"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 5"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 5"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 5"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 5"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 5"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 5"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 5"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 6"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 6"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 6"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 6"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 6"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 6"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 6"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 6"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 6"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 6"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 6"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 6"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 6"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 6"/>
<w:LsdException Locked="false" Priority="19" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Emphasis"/>
<w:LsdException Locked="false" Priority="21" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Emphasis"/>
<w:LsdException Locked="false" Priority="31" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Reference"/>
<w:LsdException Locked="false" Priority="32" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Reference"/>
<w:LsdException Locked="false" Priority="33" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Book Title"/>
<w:LsdException Locked="false" Priority="37" Name="Bibliography"/>
<w:LsdException Locked="false" Priority="39" QFormat="true" Name="TOC Heading"/>
</w:LatentStyles>
</xml><![endif]-->
<STYLE>@font-face {
font-family: Cambria Math;
}
@page Section1 {size: 612.0pt 792.0pt; margin: 72.0pt 72.0pt 72.0pt 72.0pt; mso-header-margin: 36.0pt; mso-footer-margin: 36.0pt; mso-paper-source: 0; }
P.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 6pt 0pt; FONT-FAMILY: "Times New Roman","serif"; mso-style-unhide: no; mso-style-qformat: yes; mso-style-parent: ""; mso-pagination: widow-orphan; mso-hyphenate: none; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: AR-SA
}
LI.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 6pt 0pt; FONT-FAMILY: "Times New Roman","serif"; mso-style-unhide: no; mso-style-qformat: yes; mso-style-parent: ""; mso-pagination: widow-orphan; mso-hyphenate: none; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: AR-SA
}
DIV.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 6pt 0pt; FONT-FAMILY: "Times New Roman","serif"; mso-style-unhide: no; mso-style-qformat: yes; mso-style-parent: ""; mso-pagination: widow-orphan; mso-hyphenate: none; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: AR-SA
}
SPAN.StyleCharChar522ptCustomColorRGB84141212Underline {
FONT-WEIGHT: bold; COLOR: #548dd4; FONT-FAMILY: "Arial","sans-serif"; TEXT-DECORATION: underline; mso-style-unhide: no; mso-style-name: "Style Char Char5 + 22 pt Custom Color(RGB(84141212)) Underline"; mso-ansi-font-size: 22.0pt; mso-bidi-font-size: 22.0pt; mso-ascii-font-family: Arial; mso-hansi-font-family: Arial; mso-font-kerning: .5pt; mso-ansi-language: EN-US; text-underline: single
}
.MsoChpDefault {
FONT-SIZE: 10pt; mso-ansi-font-size: 10.0pt; mso-bidi-font-size: 10.0pt; mso-style-type: export-only; mso-default-props: yes
}
DIV.Section1 {
page: Section1
}
</STYLE>
<!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-qformat:yes;
mso-style-parent:"";
mso-padding-alt:0pt 5.4pt 0pt 5.4pt;
mso-para-margin:0pt;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:"Times New Roman";
mso-fareast-theme-font:minor-fareast;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;}
</style>
<![endif]-->"CMIP5 and AR5 Data Reference Syntax (DRS)" (see <A
class=moz-txt-link-freetext
href="http://cmip-pcmdi.llnl.gov/cmip5/docs/cmip5_data_reference_syntax.pdf">http://cmip-pcmdi.llnl.gov/cmip5/docs/cmip5_data_reference_syntax.pdf</A>)..
[This is automatically assured by CMOR2, but otherwise must be enforced by the
user;]<BR>3. Certain quality control (QC) criteria are guaranteed to be
satisfied by processing output through CMOR2 (available from PCMDI), but,
alternatively, to ensure that the same QC criteria are met by output that
has *not* been processed through CMOR2, this output should be required to
successfully pass the tests imposed by the "CMOR2 checker" code (also available
from PCMDI).<BR>4. For modeling groups hosting an ESG node,
CMIP5-compliant model output is "published" to the ESG federation (i.e., it is
registered in an ESG catalog and becomes visible to the ESG federation). .
Other groups unable to host an ESG node may send output to an archival
center (e.g., PCMDI, BADC, DKRZ) which will become the surrogate "owner" of the
output. The owner will publish the data to the ESG federation. As a first step
in the publication job stream, the files will be moved from a directory at the
"realization" level to a subdirectoy at the "version" level. [CMOR2 writes data
to the following directory:
<activity>/<product>/<institute>/<model>/<experiment>/<frequency>/<modeling
realm>/<variable name>/<ensemble member>/, and in the ESG
publisher procedure, the files will be moved to a directory under <ensemble
member, which will be named v<i> where i is the version assigned to this
by ESG. Note that ESG assigns version numbers to individual files and also to
"datasets". The "i" refers to the file version number. An ESG "dataset"
comprises several variables produced from a single run (and realization) from a
single model. Output from a single variable may be stored in several
files. Thus, a dataset will include files from a number of variables and
in general for each variable the data will be stored in multiple files.
ESG will assign a version number to each file (and the directory name will be
consistent with this) and ESG will also assign a version number to each
dataset. If the version of any file within a dataset is incremented, then
the version of the dataset must be incremented.(by 1). <BR>5. As a
step in the ESG publication procedure, a subdirectory under <ensemble
member> will be created named "latest" and a link will be crated in
this subdirectory pointing to the latest version of all files that together
contribute to the latest version of the dataset. This so-called "latest"
subdirectory can be accessed to retrieve the most recent (and, presumably,
trustworthy) model output available.<BR>6. The owner (or surrogate owner)
of the model output will send the so-called "CMIP5 requested model output" (as
defined in a document available from <A class=moz-txt-link-freetext
href="http://cmip-pcmdi.llnl.gov/cmip5/output_req.html?submenuheader=3#req_list">http://cmip-pcmdi.llnl.gov/cmip5/output_req.html?submenuheader=3#req_list</A>)
via 2-Tbyte disks to PCMDI (and subsequently it will be passed on to other
archival centers). Each of the archival centers will decide whether to
store all of the requested output or some subset of the requested output, or
none of the output. There is no requirement that all of the archival
centers host exactly the same portion of model output.<BR>7. Each archival
center will store the output in a directory structure consistent with the "CMIP5
and AR5 Data Reference Syntax (DRS)", as described above. The "version
number" assigned to each file (and as automatically guaranteed by the ESG
publication procedure also assigned to the directory name?) would ideally be the
same as that found at the data owner's node, but I don't think this is
essential. Note that each of the archival centers will publish to the ESG
federation the subset (or complete) model output it chooses to archive.
<BR>8. If users find errors in the model output that has been published
(or if additional quality assurance procedures applied by the ESG federation
uncover any flaws), it is reported to the data "owner" who may withdraw the
output and possibly replace it with corrected output. If the data is
withdrawn and not replaced, the data owner informs the federation that data has
been withdrawn, and the archival centers withdraw all the affected files.
At all sites the dataset version is incemented, and the withdrawn files are not
included in this new version of the dataset. If the data is replaced, the
data owner publishes the new data (placed in an incremented "version"
subdirectory) and informs the federation that the data has been
replaced. The archival centers update their archives with the latest files
(placed in incremented "version" subdirectories). . At all sites the
dataset "version" is also incremented and this new dataset version now includes
the replacement files.<BR>9. At a time when the dataset has "matured" and
it is deemed appropriate, a (substantial) subset of the "CMIP5 requested
output" for a given model and experiment will be submitted for assignment
of a DOI's. (DOI's will be assigned with a granularity following the ESG
"dataset" granularity -- i.e., DOI's will be assigned to each subset of a single
model's output defined by a single experiment, a single realization, a single
realm, and a single frequency. The dataset will include many
variables.) The procedure for assigning a DOI to model output is described
elsewhere, but a requirement is that the data must be archived at one, some, or
all of the following locations: PCMDI, BADC, and DKRZ. The expected
persistence of these groups and their ability to support data archives makes it
likely that the output will remain accessible far into the future.<BR>10.
As part of the submission procedure for DOI status, the model output "owner"
will publish a set of new "ESG datasets" that will typically include only a
subset of the original model output. Each of these new datasets is a
candidate for DOI assignment. Because these new ESG datasets constitute a
subset of the originally published "model output", they may not be of much
interest to users who come to ESG in search of data (since the users will
presumably be keen to examine *all* the model output). Nevertheless, if
DOI status is granted, the subset of output included will presumably be
perceived as somewhat more permanent and reliable (since we expect additional
quality assurance procedures will be invoked in the procedure to gain DOI
status). The DOI's will also serve future researchers who might want to
reproduce research results that cite certain DOI-labeled datasets. The
modeling groups will also be able to substantiate claims that their data has
actually contributed to the research results that cite their DOI's. This
capability requires that the DOI-designated datasets be given special status by
ESG. With the current ESG design it may be necessary (for this purpose
defining DOI datasets) to create a parallel directory structure to the original
directory structure where the model output is stored. This parallel
directory would contain links to only the the subset of model output files that
are included in the DOI-designated (and ESG federation-replicated) subset.
A user with access to the actual DOI archive directory would only see files
included in the DOI-designated data. The user could go to the *original*
directory to see *all* the data available at the site, which would include the
DOI-designated data. <BR>11. Once the output submitted for DOI candidacy
has been published, archival centers that have copies of this data will publish
to the federation the same (subset of) model output and these copies will be
identified by ESG as "replicated" datasets. These replicated datasets will
likely be subsets of the already published corresponding model output datasets,
in which case there will be two distinct datasets registered with the ESG
federation, one containing the entire available output at the site and the other
containing only the replicated subset.<BR>12. At this point the ESG federation
will be aware of a number of different datasets that are similar but differ in
the fraction of output included from the total output available (within the
granularity defined by the ESG "dataset" definition). For example, the
total output might include all time samples simulated. PCMDI might archive only
a subset of this output. And the DOI-candidate output (which might be
"replicated" at BADC and DKRZ) might include only a subset of the variables (of
most interest). Thus, at least 3 different ESG datasets would be defined,
with only one of these being replicated across certain archival
centers.<BR>13. The user who comes to an ESG portal should be able to
search the distributed, federated ESG database and find out whether data of
interest is available. Initially it will likely be unimportant (from the
user's perspective) to learn where exactly the data is stored (and I think the
user should initially not see all the different ESG datasets that include the
data of interest). But before the user actually attempts to retrieve the
data, he/she should be given the opportunity to select a preferred site from
which to obtain it. ESG should then provide the wget script (or
equivalent) that the user can subsequently use to download the data. This
wget script would access the data from the preferred site, unless it were
unavailable there in which case it would direct the user to an archive where is
was available. <BR>14. Note that the directory structure described
above includes a "latest" subdirectory containing links that point to the most
recent versions of files available. The wget script should probably point
to the links in this "latest" subdirectory because this will make it possible
for the user to edit the script to obtain files for a different variable.
If the wget script points to the actual file location for a particular variable,
the user will in general be unable to easily edit the wget script to get a
different variable because the "version subdirectory" where the latest version
of each file is located may differ from one variable to another.<BR><BR>I have
left out the details of what specific QC procedures are required at various
points in the procedure. I have also omitted lots of details that will
have to be worked out. Note also that I do not think "replication" is of
major interest or concern. My view is that whether a given dataset is
replicated or not is not so important. ESG will say what files are
available (and it will know where copies of individual files can be
found). My guess is that most of the major "archival centers" will want to
have copies of the files that are DOI-anointed, and ESG should be able to keep
track of these "replicated" datasets. If this is not practical, I'm not
sure how making a "bigger deal" about replication remedies the any difficulty
posed by the above.<BR><BR>I look forward to your reactions/comments/alternative
suggestions.<BR><BR>Best regards,<BR>Karl<BR><BR>P.S. It's rather late in the
evening, so please allow for that in reading the above.
<BR><BR><BR><BR>
<br><p>--
<BR>Scanned by iCritical.
</p>
<br></BODY></HTML>