<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi Stephen and all,<br>
<br>
Last night I edited the document I sent earlier, but with some minor
changes (you can see because "track changes" is on). Perhaps this
should replace the document already posted on line even though we know
that Stephen will hopefully propose a far superior approach. <br>
<br>
Mostly for Stephen's benefit, I have made some comments below.<br>
<br>
<div style=""><!--[if !supportAnnotations]-->
<hr class="msocomoff" width="33%" align="left" size="1">
</div>
On 23-Mar-10 7:54 AM, <a class="moz-txt-link-abbreviated" href="mailto:stephen.pascoe@stfc.ac.uk">stephen.pascoe@stfc.ac.uk</a> wrote:
<blockquote
cite="mid:EB1E7CB92F5B35459E0B926D2A614DB60BDE8CDC@EXCHANGE19.fed.cclrc.ac.uk"
type="cite">
<meta http-equiv="Content-Type"
content="text/html; charset=ISO-8859-1">
<meta content="MSHTML 6.00.6000.16981" name="GENERATOR">
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="782111312-23032010">Hi Karl,</span></font></div>
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="782111312-23032010"></span></font> </div>
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="782111312-23032010">There is lots to consider in
this email. I'll leave the policy stuff to Bryan but I want to focus
on your description of how versions will work. I think the definition
of versions below is confused. Just to enumerate what I understand you
are saying:</span></font></div>
<div> </div>
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="782111312-23032010"> 1. ESG datanode assigns
versions to files and datasets<br>
2. A dataset is a collection of variables from a particular experiment
(& realisation) and model (presumably these collections are realms)<br>
</span></font></div>
</blockquote>
Note that the dataset is also limited to a single sampling "frequency"<br>
<blockquote
cite="mid:EB1E7CB92F5B35459E0B926D2A614DB60BDE8CDC@EXCHANGE19.fed.cclrc.ac.uk"
type="cite">
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="782111312-23032010"> 3. A version subdirectory
is inserted after <ensemble-member> in the DRS hierarchy<br>
4. variables may be stored in more than one file.</span></font></div>
<div> </div>
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="782111312-23032010">There are 3 different
version concepts here: file versions, dataset versions and the version
subdirectory (aka DRS-version) and they all apply to different levels
of granularity. </span></font></div>
</blockquote>
We should note that the ESG-defined dataset (which, presumably, will
remain at the "realm" level) also differs from the DOI dataset, which I
think will include all realms (and frequencies?? I need to check the
DOI document about this) and may informally also have a "version"
associated with it (correct? by this I mean some data associated with
a particular DOI may be corrected and a new DOI could get assigned to
the corrected DOI dataset). Also, we may need to distinguish between
DRS "atomic" dataset versions (but maybe not?), which will in general
differ from all the other versions.<br>
<br>
The "version subdirectory" should, I think, be assigned a number
consistent with one of the other 3 (or 4?) version numbers (I nominate
it be consistent with the ESG-defined dataset version, but perhaps
there are good arguments against this.)<br>
<br>
<blockquote
cite="mid:EB1E7CB92F5B35459E0B926D2A614DB60BDE8CDC@EXCHANGE19.fed.cclrc.ac.uk"
type="cite">
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="782111312-23032010">This has arisen because
"dataset" is no longer at the same level as DRS-version if we follow
Bob's advice and publish at the realm level. The DRS-version no longer
matches either types of version managed by esg publisher. I think I
suggested in a previous email that this could be solved by moving the
version subdirectory to be directly below realm. </span></font></div>
</blockquote>
If I understood our audio conference this morning, you've withdrawn
this suggestion.<br>
<blockquote
cite="mid:EB1E7CB92F5B35459E0B926D2A614DB60BDE8CDC@EXCHANGE19.fed.cclrc.ac.uk"
type="cite">
<div> </div>
<div><font face="Arial"><font size="2"><span
class="782111312-23032010"><font color="#0000ff">However, if we stick
with the DRS structure as-is there is a non-trivial relationship
between these 3 version concepts. To try and understand your proposal
I've sketched it out at </font></span><a moz-do-not-send="true"
href="http://*proj.badc.rl.ac.uk/go-essp/wiki/CMIP5/VersionStructure">http://*proj.badc.rl.ac.uk/go-essp/wiki/CMIP5/VersionStructure</a><span
class="782111312-23032010">. I've given 2 scenarios there on how
files will be moved into version folders. How these relate to "dataset
version" is TBD.</span></font></font></div>
<div><font face="Arial"><font size="2"><span
class="782111312-23032010"></span></font></font> </div>
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="782111312-23032010">You say version consistency
is not essential across datanodes (#7) and that the version directory
should contain a "latest" directory(#14). In this case how do we know
whether 2 datanodes have the same "latest" data? It would seem obvious
that datanodes need consistent versioning. Even if version numbers are
consistent I'm not convinced "latest" is a good idea. One one datanode
"latest" could mean v2 and on another it could mean v3.</span></font></div>
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="782111312-23032010"></span></font> </div>
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="782111312-23032010">If a dataset's version can't
be kept consistent across all datanodes we will need another means of
determining whether 2 datasets are the same. One possibility is a
combined hash of all the files' tracking_ids, or even combined checksum.</span></font></div>
</blockquote>
I think this deserves some consideration. I think even if we design
things to make sense, "operator error" will likely lead to
inconsistencies between labeled versions from one data location to
another across the federation.<br>
<br>
After looking at your "diagrams" describing the structure, I agree that
your suggestion will allow one to more easily recover an old version
and I think it is probably a better approach. <br>
<br>
One idea for being able to locate the files associated with a
particular DOI would be to include a parallel directory structure with
the links at the "r" level in your diagram not pointing to "latest",
but instead pointing to the files included in that DOI. On the other
hand, perhaps you don't want to create a complete parallel structure
and instead could find a place within the current structure (shown in
your diagram) to host the DOI links. Note that you must allow for
newer versions of the DOI dataset (with a new DOI number), although we
hope this won't be a common occurrence. <br>
<br>
On the other hand, perhaps it is envisioned that the files associated
with the DOI are cataloged somewhere externally and can't be retrieved
directly by just using the directory names and structure as a guide.<br>
<br>
I look forward to the next iteration.<br>
<br>
Best regards,<br>
Karl<br>
<blockquote
cite="mid:EB1E7CB92F5B35459E0B926D2A614DB60BDE8CDC@EXCHANGE19.fed.cclrc.ac.uk"
type="cite">
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="782111312-23032010"></span></font> </div>
<div dir="ltr" align="left"><font><span class="782111312-23032010"></span></font><font
color="#0000ff" face="Arial" size="2"><span class="782111312-23032010">Cheers,</span></font></div>
<div dir="ltr" align="left"><font color="#0000ff" face="Arial"
size="2"><span class="782111312-23032010">Stephen.</span></font></div>
<div> </div>
<div align="left"><font face="Arial" size="2">---</font></div>
<div align="left"><font face="Arial" size="2">Stephen Pascoe +44
(0)1235 445980</font></div>
<div align="left"><font face="Arial" size="2">British Atmospheric
Data Centre</font></div>
<div align="left"><font face="Arial" size="2">Rutherford Appleton
Laboratory</font></div>
<div> </div>
<br>
<div class="OutlookMessageHeader" dir="ltr" align="left" lang="en-us">
<hr tabindex="-1"><font face="Tahoma" size="2"><b>From:</b>
<a class="moz-txt-link-abbreviated" href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a> [<a class="moz-txt-link-freetext" href="mailto:go-essp-tech-bounces@ucar.edu">mailto:go-essp-tech-bounces@ucar.edu</a>] <b>On
Behalf Of </b>Karl Taylor<br>
<b>Sent:</b> 22 March 2010 09:30<br>
<b>To:</b> GO-ESSP<br>
<b>Subject:</b> [Go-essp-tech] How will it all work?<br>
</font><br>
</div>
Dear all,<br>
<br>
Here is an attempt to write down how CMIP5 data might be served by
ESG. Perhaps someone can find a better way to do this, but if not,
perhaps this will be acceptable. {I apologize if my limited
understanding of ESG means that either this is impractical or stupid.
It is meant to inspire others to come up with a better approach, but I
would like to see a very explicit written description of any proposed
alternative.) Perhaps some of you will have a chance to study this
before our next teleconference.<br>
<br>
Procedure for putting in place the CMIP5 archive:<br>
1. A modeling group generates model output in native format and file
structure.<br>
2. The modeling group rewrites data consistent with CMIP5 requirements
(see attached document) using either CMOR2 or an equivalent
post-processing coding. Data is placed in a directory structure
specified by the
<meta content="Word.Document" name="ProgId">
<meta content="Microsoft Word 12" name="Generator">
<meta content="Microsoft Word 12" name="Originator">
<link
href="file://*/C:%5CDOCUME%7E1%5CTAYLOR%7E1%5CLOCALS%7E1%5CTemp%5Cmsohtmlclip1%5C01%5Cclip_filelist.xml"
rel="File-List">
<link
href="file://*/C:%5CDOCUME%7E1%5CTAYLOR%7E1%5CLOCALS%7E1%5CTemp%5Cmsohtmlclip1%5C01%5Cclip_themedata.thmx"
rel="themeData">
<link
href="file://*/C:%5CDOCUME%7E1%5CTAYLOR%7E1%5CLOCALS%7E1%5CTemp%5Cmsohtmlclip1%5C01%5Cclip_colorschememapping.xml"
rel="colorSchemeMapping">
<!--[if gte mso 9]><xml>
<w:WordDocument>
<w:View>Normal</w:View>
<w:Zoom>0</w:Zoom>
<w:TrackMoves/>
<w:TrackFormatting/>
<w:DoNotShowComments/>
<w:PunctuationKerning/>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:DoNotPromoteQF/>
<w:LidThemeOther>EN-US</w:LidThemeOther>
<w:LidThemeAsian>X-NONE</w:LidThemeAsian>
<w:LidThemeComplexScript>X-NONE</w:LidThemeComplexScript>
<w:Compatibility>
<w:BreakWrappedTables/>
<w:SnapToGridInCell/>
<w:WrapTextWithPunct/>
<w:UseAsianBreakRules/>
<w:DontGrowAutofit/>
<w:SplitPgBreakAndParaMark/>
<w:DontVertAlignCellWithSp/>
<w:DontBreakConstrainedForcedTables/>
<w:DontVertAlignInTxbx/>
<w:Word11KerningPairs/>
<w:CachedColBalance/>
</w:Compatibility>
<w:BrowserLevel>MicrosoftInternetExplorer4</w:BrowserLevel>
<m:mathPr>
<m:mathFont m:val="Cambria Math"/>
<m:brkBin m:val="before"/>
<m:brkBinSub m:val="--"/>
<m:smallFrac m:val="off"/>
<m:dispDef/>
<m:lMargin m:val="0"/>
<m:rMargin m:val="0"/>
<m:defJc m:val="centerGroup"/>
<m:wrapIndent m:val="1440"/>
<m:intLim m:val="subSup"/>
<m:naryLim m:val="undOvr"/>
</m:mathPr></w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="true"
DefSemiHidden="true" DefQFormat="false" DefPriority="99"
LatentStyleCount="267">
<w:LsdException Locked="false" Priority="0" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Normal"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="heading 1"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 2"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 3"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 4"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 5"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 6"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 7"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 8"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 9"/>
<w:LsdException Locked="false" Priority="39" Name="toc 1"/>
<w:LsdException Locked="false" Priority="39" Name="toc 2"/>
<w:LsdException Locked="false" Priority="39" Name="toc 3"/>
<w:LsdException Locked="false" Priority="39" Name="toc 4"/>
<w:LsdException Locked="false" Priority="39" Name="toc 5"/>
<w:LsdException Locked="false" Priority="39" Name="toc 6"/>
<w:LsdException Locked="false" Priority="39" Name="toc 7"/>
<w:LsdException Locked="false" Priority="39" Name="toc 8"/>
<w:LsdException Locked="false" Priority="39" Name="toc 9"/>
<w:LsdException Locked="false" Priority="35" QFormat="true" Name="caption"/>
<w:LsdException Locked="false" Priority="10" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Title"/>
<w:LsdException Locked="false" Priority="1" Name="Default Paragraph Font"/>
<w:LsdException Locked="false" Priority="11" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtitle"/>
<w:LsdException Locked="false" Priority="22" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Strong"/>
<w:LsdException Locked="false" Priority="20" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Emphasis"/>
<w:LsdException Locked="false" Priority="59" SemiHidden="false"
UnhideWhenUsed="false" Name="Table Grid"/>
<w:LsdException Locked="false" UnhideWhenUsed="false" Name="Placeholder Text"/>
<w:LsdException Locked="false" Priority="1" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="No Spacing"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 1"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 1"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 1"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 1"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 1"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 1"/>
<w:LsdException Locked="false" UnhideWhenUsed="false" Name="Revision"/>
<w:LsdException Locked="false" Priority="34" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="List Paragraph"/>
<w:LsdException Locked="false" Priority="29" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Quote"/>
<w:LsdException Locked="false" Priority="30" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Quote"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 1"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 1"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 1"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 1"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 1"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 1"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 1"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 1"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 2"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 2"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 2"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 2"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 2"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 2"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 2"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 2"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 2"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 2"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 2"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 2"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 2"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 2"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 3"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 3"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 3"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 3"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 3"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 3"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 3"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 3"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 3"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 3"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 3"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 3"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 3"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 3"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 4"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 4"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 4"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 4"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 4"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 4"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 4"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 4"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 4"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 4"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 4"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 4"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 4"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 4"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 5"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 5"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 5"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 5"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 5"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 5"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 5"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 5"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 5"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 5"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 5"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 5"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 5"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 5"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 6"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 6"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 6"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 6"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 6"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 6"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 6"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 6"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 6"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 6"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 6"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 6"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 6"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 6"/>
<w:LsdException Locked="false" Priority="19" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Emphasis"/>
<w:LsdException Locked="false" Priority="21" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Emphasis"/>
<w:LsdException Locked="false" Priority="31" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Reference"/>
<w:LsdException Locked="false" Priority="32" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Reference"/>
<w:LsdException Locked="false" Priority="33" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Book Title"/>
<w:LsdException Locked="false" Priority="37" Name="Bibliography"/>
<w:LsdException Locked="false" Priority="39" QFormat="true" Name="TOC Heading"/>
</w:LatentStyles>
</xml><![endif]-->
<style>@font-face {
font-family: Cambria Math;
}
@page Section1 {size: 612.0pt 792.0pt; margin: 72.0pt 72.0pt 72.0pt 72.0pt; mso-header-margin: 36.0pt; mso-footer-margin: 36.0pt; mso-paper-source: 0; }
P.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 6pt 0pt; FONT-FAMILY: "Times New Roman","serif"; mso-style-unhide: no; mso-style-qformat: yes; mso-style-parent: ""; mso-pagination: widow-orphan; mso-hyphenate: none; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: AR-SA
}
LI.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 6pt 0pt; FONT-FAMILY: "Times New Roman","serif"; mso-style-unhide: no; mso-style-qformat: yes; mso-style-parent: ""; mso-pagination: widow-orphan; mso-hyphenate: none; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: AR-SA
}
DIV.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 6pt 0pt; FONT-FAMILY: "Times New Roman","serif"; mso-style-unhide: no; mso-style-qformat: yes; mso-style-parent: ""; mso-pagination: widow-orphan; mso-hyphenate: none; mso-fareast-font-family: "Times New Roman"; mso-fareast-language: AR-SA
}
SPAN.StyleCharChar522ptCustomColorRGB84141212Underline {
FONT-WEIGHT: bold; COLOR: #548dd4; FONT-FAMILY: "Arial","sans-serif"; TEXT-DECORATION: underline; mso-style-unhide: no; mso-style-name: "Style Char Char5 + 22 pt Custom Color(RGB(84141212)) Underline"; mso-ansi-font-size: 22.0pt; mso-bidi-font-size: 22.0pt; mso-ascii-font-family: Arial; mso-hansi-font-family: Arial; mso-font-kerning: .5pt; mso-ansi-language: EN-US; text-underline: single
}
.MsoChpDefault {
FONT-SIZE: 10pt; mso-ansi-font-size: 10.0pt; mso-bidi-font-size: 10.0pt; mso-style-type: export-only; mso-default-props: yes
}
DIV.Section1 {
page: Section1
}
</style><!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-qformat:yes;
mso-style-parent:"";
mso-padding-alt:0pt 5.4pt 0pt 5.4pt;
mso-para-margin:0pt;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-fareast-font-family:"Times New Roman";
mso-fareast-theme-font:minor-fareast;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-bidi-font-family:"Times New Roman";
mso-bidi-theme-font:minor-bidi;}
</style>
<![endif]-->"CMIP5
and AR5 Data Reference Syntax (DRS)" (see <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="http://*cmip-pcmdi.llnl.gov/cmip5/docs/cmip5_data_reference_syntax.pdf">http://*cmip-pcmdi.llnl.gov/cmip5/docs/cmip5_data_reference_syntax.pdf</a>)..
[This is automatically assured by CMOR2, but otherwise must be enforced
by the user;]<br>
3. Certain quality control (QC) criteria are guaranteed to be satisfied
by processing output through CMOR2 (available from PCMDI), but,
alternatively, to ensure that the same QC criteria are met by output
that has *not* been processed through CMOR2, this output should be
required to successfully pass the tests imposed by the "CMOR2 checker"
code (also available from PCMDI).<br>
4. For modeling groups hosting an ESG node, CMIP5-compliant model
output is "published" to the ESG federation (i.e., it is registered in
an ESG catalog and becomes visible to the ESG federation). . Other
groups unable to host an ESG node may send output to an archival
center (e.g., PCMDI, BADC, DKRZ) which will become the surrogate
"owner" of the output. The owner will publish the data to the ESG
federation. As a first step in the publication job stream, the files
will be moved from a directory at the "realization" level to a
subdirectoy at the "version" level. [CMOR2 writes data to the following
directory:
<activity>/<product>/<institute>/<model>/<experiment>/<frequency>/<modeling
realm>/<variable name>/<ensemble member>/, and in the
ESG publisher procedure, the files will be moved to a directory under
<ensemble member, which will be named v<i> where i is the
version assigned to this by ESG. Note that ESG assigns version numbers
to individual files and also to "datasets". The "i" refers to the file
version number. An ESG "dataset" comprises several variables produced
from a single run (and realization) from a single model. Output from a
single variable may be stored in several files. Thus, a dataset will
include files from a number of variables and in general for each
variable the data will be stored in multiple files. ESG will assign a
version number to each file (and the directory name will be consistent
with this) and ESG will also assign a version number to each dataset.
If the version of any file within a dataset is incremented, then the
version of the dataset must be incremented.(by 1). <br>
5. As a step in the ESG publication procedure, a subdirectory under
<ensemble member> will be created named "latest" and a link will
be crated in this subdirectory pointing to the latest version of all
files that together contribute to the latest version of the dataset.
This so-called "latest" subdirectory can be accessed to retrieve the
most recent (and, presumably, trustworthy) model output available.<br>
6. The owner (or surrogate owner) of the model output will send the
so-called "CMIP5 requested model output" (as defined in a document
available from <a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://*cmip-pcmdi.llnl.gov/cmip5/output_req.html?submenuheader=3#req_list">http://*cmip-pcmdi.llnl.gov/cmip5/output_req.html?submenuheader=3#req_list</a>)
via 2-Tbyte disks to PCMDI (and subsequently it will be passed on to
other archival centers). Each of the archival centers will decide
whether to store all of the requested output or some subset of the
requested output, or none of the output. There is no requirement that
all of the archival centers host exactly the same portion of model
output.<br>
7. Each archival center will store the output in a directory structure
consistent with the "CMIP5 and AR5 Data Reference Syntax (DRS)", as
described above. The "version number" assigned to each file (and as
automatically guaranteed by the ESG publication procedure also assigned
to the directory name?) would ideally be the same as that found at the
data owner's node, but I don't think this is essential. Note that each
of the archival centers will publish to the ESG federation the subset
(or complete) model output it chooses to archive. <br>
8. If users find errors in the model output that has been published
(or if additional quality assurance procedures applied by the ESG
federation uncover any flaws), it is reported to the data "owner" who
may withdraw the output and possibly replace it with corrected output.
If the data is withdrawn and not replaced, the data owner informs the
federation that data has been withdrawn, and the archival centers
withdraw all the affected files. At all sites the dataset version is
incemented, and the withdrawn files are not included in this new
version of the dataset. If the data is replaced, the data owner
publishes the new data (placed in an incremented "version"
subdirectory) and informs the federation that the data has been
replaced. The archival centers update their archives with the latest
files (placed in incremented "version" subdirectories). . At all sites
the dataset "version" is also incremented and this new dataset version
now includes the replacement files.<br>
9. At a time when the dataset has "matured" and it is deemed
appropriate, a (substantial) subset of the "CMIP5 requested output"
for a given model and experiment will be submitted for assignment of a
DOI's. (DOI's will be assigned with a granularity following the ESG
"dataset" granularity -- i.e., DOI's will be assigned to each subset of
a single model's output defined by a single experiment, a single
realization, a single realm, and a single frequency. The dataset will
include many variables.) The procedure for assigning a DOI to model
output is described elsewhere, but a requirement is that the data must
be archived at one, some, or all of the following locations: PCMDI,
BADC, and DKRZ. The expected persistence of these groups and their
ability to support data archives makes it likely that the output will
remain accessible far into the future.<br>
10. As part of the submission procedure for DOI status, the model
output "owner" will publish a set of new "ESG datasets" that will
typically include only a subset of the original model output. Each of
these new datasets is a candidate for DOI assignment. Because these
new ESG datasets constitute a subset of the originally published "model
output", they may not be of much interest to users who come to ESG in
search of data (since the users will presumably be keen to examine
*all* the model output). Nevertheless, if DOI status is granted, the
subset of output included will presumably be perceived as somewhat more
permanent and reliable (since we expect additional quality assurance
procedures will be invoked in the procedure to gain DOI status). The
DOI's will also serve future researchers who might want to reproduce
research results that cite certain DOI-labeled datasets. The modeling
groups will also be able to substantiate claims that their data has
actually contributed to the research results that cite their DOI's.
This capability requires that the DOI-designated datasets be given
special status by ESG. With the current ESG design it may be necessary
(for this purpose defining DOI datasets) to create a parallel directory
structure to the original directory structure where the model output is
stored. This parallel directory would contain links to only the the
subset of model output files that are included in the DOI-designated
(and ESG federation-replicated) subset. A user with access to the
actual DOI archive directory would only see files included in the
DOI-designated data. The user could go to the *original* directory to
see *all* the data available at the site, which would include the
DOI-designated data. <br>
11. Once the output submitted for DOI candidacy has been published,
archival centers that have copies of this data will publish to the
federation the same (subset of) model output and these copies will be
identified by ESG as "replicated" datasets. These replicated datasets
will likely be subsets of the already published corresponding model
output datasets, in which case there will be two distinct datasets
registered with the ESG federation, one containing the entire available
output at the site and the other containing only the replicated subset.<br>
12. At this point the ESG federation will be aware of a number of
different datasets that are similar but differ in the fraction of
output included from the total output available (within the granularity
defined by the ESG "dataset" definition). For example, the total
output might include all time samples simulated. PCMDI might archive
only a subset of this output. And the DOI-candidate output (which
might be "replicated" at BADC and DKRZ) might include only a subset of
the variables (of most interest). Thus, at least 3 different ESG
datasets would be defined, with only one of these being replicated
across certain archival centers.<br>
13. The user who comes to an ESG portal should be able to search the
distributed, federated ESG database and find out whether data of
interest is available. Initially it will likely be unimportant (from
the user's perspective) to learn where exactly the data is stored (and
I think the user should initially not see all the different ESG
datasets that include the data of interest). But before the user
actually attempts to retrieve the data, he/she should be given the
opportunity to select a preferred site from which to obtain it. ESG
should then provide the wget script (or equivalent) that the user can
subsequently use to download the data. This wget script would access
the data from the preferred site, unless it were unavailable there in
which case it would direct the user to an archive where is was
available. <br>
14. Note that the directory structure described above includes a
"latest" subdirectory containing links that point to the most recent
versions of files available. The wget script should probably point to
the links in this "latest" subdirectory because this will make it
possible for the user to edit the script to obtain files for a
different variable. If the wget script points to the actual file
location for a particular variable, the user will in general be unable
to easily edit the wget script to get a different variable because the
"version subdirectory" where the latest version of each file is located
may differ from one variable to another.<br>
<br>
I have left out the details of what specific QC procedures are required
at various points in the procedure. I have also omitted lots of
details that will have to be worked out. Note also that I do not think
"replication" is of major interest or concern. My view is that whether
a given dataset is replicated or not is not so important. ESG will say
what files are available (and it will know where copies of individual
files can be found). My guess is that most of the major "archival
centers" will want to have copies of the files that are DOI-anointed,
and ESG should be able to keep track of these "replicated" datasets.
If this is not practical, I'm not sure how making a "bigger deal" about
replication remedies the any difficulty posed by the above.<br>
<br>
I look forward to your reactions/comments/alternative suggestions.<br>
<br>
Best regards,<br>
Karl<br>
<br>
P.S. It's rather late in the evening, so please allow for that in
reading the above. <br>
<br>
<br>
<br>
<br>
<p>-- <br>
Scanned by iCritical.
</p>
<br>
</blockquote>
<br>
</body>
</html>