<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Dear all,<br>
<br>
I'll try here to summarize my understanding of how data will get
published and replicated as part of ESG. If my summary is accurate,
there are a number of items we'll need to address soon, which I'll come
back to at the end of this email.<br>
<br>
Consider the following simplified case:<br>
<br>
1. Model A produces precipitation and temperature data for a 100-year
simulation. This will be considered the entire "output".<br>
2. CMIP5 requests that only temperature data be archived, and this
temperature data then constitutes the entire "requested" model output.<br>
3. The ESG federation has agreed that the last 20 years of temperature
data will be replicated at the archival gateways (PCMDI, BADC, DKRZ,
...).<br>
<br>
Thus, the "replicated" output is a subset of the "requested" output,
which is a subset of the "output". Note that in what follows I assume
there is no good reason to separate "requested" from "output". The
"replicated" output, however, needs to be treated someone separately
because of the issues having to do with quality control, versioning and
replication.<br>
<br>
As I understand it, one possible route by which data will appear on ESG
is as follows:<br>
<br>
1. Modeling group A publishes all its "output" on an ESG node. This
requires writing the output files into directories, determining which
data will be part of the official replicated subset and which will not,
collecting files into ESG datasets, and assigning version numbers to
files and to datasets. <br>
<br>
1a) files are initially placed in directories following the DRS
specifications, without assigning a version number. Thus, they are
placed directly in the <ensemble member> directory. Here is the
DRS directory structure: ...../
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
<meta name="ProgId" content="Word.Document">
<meta name="Generator" content="Microsoft Word 12">
<meta name="Originator" content="Microsoft Word 12">
<link rel="File-List"
href="file:///C:%5CDOCUME%7E1%5CTAYLOR%7E1%5CLOCALS%7E1%5CTemp%5Cmsohtmlclip1%5C01%5Cclip_filelist.xml">
<link rel="themeData"
href="file:///C:%5CDOCUME%7E1%5CTAYLOR%7E1%5CLOCALS%7E1%5CTemp%5Cmsohtmlclip1%5C01%5Cclip_themedata.thmx">
<link rel="colorSchemeMapping"
href="file:///C:%5CDOCUME%7E1%5CTAYLOR%7E1%5CLOCALS%7E1%5CTemp%5Cmsohtmlclip1%5C01%5Cclip_colorschememapping.xml">
<!--[if gte mso 9]><xml>
<w:WordDocument>
<w:View>Normal</w:View>
<w:Zoom>0</w:Zoom>
<w:TrackMoves/>
<w:TrackFormatting/>
<w:DoNotShowComments/>
<w:PunctuationKerning/>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:DoNotPromoteQF/>
<w:LidThemeOther>EN-US</w:LidThemeOther>
<w:LidThemeAsian>X-NONE</w:LidThemeAsian>
<w:LidThemeComplexScript>X-NONE</w:LidThemeComplexScript>
<w:Compatibility>
<w:BreakWrappedTables/>
<w:SnapToGridInCell/>
<w:WrapTextWithPunct/>
<w:UseAsianBreakRules/>
<w:DontGrowAutofit/>
<w:SplitPgBreakAndParaMark/>
<w:DontVertAlignCellWithSp/>
<w:DontBreakConstrainedForcedTables/>
<w:DontVertAlignInTxbx/>
<w:Word11KerningPairs/>
<w:CachedColBalance/>
</w:Compatibility>
<w:BrowserLevel>MicrosoftInternetExplorer4</w:BrowserLevel>
<m:mathPr>
<m:mathFont m:val="Cambria Math"/>
<m:brkBin m:val="before"/>
<m:brkBinSub m:val="--"/>
<m:smallFrac m:val="off"/>
<m:dispDef/>
<m:lMargin m:val="0"/>
<m:rMargin m:val="0"/>
<m:defJc m:val="centerGroup"/>
<m:wrapIndent m:val="1440"/>
<m:intLim m:val="subSup"/>
<m:naryLim m:val="undOvr"/>
</m:mathPr></w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="true"
DefSemiHidden="true" DefQFormat="false" DefPriority="99"
LatentStyleCount="267">
<w:LsdException Locked="false" Priority="0" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Normal"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="heading 1"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 2"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 3"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 4"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 5"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 6"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 7"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 8"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 9"/>
<w:LsdException Locked="false" Priority="39" Name="toc 1"/>
<w:LsdException Locked="false" Priority="39" Name="toc 2"/>
<w:LsdException Locked="false" Priority="39" Name="toc 3"/>
<w:LsdException Locked="false" Priority="39" Name="toc 4"/>
<w:LsdException Locked="false" Priority="39" Name="toc 5"/>
<w:LsdException Locked="false" Priority="39" Name="toc 6"/>
<w:LsdException Locked="false" Priority="39" Name="toc 7"/>
<w:LsdException Locked="false" Priority="39" Name="toc 8"/>
<w:LsdException Locked="false" Priority="39" Name="toc 9"/>
<w:LsdException Locked="false" Priority="35" QFormat="true" Name="caption"/>
<w:LsdException Locked="false" Priority="10" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Title"/>
<w:LsdException Locked="false" Priority="1" Name="Default Paragraph Font"/>
<w:LsdException Locked="false" Priority="11" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtitle"/>
<w:LsdException Locked="false" Priority="22" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Strong"/>
<w:LsdException Locked="false" Priority="20" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Emphasis"/>
<w:LsdException Locked="false" Priority="59" SemiHidden="false"
UnhideWhenUsed="false" Name="Table Grid"/>
<w:LsdException Locked="false" UnhideWhenUsed="false" Name="Placeholder Text"/>
<w:LsdException Locked="false" Priority="1" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="No Spacing"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 1"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 1"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 1"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 1"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 1"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 1"/>
<w:LsdException Locked="false" UnhideWhenUsed="false" Name="Revision"/>
<w:LsdException Locked="false" Priority="34" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="List Paragraph"/>
<w:LsdException Locked="false" Priority="29" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Quote"/>
<w:LsdException Locked="false" Priority="30" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Quote"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 1"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 1"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 1"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 1"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 1"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 1"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 1"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 1"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 2"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 2"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 2"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 2"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 2"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 2"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 2"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 2"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 2"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 2"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 2"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 2"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 2"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 2"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 3"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 3"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 3"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 3"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 3"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 3"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 3"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 3"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 3"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 3"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 3"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 3"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 3"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 3"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 4"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 4"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 4"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 4"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 4"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 4"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 4"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 4"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 4"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 4"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 4"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 4"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 4"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 4"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 5"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 5"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 5"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 5"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 5"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 5"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 5"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 5"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 5"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 5"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 5"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 5"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 5"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 5"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 6"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 6"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 6"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 6"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 6"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 6"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 6"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 6"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 6"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 6"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 6"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 6"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 6"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 6"/>
<w:LsdException Locked="false" Priority="19" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Emphasis"/>
<w:LsdException Locked="false" Priority="21" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Emphasis"/>
<w:LsdException Locked="false" Priority="31" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Reference"/>
<w:LsdException Locked="false" Priority="32" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Reference"/>
<w:LsdException Locked="false" Priority="33" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Book Title"/>
<w:LsdException Locked="false" Priority="37" Name="Bibliography"/>
<w:LsdException Locked="false" Priority="39" QFormat="true" Name="TOC Heading"/>
</w:LatentStyles>
</xml><![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;
mso-font-charset:0;
mso-generic-font-family:roman;
mso-font-pitch:variable;
mso-font-signature:-1610611985 1107304683 0 0 159 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-parent:"";
margin-top:6.0pt;
margin-right:0pt;
margin-bottom:6.0pt;
margin-left:0pt;
mso-pagination:widow-orphan;
mso-hyphenate:none;
font-size:12.0pt;
font-family:"Times New Roman","serif";
mso-fareast-font-family:"Times New Roman";
mso-fareast-language:AR-SA;}
.MsoChpDefault
{mso-style-type:export-only;
mso-default-props:yes;
font-size:10.0pt;
mso-ansi-font-size:10.0pt;
mso-bidi-font-size:10.0pt;}
@page Section1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;
mso-header-margin:36.0pt;
mso-footer-margin:36.0pt;
mso-paper-source:0;}
div.Section1
{page:Section1;}
--> </style><span
style="font-size: 12pt; font-family: "Times New Roman","serif";"><<i
style="">activity</i>>/<<i style="">product</i>>/<<i
style="">institute</i>>/<<i style="">model</i>>/<<i
style="">experiment</i>>/<<i style="">frequency</i>>/<<i
style="">modeling
realm</i>>/<<i style="">variable name</i>>/<<i style="">ensemble
member</i>>/</span><br>
1b) ESG decides whether any of the files are supposed to replace
earlier versions and assigns a version number to each file. It also
moves any replaced or withdrawn files down one level into directories
named by the version of the files they contain. Thus, both the latest
versions of files and directories containing earlier versions of the
files will appear under <span
style="font-size: 12pt; font-family: "Times New Roman","serif";"><<i
style="">ensemble member</i>>.</span><br>
1c) ESG decides whether any portion of the data in each file is
included in the officially called for replicated set. [The code has
not yet been written that can make this decision.] If it finds any
data in the "replicated" category, ESG creates a parallel directory
(under the "requested" directory as called for by the current DRS, but
perhaps this should be changed to "replicated"). ESG creates a link
from the new directory to the file itself under "output" (i.e., to the
file or files containing temperature data that falls within the
specified 20-year period.<br>
1d) Assuming publication at the "realm" level, the ESG publisher is
executed on the "output" side of the directory tree, yielding a single
dataset containing all the temperature and precipitation data. Note,
that publication at the "realm" level means all variables from a single
realization of a simulation will be in the same ESG dataset. [The
different members of an ensemble will appear in separate ESG
datasets.] <br>
1e) The ESG publisher will then act on the "links" in the "replicated"
side of the directory tree, and publish a dataset containing only
temperature data from 20 years (plus perhaps any additional years that
might be included as part of the needed files). Thus, the replicated
data will be found either under the "replicated" dataset or the
original "output" dataset (along with additional data stored there).<br>
<br>
2. Modeling group A sends all of its temperature data to PCMDI (because
PCMDI plans to archive as much of the requested data as it can). <br>
<br>
3. PCMDI publishes the data it receives, following a similar procedure
as in 1a-e above, but of course the realm dataset will only include
temperature data. The "replicated" dataset, on the other hand should
be identical to the one published by Modeling Group A on its own node.<br>
<br>
4. The data is sent to other archival gateways (e.g., BADC and DKRZ)
who might choose only to archive the "replicated" subset. They might
place their files directly in the "replicated" side of the directory
tree (and omit the "output" side of the tree). <br>
<br>
Questions:<br>
<br>
1. How we can make sure that the version numbers assigned by the
publishers at the different nodes/gateways are the same across the
federation?<br>
<br>
2. Isn't there an easier way to do all of this?<br>
<br>
3. When a user looks for data, he is more likely to find what he needs
by searching "output" not "replicated" (since "output" is more
complete). In fact I'm not sure the typical user will care what
portion of the data has been replicated. What are the advantages to
the user of requiring that some defined subset of the data be
replicated. Is it in the DOI assignment?<br>
<br>
I've run out of time for now, but I think we still have envisioned how
this is going to work end to end. Also, the requirements of the
"search" capability and the "notification" service still seem quite
vague to me. It seems to me we need to get the specifications down on
paper soon.<br>
<br>
Best regards,<br>
Karl<br>
<br>
<br>
<br>
<br>
Hi Karl,
<br>
this is a VERY good use case, and thinking about it can really
help clarify how the system will or should work, even for me. It might
be worth discussing this with the go-essp list just to make sure
everybody is on the same page. I'm cc'ing Eric too because he is
working on wget scripts right these days...
<br>
<br>
That said, I think the use case is flawed, because, as it stands,
it involves partial replicas of datasets, a thing that we said we
wouldn't support. To be specific, the only way that the modeling center
can only ship one file to PCMDI is that the output stream is split into
2 datasets: "requested" and "full" (or whatever), contrary to
assumption 4) below.
<br>
<br>
So, if we assume that the "full" dataset is composed of two files, and
the "requested" dataset of 1 file, the following happens:
<br>
<br>
o The modeling center publishes the full dataset onto its data node and
to the PCMDI gateway
<br>
o The "requested" dataset is replicated to PCMDI, and published to the
PCMDI datanode and the PCMDI gateway
<br>
o The PCMDI gateway exposes both datasets in the search interface.
The 2 datasets share all the same DRS facets (model, experiment, time
frequency,...) except perhaps a facet called "product" that has the two
possible values "Full CMIP5 output" and "Core CMIP5 output". To be
distinguishable, the two datasets must come with a name/description
that specify their time extent, and/or their product type. We could
also harvest the overall time information and display it, if it can be
helpful.
<br>
o So when users 1, 2, 3 below make a search, 2 results will be
returned: by inspecting the results descriptions, they will realize
that all the original data is available from the modeling center, and
only a subset of it from PCMDI. Depending on what they want, they will
make their dataset selection, click a button, and obtain a files
listing which contains all the files for that particular dataset. At
this point they can still presumably deselect any files they don't want
(perhaps based on the total size displayed) before asking for a wget
script to be generated.
<br>
<br>
In summary, I think the system fully supports this use case
provided the two datasets are identified as distinct at the time of
publication.
<br>
<br>
Also, let me add a few comments. This is a simple use case because
there is only one gateway serving two datanodes. In this case, the
gateway knows exactly which files are present at each data node. If the
user (1, 2 or 3) was going to select BOTH datasets in the search
results and ask for the files, a single web page would be presented
that contains all the files from the two datasets. Since some of the
files share the same name, the gateway can either present two options
for download, or maybe make an authoritative decision and present one
only.
<br>
<br>
More complicated is the case where the modeling center publishes
the "full" dataset to BADC (for example), and the "requested" dataset
is replicated to the PCMDI data node and published to the PCMDI
gateway. In this case, the PCMDI gateway knows about two datasets, but
only the files of its datanode, and similarly the BADC gateway knows
about two datasets, but only the files of the modeling center. In this
scenario, it's very important that the two datasets be accurately
described so that the user can make the proper selection, after which
the listing of files is presented. If the user were to select both
datasets, he would be presented with two sets of files, and two wget
scripts to download them. Probably the worst that can happen in this
case is that if the user doesn't pay attention to the file listing,
he'll download the files twice.
<br>
<br>
I hope this helps in understanding - Bob, Eric please speak up if you
think I got any of this wrong.
<br>
<br>
thanks, Luca
<br>
<br>
<br>
<br>
On Mar 6, 2010, at 11:31 AM, Karl Taylor wrote:
<br>
<br>
Hi Bob and Luca,
<br>
<br>
I'm trying to get a feel for what to expect from a user's
perspective out of a federated ESG, assuming only what software will be
in place at the time of the first release. Consider the following
simple federated archive, involving just two partners -- a modeling
center hosting a data node and PCMDI hosting a data node and a portal
(i.e., a gateway).
<br>
<br>
1. Suppose the archive is tiny and comprises only two files: one
file with precipitation data for years 1-100 of a single simulation,
and the other years 101-200 from the same simulations.
<br>
<br>
2. Suppose the modeling center responsible for the simulation
publishes the data (years 1-200) on its node, and then sends a copy of
only the 2nd file (years 101-200) to PCMDI, which subsequently
publishes it on the PCMDI node.
<br>
<br>
3. The ESG portal at PCMDI knows about both nodes.
<br>
<br>
4. Suppose that there is no special designation associated with
any of the data (e.g., we have not defined a "requested" or
"replicated" subset).
<br>
<br>
I presume the gateway will see 2 different datasets. Could you
please tell me whether the gateway will be aware of all the information
found in the catalogs at both nodes, or only a subset of the
information? (And will the gateway have to retrieve this information
from each node whenever it is needed by a user, or will the gateway
already have a copy?) In particular will the gateway be able to
access (locally?):
<br>
a) the full list of files at each node?
<br>
b) what time period the data covers in each node?
<br>
<br>
Could you also tell me what information/scripts each of the
following users will receive from ESG that will allow him to get the
data he wants?
<br>
User 1:
<br>
This user wants to download all precipitation data available in the
archive. How will he know he should download his data from the
original node, rather than from PCMDI?
<br>
<br>
User 2:
<br>
This user wants to download only years 1-100 of the data. How will
he know he should download his data from the original node, rather than
from PCMDI?
<br>
<br>
'User 3:
<br>
This user wants to download only years 101-200 of the data. How will
he know that he can get his data from either site?
<br>
<br>
The answers to these questions may help guide us in setting priorities
beyond the first release.
<br>
thanks,
<br>
Karl
<br>
<br>
On 26-Feb-10 5:47 AM, Luca Cinquini wrote:
<blockquote cite="mid:9CF9CEA0-F36E-4FF7-B228-39E82EAEB6C5@ucar.edu"
type="cite">Hi Stephen,
<div>it's good to think of all possible scenarios...</div>
<div><br>
</div>
<div>It seems to me like in this case:</div>
<div>o) it would make more sense to change the propose notification
system to operate on datasets, not single files</div>
<div>o) in any case, when the two users compare the plots for
variable V1, the first thing they should do is exchange information
about which file versions they are using - and they would find they
have different versions. If instead they'd rather exchange information
about dataset versions, they can do that too, and they would still find
they are using different versions.</div>
<div><br>
</div>
<div>thanks, Luca</div>
<div><br>
<div>
<div>On Feb 26, 2010, at 4:50 AM, <<a moz-do-not-send="true"
href="mailto:stephen.pascoe@stfc.ac.uk">stephen.pascoe@stfc.ac.uk</a>>
<<a moz-do-not-send="true" href="mailto:stephen.pascoe@stfc.ac.uk">stephen.pascoe@stfc.ac.uk</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div>
<div><font face="Arial" size="2"><span class="009573911-26022010">Another
issue with changing the publication granularity.</span></font></div>
<div><font face="Arial" size="2"><span class="009573911-26022010"></span></font> </div>
<div><font face="Arial" size="2"><span class="009573911-26022010">Will
users be notified about changes to files, atomic-datasets or
realm-datasets? I think Gavin has said in the past that users will be
emailed when *files* change. Consider the scenario:</span></font></div>
<div><font face="Arial" size="2"><span class="009573911-26022010"></span></font> </div>
<div><font face="Arial" size="2"><span class="009573911-26022010"> 1.
A realm-dataset DS1 is published at version v1.</span></font></div>
<div><font face="Arial" size="2"><span class="009573911-26022010"> 2.
User A downloads variable V1 from DS1.</span></font></div>
<div><font face="Arial" size="2"><span class="009573911-26022010"> 3.
User B downloads all of DS1.</span></font></div>
<div><font face="Arial" size="2"><span class="009573911-26022010"> 4.
An error is found in variable V2 of DS1.</span></font></div>
<div><font face="Arial" size="2"><span class="009573911-26022010"> 5.
The files for V2 are replaced and DS1 is republished as version v2.</span></font></div>
<div><font face="Arial" size="2"><span class="009573911-26022010"> 6.
User B is notified that some files have changed in DS1.</span></font></div>
<div><font face="Arial" size="2"><span class="009573911-26022010"> 7.
User A is *not* notified because he never downloaded the files that
changed.</span></font></div>
<div><font face="Arial" size="2"><span class="009573911-26022010"> 8.
User A & B collaborate discussing the data from DS1 v1. THEY HAVE
DIFFERENT FILES!</span></font></div>
<div><font face="Arial" size="2"><span class="009573911-26022010"></span></font> </div>
<div><font face="Arial" size="2"><span class="009573911-26022010">If
this is how the system is supposed to work it's going to be very
confusing.</span></font></div>
<div><font face="Arial" size="2"><span class="009573911-26022010"></span></font> </div>
<div><font face="Arial" size="2"><span class="009573911-26022010">S.</span></font></div>
<div> </div>
<div align="left"><font face="Arial" size="2">---</font></div>
<div align="left"><font face="Arial" size="2">Stephen Pascoe +44
(0)1235 445980</font></div>
<div align="left"><font face="Arial" size="2">British Atmospheric
Data Centre</font></div>
<div align="left"><font face="Arial" size="2">Rutherford Appleton
Laboratory</font></div>
<div> </div>
<br>
<p>-- <br>
Scanned by iCritical. </p>
<br>
</div>
_______________________________________________<br>
GO-ESSP-TECH mailing list<br>
<a moz-do-not-send="true" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a><br>
<a class="moz-txt-link-freetext" href="http://*mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://*mailman.ucar.edu/mailman/listinfo/go-essp-tech</a><br>
</blockquote>
</div>
<br>
</div>
<pre wrap="">
<fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://*mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://*mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<br>
</body>
</html>