<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<base href="x-msg://5/"><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.apple-tab-span
{mso-style-name:apple-tab-span;}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple" style="word-wrap: break-word;-webkit-nbsp-mode: space;-webkit-line-break: after-white-space">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi Luca,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal">> o Each catalog is harvested as a single discoverable dataset - the reason being that hopefully the data provider thought about how to generate
<o:p></o:p></p>
<p class="MsoNormal">> the catalogs, and decided on what should be the single unit of discovery<o:p></o:p></p>
<p class="MsoNormal">><o:p> </o:p></p>
<p class="MsoNormal">> o For each catalog, all files are assigned to the top-level dataset container - so if there were many nested datasets with files, it still would result
<o:p></o:p></p>
<p class="MsoNormal">> in a single discoverable dataset with as many files<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Does this mean that each catalog should contain 0 or 1 top-level datasets and any further nesting below that is collapsed down? That sounds quite sensible.
What happens to any properties in any dataset below the top-level one?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Stephen.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Consolas;color:#1F497D">---<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Consolas;color:#1F497D">Stephen Pascoe +44 (0)1235 445980<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Consolas;color:#1F497D">Centre of Environmental Data Archival<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Consolas;color:#1F497D">STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Cinquini, Luca (3880) [mailto:Luca.Cinquini@jpl.nasa.gov]
<br>
<b>Sent:</b> 23 February 2012 15:02<br>
<b>To:</b> Pascoe, Stephen (STFC,RAL,RALSP)<br>
<b>Cc:</b> Juckes, Martin (STFC,RAL,RALSP); go-essp-tech@ucar.edu<br>
<b>Subject:</b> Re: [Go-essp-tech] Corrected THREDDS group wiki page link<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi Stephen and Martin,<o:p></o:p></p>
<div>
<p class="MsoNormal"><span class="apple-tab-span"> </span>just for clarification, this is what the P2P harvesting software currently does - this doesn't mean that it cannot be changed if desired:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">o Each catalog can contain an arbitrary hierarchy of datasets and catalogRefs<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">o Each catalog is harvested as a single discoverable dataset - the reason being that hopefully the data provider thought about how to generate the catalogs, and decided on what should be the single unit of discovery<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">o For each catalog, all files are assigned to the top-level dataset container - so if there were many nested datasets with files, it still would result in a single discoverable dataset with as many files<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">o And obviously, all catalogRef are followed in harvesting, and generate separate discoverable datasets.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">thanks, Luca<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Feb 23, 2012, at 7:33 AM, <<a href="mailto:stephen.pascoe@stfc.ac.uk">stephen.pascoe@stfc.ac.uk</a>> <<a href="mailto:stephen.pascoe@stfc.ac.uk">stephen.pascoe@stfc.ac.uk</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks Martin. There is a catalog_version attribute already, although I don't think there is any documentation on what it means.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">On the hierarchy, I personally believe we could allow any number of intermediate catalogues containing <catalogRef> elements in the spec. Datanodes currently
only produce 2 levels .../thredds/catalog.xml and .../thredds/esgcet/catalog.xml, but there would be no harm in having deeper nesting. What I think is less flexible is the constraint that "leaf-catalogs" contain a single container <dataset> element and a
set of child <dataset> elements representing files and aggregations. This design is what LAS and other bits of ESGF rely on. General THREDDS allows you to mix catalogRef, container datasets and "real" datasets throughout the hierarchy.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Anyone, please chip in if you dissagree.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Cheers,</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Stephen.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Consolas;color:#1F497D">---</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Consolas;color:#1F497D">Stephen Pascoe +44 (0)1235 445980</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Consolas;color:#1F497D">Centre of Environmental Data Archival</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Consolas;color:#1F497D">STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm;border-width:initial;border-color:initial">
<div>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span class="apple-converted-space"><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span></span><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">Juckes,
Martin (STFC,RAL,RALSP)<span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>23 February 2012 12:11<br>
<b>To:</b><span class="apple-converted-space"> </span>Pascoe, Stephen (STFC,RAL,RALSP);<span class="apple-converted-space"> </span><a href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a><br>
<b>Subject:</b><span class="apple-converted-space"> </span>RE: Corrected THREDDS group wiki page link</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hello All,</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Sorry I had to leave the telco early – but it was a useful discussion.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">After leaving, I had a couple of thoughts:</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-left:36.0pt">
<p class="MsoNormal" style="text-indent:-18.0pt"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(1)</span><span style="font-size:7.0pt;color:#1F497D"> <span class="apple-converted-space"> </span></span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">There
syntax should be versioned , and the version should be indicated in the catalogue somewhere – whatever we agree on, there is bound to be need to change in the future, and changes will be much easier to manage if we have the version in the catalogue. There
could be independent syntax versions for the top level catalogue and the “publication unit” catalogue. The cleanest way to do this would be with an xsd document referenced in the schemaLocation attribute. We could set this up initially with a “permissive”
xsd schema imposing necessary constraints, but not all the required constraints.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-left:36.0pt">
<p class="MsoNormal" style="text-indent:-18.0pt"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">(2)</span><span style="font-size:7.0pt;color:#1F497D"> <span class="apple-converted-space"> </span></span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">The
decision to stick to a 2-level hierarchy of THREDDS documents (a top-level catalogue with a list of “catalogRef”s and a sub-catalogue for each publication unit) is certainly right for now, but may be too restrictive in the medium term. The specification of
“catalogRef” means that very little information is in the top level, and at the next level you have to fetch everything. Having an 3<sup>rd</sup><span class="apple-converted-space"> </span>level – e.g. for each simulation – would allow more flexibility in
recording changes and pointing to documentation.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Cheers,</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Martin</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt;border-width:initial;border-color:initial">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm;border-width:initial;border-color:initial">
<div>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span class="apple-converted-space"><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span></span><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""><a href="mailto:go-essp-tech-bounces@ucar.edu">go-essp-tech-bounces@ucar.edu</a><span class="apple-converted-space"> </span>[mailto:go-essp-tech-bounces@ucar.edu]<span class="apple-converted-space"> </span><b>On
Behalf Of<span class="apple-converted-space"> </span></b><a href="mailto:stephen.pascoe@stfc.ac.uk">stephen.pascoe@stfc.ac.uk</a><br>
<b>Sent:</b><span class="apple-converted-space"> </span>21 February 2012 16:14<br>
<b>To:</b><span class="apple-converted-space"> </span><a href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a><br>
<b>Subject:</b><span class="apple-converted-space"> </span>[Go-essp-tech] Corrected THREDDS group wiki page link</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><a href="http://esgf.org/wiki/ESGFInterfaceGroups/ThreddsGroup">http://esgf.org/wiki/ESGFInterfaceGroups/ThreddsGroup</a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Consolas">---</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Consolas">Stephen Pascoe +44 (0)1235 445980</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Consolas">Centre of Environmental Data Archival</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Consolas">STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<p>--<span class="apple-converted-space"> </span><br>
Scanned by iCritical.<o:p></o:p></p>
<div>
<p class="MsoNormal"> <span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica","sans-serif""><o:p> </o:p></span></p>
<p>--<span class="apple-converted-space"> </span><br>
Scanned by iCritical.<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica","sans-serif""><br>
_______________________________________________<br>
GO-ESSP-TECH mailing list<br>
<a href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a><br>
<a href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<br><p>--
<BR>Scanned by iCritical.
</p>
<br></body>
</html>