<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="#FFFFCC" lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Gavin et al.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I'm sure a shell would be a great contribution but if we have well documented APIs people will build their own shells -- in python/perl/ruby/jython/beanshell/etc/etc.
The disadvantage of a shell is that users will naturally prefer to drive it from their own environment. You will end up with shell scripts piping commands to the shell because the user is familiar with bash. There in lies a big mess IMHO.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">This is a small point. I'd love to see your shell Gavin -- I can then take it a part and write an equivalent Python API :-)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Stephen.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Consolas;color:#1F497D">---<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Consolas;color:#1F497D">Stephen Pascoe +44 (0)1235 445980<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Consolas;color:#1F497D">Centre of Environmental Data Archival<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Consolas;color:#1F497D">STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> Gavin M. Bell [mailto:gavin@llnl.gov]
<br>
<b>Sent:</b> 06 June 2011 23:31<br>
<b>To:</b> Mattmann, Chris A (388J)<br>
<b>Cc:</b> Pascoe, Stephen (STFC,RAL,RALSP); go-essp-tech@ucar.edu; Cinquini, Luca (3880); esg-node-dev@lists.llnl.gov<br>
<b>Subject:</b> Re: [Go-essp-tech] [esg-node-dev] Use of <metadata> element in THREDDS catalogs<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi Chris, <br>
<br>
Thanks for posting the docs.<br>
I'll take a look at it.<br>
<br>
I am pretty hell bent on building a shell. It is the right thing to do. Maybe this is a place where we can play well with CDX?<br>
I don't know enough about CDX. I do know that command line is the way to go. I don't have a dog in the RPC hunt because if you design things properly all RPC layers should be interchangeable (IMHO).<br>
<br>
I'll have more meat on this skeleton... so in a couple weeks can we gather up interested parties and have this conversation.<br>
<br>
On 6/4/11 2:34 PM, Mattmann, Chris A (388J) wrote: <o:p></o:p></p>
<pre>(from the peanut gallery)<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>If a user wants a file from a dataset, they should be able to get the file but we should maintain the context of the dataset by maintaining the dataset as a physical filesystem construct. For example if you use a mac you will see that an "application" is really a top level directory for a set of files. When you download an application what you get is a set of files in a file hierarchy such that in concert they manifest the application you expect. Along the same lines, I would propose that we have a similar construct for datasets and their relationship to files.<o:p></o:p></pre>
</blockquote>
<pre><o:p> </o:p></pre>
<pre>+1 and FWIW, this is the way that we do it in Apache OODT-ville, and on related projects.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>The details of this layout is something I'd like to bring up for discussion, given that the basic premise of what I am saying is accepted.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>We can then build tools to provide that internalize this construct and thus able to manipulate datasets directly. I have been mulling over building an ESG SHELL and I think I will finally do so.... as a part of that shell you would be able to perform augmented shell commands like "ls" that would operate accordingly in the context of our notion of dataset.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>something like<o:p></o:p></pre>
<pre>.<o:p></o:p></pre>
<pre>`-- foo_dataset<o:p></o:p></pre>
<pre> |-- foo_datafile1.nc<o:p></o:p></pre>
<pre> |-- foo_datafile2.nc<o:p></o:p></pre>
<pre> |-- foo_datafile3.nc<o:p></o:p></pre>
<pre> |-- foo_datafile5.nc<o:p></o:p></pre>
<pre> `-- foo_dataset.catalog<o:p></o:p></pre>
</blockquote>
<pre><o:p> </o:p></pre>
<pre>Note here too that based on our experience on the CDX project, these types of tools are useful that's for sure. See this paper for some more information [1]. One tradeoff is the maintainability of specific UNIX-like commands versus simply putting up a POSIX or service style facade interface in front of the underlying grid services that in turn talk and speak nicely with the UNIX-like commands. One way to do this would be to front ESGF services with WebDAV or something similar (aka FUSE at the filesystem/OS level) and then "mount" ESGF datasets.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>This doesn't solve the problem of downstream services though, and value-added metadata (unless we go so far as to build Spotlight :-), which I don't think is the way to go. Some examples of value-added services that are interesting can be seen in the context of OODT-139 [2] over at Apache. We've been working on pedigree/tracing, metadata dumping, etc. that might be nice to look at and collaborate on here. <o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre><o:p> </o:p></pre>
<pre>With this kind of structure you would always have the full catalog for the dataset present and represented. You may have all or a subset of files that are in the catalog present. In the replication scenario, you would have them all. In the end user scenario you may have a subset. The augmented esgf-shell "ls" command you would be able to additionally see what files are present vs what files are not. <o:p></o:p></pre>
</blockquote>
<pre><o:p> </o:p></pre>
<pre>Yep, we've built some commands to do this too in CDX, called cdxls. <o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>Also because you have the catalog you can check the checksums of the files and you can then issue an esgf-shell command to "complete" the dataset and have it pull down the rest of the files. <o:p></o:p></pre>
</blockquote>
<pre><o:p> </o:p></pre>
<pre>(for us, this is cdxget). <o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre> In the replication scheme I am exploring this is how this is intended to work. Also the location of the top level foo_dataset is under the data.repl directory where all replicas are kept. This bears fruit down the line by simplifying several operations down the line. This imposition is not required for the data publisher over datasets that they are custodians for, because of the ability to use the publisher's database to perform this file location - which is part of another scheme I have hatched to divorce the filesystem from the tyranny of the DRS's overreaching (IMHO) filesystem mandate. <o:p></o:p></pre>
</blockquote>
<pre><o:p> </o:p></pre>
<pre>Heh.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre><o:p> </o:p></pre>
<pre>Now I'll be the first to mention that this proposal to impose a filesystem structure is somewhat hypocritical, since I have railed against the DRS's imposition of structure on the filesystem... but I think in this context is it limited enough in scope and provides enough of a benefit to be justified.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>I'd like to have this conversation.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Trust me... this is the way to go. (IMHO) :-)<o:p></o:p></pre>
</blockquote>
<pre><o:p> </o:p></pre>
<pre>I'd like to be a part of this conversation as well.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Thanks!<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Cheers,<o:p></o:p></pre>
<pre>Chris<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>[1] <a href="http://sunset.usc.edu/~mattmann/pubs/SEN10.pdf">http://sunset.usc.edu/~mattmann/pubs/SEN10.pdf</a><o:p></o:p></pre>
<pre>[2] <a href="http://issues.apache.org/jira/browse/OODT-139">http://issues.apache.org/jira/browse/OODT-139</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<o:p></o:p></pre>
<pre>Chris Mattmann, Ph.D.<o:p></o:p></pre>
<pre>Senior Computer Scientist<o:p></o:p></pre>
<pre>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA<o:p></o:p></pre>
<pre>Office: 171-266B, Mailstop: 171-246<o:p></o:p></pre>
<pre>Email: <a href="mailto:chris.a.mattmann@nasa.gov">chris.a.mattmann@nasa.gov</a><o:p></o:p></pre>
<pre>WWW: <a href="http://sunset.usc.edu/~mattmann/">http://sunset.usc.edu/~mattmann/</a><o:p></o:p></pre>
<pre>Phone: +1 (818) 354-8810<o:p></o:p></pre>
<pre>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<o:p></o:p></pre>
<pre>Adjunct Assistant Professor, Computer Science Department<o:p></o:p></pre>
<pre>University of Southern California, Los Angeles, CA 90089 USA<o:p></o:p></pre>
<pre>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<pre>-- <o:p></o:p></pre>
<pre>Gavin M. Bell<o:p></o:p></pre>
<pre>Lawrence Livermore National Labs<o:p></o:p></pre>
<pre>--<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre> "Never mistake a clear view for a short distance."<o:p></o:p></pre>
<pre> -Paul Saffo<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>(GPG Key - <a href="http://rainbow.llnl.gov/dist/keys/gavin.asc">http://rainbow.llnl.gov/dist/keys/gavin.asc</a>)<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre> A796 CE39 9C31 68A4 52A7 1F6B 66B7 B250 21D5 6D3E<o:p></o:p></pre>
</div>
<br><p>--
<BR>Scanned by iCritical.
</p>
<br></body>
</html>