<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hey Gavin,<br>
<br>
maybe you want to talk to the globus people about their CLI
(<a class="moz-txt-link-freetext" href="https://www.globusonline.org/usingcli/">https://www.globusonline.org/usingcli/</a>)?<br>
<br>
I'm not sure this is the moment for that though, but if you still
have spare resources, godspeed to you :-)<br>
<br>
Cheers,<br>
Estani<br>
<br>
Am 07.06.2011 19:35, schrieb Gavin M. Bell:
<blockquote cite="mid:4DEE6165.8090502@llnl.gov" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
Hi <br>
<br>
Let's not take things apart, let's just work on things together.
I am all about interfaces. :-)<br>
<br>
For good or bad, I will take a stab at writing this in java, which
means that I have to bring in some machinery to make the java io
work nicely with the terminal, I am pretty much thinking of
leveraging jline and apache cli. I figure since python has a repl
that it is much easier to build such things leveraging python's
repl. The basic command structure is what we should hash out
together. Basically, the anatomy of a command and the basic IO
and how we string them together. Once that is settled the we can
build new commands all we want. I think this is a perfect thing
that we can do together to have both shells available. I dunno on
the state of Jython, maybe this would be something to help bridge
things?<br>
<br>
We can also talk to our CDX friends in the group, Chris et. al. to
see what they have come up with.<br>
<br>
Having a shell saves us from having to go through the pain that it
takes to build a darn webby front end for every darn thing we want
to interact with and do. No disrespect to the webby folks, I
realize that the web is here to stay and it takes talent to make
that happen (lots of HCI issues) but long live the command line!<br>
<br>
P.S.<br>
On a semi-related note, I am thinking starting to code things up
(side little things like this) in clojure (<a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://clojure.org/">http://clojure.org/</a>) - have to
get back to LISP ;-). LISP would be perfect for this task. But
this is neither here nor there at the moment. Indeed I am aware
of the social/group ramifications of this, but maybe it will do us
all good to get some LISP in our lives... after all it is the
basis for both Java and Python in many ways.<br>
<br>
On 6/7/11 2:35 AM, <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:stephen.pascoe@stfc.ac.uk">stephen.pascoe@stfc.ac.uk</a>
wrote:
<blockquote
cite="mid:4C353E6E4A08AE4792B350DAA392B52119F750@EXCHMBX01.fed.cclrc.ac.uk"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<meta name="Generator" content="Microsoft Word 12 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size: 11pt;
font-family: "Calibri","sans-serif";
color: rgb(31, 73, 125);">Gavin et al.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt;
font-family: "Calibri","sans-serif";
color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt;
font-family: "Calibri","sans-serif";
color: rgb(31, 73, 125);">I'm sure a shell would be a
great contribution but if we have well documented APIs
people will build their own shells -- in
python/perl/ruby/jython/beanshell/etc/etc. The
disadvantage of a shell is that users will naturally
prefer to drive it from their own environment. You will
end up with shell scripts piping commands to the shell
because the user is familiar with bash. There in lies a
big mess IMHO.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt;
font-family: "Calibri","sans-serif";
color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt;
font-family: "Calibri","sans-serif";
color: rgb(31, 73, 125);">This is a small point. I'd love
to see your shell Gavin -- I can then take it a part and
write an equivalent Python API :-)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt;
font-family: "Calibri","sans-serif";
color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt;
font-family: "Calibri","sans-serif";
color: rgb(31, 73, 125);">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt;
font-family: "Calibri","sans-serif";
color: rgb(31, 73, 125);">Stephen.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt;
font-family: "Calibri","sans-serif";
color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size: 10.5pt;
font-family: Consolas; color: rgb(31, 73, 125);">---<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 10.5pt;
font-family: Consolas; color: rgb(31, 73, 125);">Stephen
Pascoe +44 (0)1235 445980<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 10.5pt;
font-family: Consolas; color: rgb(31, 73, 125);">Centre
of Environmental Data Archival<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 10.5pt;
font-family: Consolas; color: rgb(31, 73, 125);">STFC
Rutherford Appleton Laboratory, Harwell Oxford, Didcot
OX11 0QX, UK<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size: 11pt;
font-family: "Calibri","sans-serif";
color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<div>
<div style="border-width: 1pt medium medium; border-style:
solid none none; border-color: rgb(181, 196, 223)
-moz-use-text-color -moz-use-text-color; padding: 3pt 0cm
0cm;">
<p class="MsoNormal"><b><span style="font-size: 10pt;
font-family:
"Tahoma","sans-serif"; color:
windowtext;" lang="EN-US">From:</span></b><span
style="font-size: 10pt; font-family:
"Tahoma","sans-serif"; color:
windowtext;" lang="EN-US"> Gavin M. Bell [<a
moz-do-not-send="true" class="moz-txt-link-freetext"
href="mailto:gavin@llnl.gov">mailto:gavin@llnl.gov</a>]
<br>
<b>Sent:</b> 06 June 2011 23:31<br>
<b>To:</b> Mattmann, Chris A (388J)<br>
<b>Cc:</b> Pascoe, Stephen (STFC,RAL,RALSP); <a
moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a>;
Cinquini, Luca (3880); <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:esg-node-dev@lists.llnl.gov">esg-node-dev@lists.llnl.gov</a><br>
<b>Subject:</b> Re: [Go-essp-tech] [esg-node-dev] Use
of <metadata> element in THREDDS catalogs<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi Chris, <br>
<br>
Thanks for posting the docs.<br>
I'll take a look at it.<br>
<br>
I am pretty hell bent on building a shell. It is the right
thing to do. Maybe this is a place where we can play well
with CDX?<br>
I don't know enough about CDX. I do know that command line
is the way to go. I don't have a dog in the RPC hunt
because if you design things properly all RPC layers should
be interchangeable (IMHO).<br>
<br>
I'll have more meat on this skeleton... so in a couple weeks
can we gather up interested parties and have this
conversation.<br>
<br>
On 6/4/11 2:34 PM, Mattmann, Chris A (388J) wrote: <o:p></o:p></p>
<pre>(from the peanut gallery)<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;">
<pre>If a user wants a file from a dataset, they should be able to get the file but we should maintain the context of the dataset by maintaining the dataset as a physical filesystem construct. For example if you use a mac you will see that an "application" is really a top level directory for a set of files. When you download an application what you get is a set of files in a file hierarchy such that in concert they manifest the application you expect. Along the same lines, I would propose that we have a similar construct for datasets and their relationship to files.<o:p></o:p></pre>
</blockquote>
<pre><o:p> </o:p></pre>
<pre>+1 and FWIW, this is the way that we do it in Apache OODT-ville, and on related projects.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;">
<pre>The details of this layout is something I'd like to bring up for discussion, given that the basic premise of what I am saying is accepted.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>We can then build tools to provide that internalize this construct and thus able to manipulate datasets directly. I have been mulling over building an ESG SHELL and I think I will finally do so.... as a part of that shell you would be able to perform augmented shell commands like "ls" that would operate accordingly in the context of our notion of dataset.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>something like<o:p></o:p></pre>
<pre>.<o:p></o:p></pre>
<pre>`-- foo_dataset<o:p></o:p></pre>
<pre> |-- foo_datafile1.nc<o:p></o:p></pre>
<pre> |-- foo_datafile2.nc<o:p></o:p></pre>
<pre> |-- foo_datafile3.nc<o:p></o:p></pre>
<pre> |-- foo_datafile5.nc<o:p></o:p></pre>
<pre> `-- foo_dataset.catalog<o:p></o:p></pre>
</blockquote>
<pre><o:p> </o:p></pre>
<pre>Note here too that based on our experience on the CDX project, these types of tools are useful that's for sure. See this paper for some more information [1]. One tradeoff is the maintainability of specific UNIX-like commands versus simply putting up a POSIX or service style facade interface in front of the underlying grid services that in turn talk and speak nicely with the UNIX-like commands. One way to do this would be to front ESGF services with WebDAV or something similar (aka FUSE at the filesystem/OS level) and then "mount" ESGF datasets.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>This doesn't solve the problem of downstream services though, and value-added metadata (unless we go so far as to build Spotlight :-), which I don't think is the way to go. Some examples of value-added services that are interesting can be seen in the context of OODT-139 [2] over at Apache. We've been working on pedigree/tracing, metadata dumping, etc. that might be nice to look at and collaborate on here. <o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;">
<pre><o:p> </o:p></pre>
<pre>With this kind of structure you would always have the full catalog for the dataset present and represented. You may have all or a subset of files that are in the catalog present. In the replication scenario, you would have them all. In the end user scenario you may have a subset. The augmented esgf-shell "ls" command you would be able to additionally see what files are present vs what files are not. <o:p></o:p></pre>
</blockquote>
<pre><o:p> </o:p></pre>
<pre>Yep, we've built some commands to do this too in CDX, called cdxls. <o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;">
<pre>Also because you have the catalog you can check the checksums of the files and you can then issue an esgf-shell command to "complete" the dataset and have it pull down the rest of the files. <o:p></o:p></pre>
</blockquote>
<pre><o:p> </o:p></pre>
<pre>(for us, this is cdxget). <o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;">
<pre> In the replication scheme I am exploring this is how this is intended to work. Also the location of the top level foo_dataset is under the data.repl directory where all replicas are kept. This bears fruit down the line by simplifying several operations down the line. This imposition is not required for the data publisher over datasets that they are custodians for, because of the ability to use the publisher's database to perform this file location - which is part of another scheme I have hatched to divorce the filesystem from the tyranny of the DRS's overreaching (IMHO) filesystem mandate. <o:p></o:p></pre>
</blockquote>
<pre><o:p> </o:p></pre>
<pre>Heh.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<blockquote style="margin-top: 5pt; margin-bottom: 5pt;">
<pre><o:p> </o:p></pre>
<pre>Now I'll be the first to mention that this proposal to impose a filesystem structure is somewhat hypocritical, since I have railed against the DRS's imposition of structure on the filesystem... but I think in this context is it limited enough in scope and provides enough of a benefit to be justified.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>I'd like to have this conversation.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Trust me... this is the way to go. (IMHO) :-)<o:p></o:p></pre>
</blockquote>
<pre><o:p> </o:p></pre>
<pre>I'd like to be a part of this conversation as well.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Thanks!<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Cheers,<o:p></o:p></pre>
<pre>Chris<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>[1] <a moz-do-not-send="true" href="http://sunset.usc.edu/%7Emattmann/pubs/SEN10.pdf">http://sunset.usc.edu/~mattmann/pubs/SEN10.pdf</a><o:p></o:p></pre>
<pre>[2] <a moz-do-not-send="true" href="http://issues.apache.org/jira/browse/OODT-139">http://issues.apache.org/jira/browse/OODT-139</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<o:p></o:p></pre>
<pre>Chris Mattmann, Ph.D.<o:p></o:p></pre>
<pre>Senior Computer Scientist<o:p></o:p></pre>
<pre>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA<o:p></o:p></pre>
<pre>Office: 171-266B, Mailstop: 171-246<o:p></o:p></pre>
<pre>Email: <a moz-do-not-send="true" href="mailto:chris.a.mattmann@nasa.gov">chris.a.mattmann@nasa.gov</a><o:p></o:p></pre>
<pre>WWW: <a moz-do-not-send="true" href="http://sunset.usc.edu/%7Emattmann/">http://sunset.usc.edu/~mattmann/</a><o:p></o:p></pre>
<pre>Phone: +1 (818) 354-8810<o:p></o:p></pre>
<pre>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<o:p></o:p></pre>
<pre>Adjunct Assistant Professor, Computer Science Department<o:p></o:p></pre>
<pre>University of Southern California, Los Angeles, CA 90089 USA<o:p></o:p></pre>
<pre>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<pre>-- <o:p></o:p></pre>
<pre>Gavin M. Bell<o:p></o:p></pre>
<pre>Lawrence Livermore National Labs<o:p></o:p></pre>
<pre>--<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre> "Never mistake a clear view for a short distance."<o:p></o:p></pre>
<pre> -Paul Saffo<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>(GPG Key - <a moz-do-not-send="true" href="http://rainbow.llnl.gov/dist/keys/gavin.asc">http://rainbow.llnl.gov/dist/keys/gavin.asc</a>)<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre> A796 CE39 9C31 68A4 52A7 1F6B 66B7 B250 21D5 6D3E<o:p></o:p></pre>
</div>
<br>
<p>-- <br>
Scanned by iCritical. </p>
<br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Gavin M. Bell
--
"Never mistake a clear view for a short distance."
-Paul Saffo
</pre>
<pre wrap="">
<fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Estanislao Gonzalez
Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: <a class="moz-txt-link-abbreviated" href="mailto:gonzalez@dkrz.de">gonzalez@dkrz.de</a> </pre>
</body>
</html>