<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#ffffff">
On 14/12/2011 21:04, Jennifer Adams wrote:
<blockquote
cite="mid:10DAFC25-1766-41FB-A8F2-79A79F2E34DB@cola.iges.org"
type="cite"><br>
<div>
<div>On Dec 14, 2011, at 12:51 PM, Steve Hankin wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
<div bgcolor="#FFFFFF" text="#000000"> Hi Jennifer,<br>
<br>
I imagine that I am speaking for everyone in ESG in saying
that your clear and constructive comments have been MOST
HELPFUL!<br>
<br>
A question for you: What do you see as the potential
contributions of OPeNDAP access to CMIP datasets? -- both
for yourself and the users that you represent? <br>
</div>
</blockquote>
<div><br>
</div>
I don't see OPeNDAP access solving any of the problems we're
facing now. COLA users need this data on our local high-speed
file system, where we can carve it up and aggregate it and throw
all our analysis methods at it and compare it to other data sets
without worrying about authenitcation, internet bandwidth, or
the status of all the distributed servers. </div>
<div>--Jennifer</div>
<div><br>
</div>
</blockquote>
<br>
We have the same requirements as above. Jennifer is right stating
that opendap won't help on the production datanode side (except for
some quick look here and there).<br>
<br>
But I just want to emphasis that opendap aggregation is a key
feature, because not all groups followed the same logic to "chunk"
their files. Some did it per year, others per 2 years, others per 13
years, other cut their files in November!<br>
<br>
What we do is to *locally* aggregate those files (thredds
aggregation), make them available from an opendap endpoint, mount an
ethernet interface over infiniband for this endpoint.<br>
<br>
Result is a filesystem performance access level to an opendap
aggregation endpoint, and not *just* a network performance access
level. You don't have to care about chunking, you don't have to
create physical aggregated files (with nco and such) and you have
high performance ===> heavy time saving.<br>
<br>
Regards.<br>
Sébastien<br>
<br>
<br>
<blockquote
cite="mid:10DAFC25-1766-41FB-A8F2-79A79F2E34DB@cola.iges.org"
type="cite">
<div><br>
</div>
<div><br>
</div>
<div>
<blockquote type="cite">
<div bgcolor="#FFFFFF" text="#000000"> <br>
- Steve<br>
<br>
==============================<br>
<br>
On 12/14/2011 9:38 AM, Jennifer Adams wrote:
<blockquote
cite="mid:73E2239C-9CDF-47C2-8C0A-F6AF4A6BAB35@cola.iges.org"
type="cite">Well, after working from the client side to
get CMIP3 and CMIP5 data, I can say that wget is a fine
tool to rely on at the core of the workflow.
Unfortunately, the step up in complexity from CMIP3 to
CMIP5 and the switch from FTP to HTTP trashed the elegant
use of wget. No amount of customized wrapper software,
browser interfaces, or pre-packaged tools like DML fixes
that problem.
<div><br>
</div>
<div>At the moment, the burden on the user is
embarrassingly high. It's so easy to suggest that the
user should "filter to remove what is not required" from
a downloaded script, but the actual pratice of doing
that in a timely and automated and distributed way is
NOT simple! And if the solution to my problem of filling
in the gaps in my incomplete collection is to go back to
clicking in my browser and do the whole thing over again
but make my filters smarter by looking for what's
already been acquired or what has a new version number …
this is unacceptable. The filtering must be a
server-side responsibility and the interface must be
accessible by automated scripts. Make it so! </div>
<div><br>
</div>
<div>By the way, the version number is a piece of metadata
that is not in the downloaded files or the gateway's
search criteria. It appears in the wget script as part
of the path in the file's http location, but the path is
not preserved after the wget is complete, so it is
effectively lost after the download is done. I guess the
file's date stamp would be the only way to know if the
version number of the data file in question has been
changed, but I'm not going to write that check into my
filtering scripts. </div>
<div><br>
</div>
<div>--Jennifer</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>
<div>
<div>
<div apple-content-edited="true"> <span
class="Apple-style-span"
style="border-collapse: separate; font-family:
Helvetica; font-size: 12px; font-style:
normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height:
normal; text-indent: 0px; text-transform:
none; orphans: 2; white-space: normal; widows:
2; word-spacing: 0px;"><span
class="Apple-style-span"
style="border-collapse: separate;
font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal;
line-height: normal; text-indent: 0px;
text-transform: none; orphans: 2;
white-space: normal; widows: 2;
word-spacing: 0px;"><span
class="Apple-style-span"
style="border-collapse: separate;
font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing:
normal; line-height: normal; text-indent:
0px; text-transform: none; orphans: 2;
white-space: normal; widows: 2;
word-spacing: 0px;">
<div>--</div>
<div>Jennifer M. Adams</div>
<div>IGES/COLA</div>
<div>4041 Powder Mill Road, Suite 302</div>
<div>Calverton, MD 20705</div>
<div><a moz-do-not-send="true"
href="mailto:jma@cola.iges.org">jma@cola.iges.org</a></div>
<div><br class="khtml-block-placeholder">
</div>
<br class="Apple-interchange-newline">
</span></span></span> </div>
<br>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
GO-ESSP-TECH mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
</div>
</blockquote>
</div>
<br>
<div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate;
border-spacing: 0px; color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal; font-variant:
normal; font-weight: normal; letter-spacing: normal;
line-height: normal; text-indent: 0px; text-transform: none;
orphans: 2; white-space: normal; widows: 2; word-spacing:
0px;"><span class="Apple-style-span" style="border-collapse:
separate; border-spacing: 0px; color: rgb(0, 0, 0);
font-family: Helvetica; font-size: 12px; font-style: normal;
font-variant: normal; font-weight: normal; letter-spacing:
normal; line-height: normal; text-indent: 0px;
text-transform: none; orphans: 2; white-space: normal;
widows: 2; word-spacing: 0px;"><span
class="Apple-style-span" style="border-collapse: separate;
border-spacing: 0px; color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant: normal; font-weight: normal; letter-spacing:
normal; line-height: normal; text-indent: 0px;
text-transform: none; orphans: 2; white-space: normal;
widows: 2; word-spacing: 0px;">
<div>--</div>
<div>Jennifer M. Adams</div>
<div>IGES/COLA</div>
<div>4041 Powder Mill Road, Suite 302</div>
<div>Calverton, MD 20705</div>
<div><a moz-do-not-send="true"
href="mailto:jma@cola.iges.org">jma@cola.iges.org</a></div>
<div><br class="khtml-block-placeholder">
</div>
<br class="Apple-interchange-newline">
</span></span></span>
</div>
<br>
<pre wrap="">
<fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Sébastien Denvil
IPSL, Pôle de modélisation du climat
UPMC, Case 101, 4 place Jussieu,
75252 Paris Cedex 5
Tour 45-55 2ème étage Bureau 209
Tel: 33 1 44 27 21 10
Fax: 33 1 44 27 39 02
</pre>
</body>
</html>