<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#ffffff">
Hi Jennifer,<br>
<br>
having the same use case we developed a tool to solve that and the
fact that the archive is extremely inhomogeneous. Quoting you:<br>
"Right now, I am diligently downloading CMIP5 data, gathering runs
for a subset of experiments and variables, for all available models
and ensemble members."<br>
<br>
It's a light weight client side application written in Python
(actually independent from any gateway). We plan to introduce a
minimal loosely coupling. As you rerun it the code will fill the gap
of your collection. It will preserve DRS structure, version
information...<br>
<br>
We will release a new version within a few days and will set up a
new web page. But if you want to give a try before then have a look
here if you are interested. We will answer to any question.
<a class="moz-txt-link-freetext" href="http://dods.ipsl.jussieu.fr/jripsl/synchro_data/README">http://dods.ipsl.jussieu.fr/jripsl/synchro_data/README</a><br>
<br>
This use-case is mandatory for the CMIP5 archive.<br>
<br>
Regards.<br>
Sébastien<br>
<br>
On 14/12/2011 18:50, Jennifer Adams wrote:
<blockquote
cite="mid:EE75DE5F-C7E6-4A05-94DC-65D26A945BBC@cola.iges.org"
type="cite"><br>
<div>
<div>On Dec 14, 2011, at 12:44 PM, Cinquini, Luca (3880) wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div style="word-wrap: break-word;">So Jennifer, would having
the capability of doing negative searches (model=!CCSM), and
generate the corresponding wget scripts, help you ?</div>
</blockquote>
Yes, I think it would help me. I'm ready to start testing the
P2P system whenever you are. --Jennifer</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
<blockquote type="cite">
<div style="word-wrap: break-word;">
<div>thanks, Luca</div>
<div><br>
<div>
<div>On Dec 14, 2011, at 10:38 AM, Jennifer Adams wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div style="word-wrap: break-word;">Well, after
working from the client side to get CMIP3 and CMIP5
data, I can say that wget is a fine tool to rely on
at the core of the workflow. Unfortunately, the step
up in complexity from CMIP3 to CMIP5 and the switch
from FTP to HTTP trashed the elegant use of wget. No
amount of customized wrapper software, browser
interfaces, or pre-packaged tools like DML fixes
that problem.
<div><br>
</div>
<div>At the moment, the burden on the user is
embarrassingly high. It's so easy to suggest that
the user should "filter to remove what is not
required" from a downloaded script, but the actual
pratice of doing that in a timely and automated
and distributed way is NOT simple! And if the
solution to my problem of filling in the gaps in
my incomplete collection is to go back to clicking
in my browser and do the whole thing over again
but make my filters smarter by looking for what's
already been acquired or what has a new version
number … this is unacceptable. The filtering must
be a server-side responsibility and the interface
must be accessible by automated scripts. Make it
so! </div>
<div><br>
</div>
<div>By the way, the version number is a piece of
metadata that is not in the downloaded files or
the gateway's search criteria. It appears in the
wget script as part of the path in the file's http
location, but the path is not preserved after the
wget is complete, so it is effectively lost after
the download is done. I guess the file's date
stamp would be the only way to know if the version
number of the data file in question has been
changed, but I'm not going to write that check
into my filtering scripts. </div>
<div><br>
</div>
<div>--Jennifer</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>
<div>
<div>
<div apple-content-edited="true">
<span class="Apple-style-span"
style="border-collapse: separate;
font-family: Helvetica; font-size: 12px;
font-style: normal; font-variant:
normal; font-weight: normal;
letter-spacing: normal; line-height:
normal; text-indent: 0px;
text-transform: none; orphans: 2;
white-space: normal; widows: 2;
word-spacing: 0px;"><span
class="Apple-style-span"
style="border-collapse: separate;
font-family: Helvetica; font-size:
12px; font-style: normal;
font-variant: normal; font-weight:
normal; letter-spacing: normal;
line-height: normal; text-indent: 0px;
text-transform: none; orphans: 2;
white-space: normal; widows: 2;
word-spacing: 0px;"><span
class="Apple-style-span"
style="border-collapse: separate;
font-family: Helvetica; font-size:
12px; font-style: normal;
font-variant: normal; font-weight:
normal; letter-spacing: normal;
line-height: normal; text-indent:
0px; text-transform: none; orphans:
2; white-space: normal; widows: 2;
word-spacing: 0px;">
<div>--</div>
<div>Jennifer M. Adams</div>
<div>IGES/COLA</div>
<div>4041 Powder Mill Road, Suite
302</div>
<div>Calverton, MD 20705</div>
<div><a moz-do-not-send="true"
href="mailto:jma@cola.iges.org">jma@cola.iges.org</a></div>
<div><br
class="khtml-block-placeholder">
</div>
<br
class="Apple-interchange-newline">
</span></span></span>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
GO-ESSP-TECH mailing list<br>
<a moz-do-not-send="true"
href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a><br>
<a moz-do-not-send="true"
href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
<div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate;
border-spacing: 0px; color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal; font-variant:
normal; font-weight: normal; letter-spacing: normal;
line-height: normal; text-indent: 0px; text-transform: none;
orphans: 2; white-space: normal; widows: 2; word-spacing:
0px;"><span class="Apple-style-span" style="border-collapse:
separate; border-spacing: 0px; color: rgb(0, 0, 0);
font-family: Helvetica; font-size: 12px; font-style: normal;
font-variant: normal; font-weight: normal; letter-spacing:
normal; line-height: normal; text-indent: 0px;
text-transform: none; orphans: 2; white-space: normal;
widows: 2; word-spacing: 0px;"><span
class="Apple-style-span" style="border-collapse: separate;
border-spacing: 0px; color: rgb(0, 0, 0); font-family:
Helvetica; font-size: 12px; font-style: normal;
font-variant: normal; font-weight: normal; letter-spacing:
normal; line-height: normal; text-indent: 0px;
text-transform: none; orphans: 2; white-space: normal;
widows: 2; word-spacing: 0px;">
<div>--</div>
<div>Jennifer M. Adams</div>
<div>IGES/COLA</div>
<div>4041 Powder Mill Road, Suite 302</div>
<div>Calverton, MD 20705</div>
<div><a moz-do-not-send="true"
href="mailto:jma@cola.iges.org">jma@cola.iges.org</a></div>
<div><br class="khtml-block-placeholder">
</div>
<br class="Apple-interchange-newline">
</span></span></span>
</div>
<br>
<pre wrap="">
<fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Sébastien Denvil
IPSL, Pôle de modélisation du climat
UPMC, Case 101, 4 place Jussieu,
75252 Paris Cedex 5
Tour 45-55 2ème étage Bureau 209
Tel: 33 1 44 27 21 10
Fax: 33 1 44 27 39 02
</pre>
</body>
</html>