<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi Jennifer,<div><span class="Apple-tab-span" style="white-space:pre"> </span>although I totally agree with Estani, that we should build more intelligence in the client side, developing a "negative" syntax for the P2P api should not be difficult... </div><div><br></div><div>Because of the HTTP URL syntax, it would need to be something like:</div><div><br></div><div>&model=!HadCM3&model=!CCSM</div><div><br></div><div>(i.e. search all models that are not HadCM# and not CCSM).</div><div><br></div><div>I'll put on our to-do...</div><div><br></div><div>thanks, Luca</div><div><br><div><div>On Dec 14, 2011, at 8:05 AM, Estanislao Gonzalez wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div bgcolor="#FFFFFF" text="#000000">
Hi Jennifer,<br>
<br>
AFAIK no, it's not possible to negate values, but that doesn't seem
to correspond with the definition of the use-case you've presented.<br>
AFAICT the user would just issue the same search and new files would
be added (what about changed ones and those deleted from the server
because of new versions?)<br>
<br>
This all might be possible to implement, but before people start
coding like frenzy, I think there are a couple of things that should
be said:<br>
1) I don't think any system should provide for every single
use-case. CMIP5 is nothing like CMIP3; it of course about the amount
of data (impossible to store centrally, both for storage and
throughput constrains) but also the federative aspect (that allows
every institution some freedom on how to provide the data and forces
them to provide also the know-how for this) and the quality
assurance (data changes a lot until it gets verified).<br>
2) The main idea behind the P2P, and to where we all want the
Gateway 2.0 to go, is modularity. I think this is a required
architectural decision we need to embrace. So I don't think it's a
good idea to extend the Server side to provide a functionality
typically available at the client, like download management.<br>
<br>
So, the way I picture this is:<br>
1) get the list of files to be downloaded (in the wget script or by
any other means)<br>
2) filter that to remove what is not required<br>
<br>
Better yet, rely on a download manager for this like the DML or
extend it if it's not providing this. The key here is the ability
to:<br>
1) Know when something changed (we have many more users than CMIP3
and it exceeds the climate community, so the less load the servers
have, the better).<br>
For this both systems has some kind of atom feed<br>
2) Do something with this information. This is user dependent. Some
will want it to be kept "synchronized" gathering always the latest
version, some will want to have the multiple versions so they can
compare them, some might just want to be notified and other won't
care at all :-)<br>
<br>
I think 2) is not really implemented, although some use cases are
already supported in P2P (not sure about Gateway 2.0). <br>
So you could add a cronjob with this:<br>
bash <(wget
<a class="moz-txt-link-freetext" href="http://p2pnode/wgetscript?experiment=decadal1960&realm=atmos&time_frequency=month&variable=clt">http://p2pnode/wgetscript?experiment=decadal1960&realm=atmos&time_frequency=month&variable=clt</a>
-qO - | grep -v HadCM3)<br>
and as soon as something changes you would have your new file
(everything but HadCM3 as you pointed out in your example)<br>
<br>
But there are a lot of cave-eats there and I don't think we want to
encourage users to "misbehave" so we need some intelligence that
doesn't try to download thousands of file if not required, uses
multi-threading in a useful manner, handle certificate expiration,
etc.<br>
<br>
I think we should improve the client-side of the CMIP5 system.<br>
<br>
My 2c,<br>
Estani<br>
<br>
Am 14.12.2011 15:30, schrieb Jennifer Adams:
<blockquote cite="mid:982953F4-1AA4-4D25-BEA5-737AD43F293D@cola.iges.org" type="cite">Dear Colleagues,
<div>I have thought of another use case for the Gateway 2.0 and
P2P developers to consider. </div>
<div><br>
</div>
<div>Right now, I am diligently downloading CMIP5 data, gathering
runs for a subset of experiements and variables, for all
available models and ensemble members. I realize that not all
modeling centers have submitted their data yet, so in a few
months time, I will have to go back and fill in the gaps in my
collection. At that point, I will have a list of
experiments/variables/models/ensembles that I have already
acquired, and I will need to search the available CMIP5 data for
runs that I do not have. </div>
<div><br>
</div>
<div>I know it is unfair to compare to the good old days of using
wget to grab CMIP3 data via FTP from <a moz-do-not-send="true" href="http://ftp-esg.ucllnl.org/">ftp-esg.ucllnl.org</a>, but
this particular use case used to be so easy … I would just rerun
my single wget command and it would fill in the missing files in
my collection automatically. </div>
<div><br>
</div>
<div>Is there a syntax in the Gateway 2.0 and P2P search URLs that
allows for a NOT? </div>
<div><a moz-do-not-send="true" href="http://esg-datanode.jpl.nasa.gov/esg-search/wget?experiment=decadal1960&realm=atmos&time_frequency=month&variable=clt">http://esg-datanode.jpl.nasa.gov/esg-search/wget?experiment=decadal1960&realm=atmos&time_frequency=month&variable=clt</a>&model<font class="Apple-style-span" color="#ff2700">!=</font>HadCM3</div>
<div><br>
</div>
<div>Any better ideas or plans for how to do this? </div>
<div><br>
</div>
<div>--Jennifer</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
<div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; -webkit-text-decorations-in-effect: none; text-indent: 0px; -webkit-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><span class="Apple-style-span" style="border-collapse: separate; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; -webkit-text-decorations-in-effect: none; text-indent: 0px; -webkit-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><span class="Apple-style-span" style="border-collapse: separate; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; -webkit-text-decorations-in-effect: none; text-indent: 0px; -webkit-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; ">
<div>--</div>
<div>Jennifer M. Adams</div>
<div>IGES/COLA</div>
<div>4041 Powder Mill Road, Suite 302</div>
<div>Calverton, MD 20705</div>
<div><a moz-do-not-send="true" href="mailto:jma@cola.iges.org">jma@cola.iges.org</a></div>
<div><br class="khtml-block-placeholder">
</div>
<br class="Apple-interchange-newline">
</span></span></span>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
GO-ESSP-TECH mailing list
<a class="moz-txt-link-abbreviated" href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a>
<a class="moz-txt-link-freetext" href="http://mailman.ucar.edu/mailman/listinfo/go-essp-tech">http://mailman.ucar.edu/mailman/listinfo/go-essp-tech</a>
</pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Estanislao Gonzalez
Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: <a class="moz-txt-link-abbreviated" href="mailto:gonzalez@dkrz.de">gonzalez@dkrz.de</a> </pre>
</div>
_______________________________________________<br>GO-ESSP-TECH mailing list<br><a href="mailto:GO-ESSP-TECH@ucar.edu">GO-ESSP-TECH@ucar.edu</a><br>http://mailman.ucar.edu/mailman/listinfo/go-essp-tech<br></blockquote></div><br></div></body></html>