[Go-essp-tech] Status of Gateway 2.0 (another use case)

[Sébastien Denvil]

  Hi Jennifer,

having the same use case we developed a tool to solve that and the fact 
that the archive is extremely inhomogeneous. Quoting you:
"Right now, I am diligently downloading CMIP5 data, gathering runs for a 
subset of experiments and variables, for all available models and 
ensemble members."

It's a light weight client side application written in Python (actually 
independent from any gateway). We plan to introduce a minimal loosely 
coupling. As you rerun it the code will fill the gap of your collection. 
It will preserve DRS structure, version information...

We will release a new version within a few days and will set up a new 
web page. But if you want to give a try before then have a look here if 
you are interested. We will answer to any question. 
http://dods.ipsl.jussieu.fr/jripsl/synchro_data/README

This use-case is mandatory for the CMIP5 archive.

Regards.
Sébastien

On 14/12/2011 18:50, Jennifer Adams wrote:
>
> On Dec 14, 2011, at 12:44 PM, Cinquini, Luca (3880) wrote:
>
>> So Jennifer, would having the capability of doing negative searches 
>> (model=!CCSM), and generate the corresponding wget scripts, help you ?
> Yes, I think it would help me. I'm ready to start testing the P2P 
> system whenever you are. --Jennifer
>
>
>
>> thanks, Luca
>>
>> On Dec 14, 2011, at 10:38 AM, Jennifer Adams wrote:
>>
>>> Well, after working from the client side to get CMIP3 and CMIP5 
>>> data, I can say that wget is a fine tool to rely on at the core of 
>>> the workflow. Unfortunately, the step up in complexity from CMIP3 to 
>>> CMIP5 and the switch from FTP to HTTP trashed the elegant use of 
>>> wget. No amount of customized wrapper software, browser interfaces, 
>>> or pre-packaged tools like DML fixes that problem.
>>>
>>> At the moment, the burden on the user is embarrassingly high. It's 
>>> so easy to suggest that the user should "filter to remove what is 
>>> not required" from a downloaded script, but the actual pratice of 
>>> doing that in a timely and automated and distributed way is NOT 
>>> simple! And if the solution to my problem of filling in the gaps in 
>>> my incomplete collection is to go back to clicking in my browser and 
>>> do the whole thing over again but make my filters smarter by looking 
>>> for what's already been acquired or what has a new version number … 
>>> this is unacceptable. The filtering must be a server-side 
>>> responsibility and the interface must be accessible by automated 
>>> scripts. Make it so!
>>>
>>> By the way, the version number is a piece of metadata that is not in 
>>> the downloaded files or the gateway's search criteria. It appears in 
>>> the wget script as part of the path in the file's http location, but 
>>> the path is not preserved after the wget is complete, so it is 
>>> effectively lost after the download is done. I guess the file's date 
>>> stamp would be the only way to know if the version number of the 
>>> data file in question has been changed, but I'm not going to write 
>>> that check into my filtering scripts.
>>>
>>> --Jennifer
>>>
>>>
>>> --
>>> Jennifer M. Adams
>>> IGES/COLA
>>> 4041 Powder Mill Road, Suite 302
>>> Calverton, MD 20705
>>> jma at cola.iges.org <mailto:jma at cola.iges.org>
>>>
>>>
>>>
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu <mailto:GO-ESSP-TECH at ucar.edu>
>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>
>
> --
> Jennifer M. Adams
> IGES/COLA
> 4041 Powder Mill Road, Suite 302
> Calverton, MD 20705
> jma at cola.iges.org <mailto:jma at cola.iges.org>
>
>
>
>
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech


-- 
Sébastien Denvil
IPSL, Pôle de modélisation du climat
UPMC, Case 101, 4 place Jussieu,
75252 Paris Cedex 5

Tour 45-55 2ème étage Bureau 209
Tel: 33 1 44 27 21 10
Fax: 33 1 44 27 39 02

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20111214/ce4dfa32/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4172 bytes
Desc: S/MIME Cryptographic Signature
Url : http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20111214/ce4dfa32/attachment.bin