[Go-essp-tech] Status of Gateway 2.0 (another use case)
Sébastien Denvil
sebastien.denvil at ipsl.jussieu.fr
Wed Dec 14 14:07:03 MST 2011
Hi Jennifer,
having the same use case we developed a tool to solve that and the fact
that the archive is extremely inhomogeneous. Quoting you:
"Right now, I am diligently downloading CMIP5 data, gathering runs for a
subset of experiments and variables, for all available models and
ensemble members."
It's a light weight client side application written in Python (actually
independent from any gateway). We plan to introduce a minimal loosely
coupling. As you rerun it the code will fill the gap of your collection.
It will preserve DRS structure, version information...
We will release a new version within a few days and will set up a new
web page. But if you want to give a try before then have a look here if
you are interested. We will answer to any question.
http://dods.ipsl.jussieu.fr/jripsl/synchro_data/README
This use-case is mandatory for the CMIP5 archive.
Regards.
Sébastien
On 14/12/2011 18:50, Jennifer Adams wrote:
>
> On Dec 14, 2011, at 12:44 PM, Cinquini, Luca (3880) wrote:
>
>> So Jennifer, would having the capability of doing negative searches
>> (model=!CCSM), and generate the corresponding wget scripts, help you ?
> Yes, I think it would help me. I'm ready to start testing the P2P
> system whenever you are. --Jennifer
>
>
>
>> thanks, Luca
>>
>> On Dec 14, 2011, at 10:38 AM, Jennifer Adams wrote:
>>
>>> Well, after working from the client side to get CMIP3 and CMIP5
>>> data, I can say that wget is a fine tool to rely on at the core of
>>> the workflow. Unfortunately, the step up in complexity from CMIP3 to
>>> CMIP5 and the switch from FTP to HTTP trashed the elegant use of
>>> wget. No amount of customized wrapper software, browser interfaces,
>>> or pre-packaged tools like DML fixes that problem.
>>>
>>> At the moment, the burden on the user is embarrassingly high. It's
>>> so easy to suggest that the user should "filter to remove what is
>>> not required" from a downloaded script, but the actual pratice of
>>> doing that in a timely and automated and distributed way is NOT
>>> simple! And if the solution to my problem of filling in the gaps in
>>> my incomplete collection is to go back to clicking in my browser and
>>> do the whole thing over again but make my filters smarter by looking
>>> for what's already been acquired or what has a new version number …
>>> this is unacceptable. The filtering must be a server-side
>>> responsibility and the interface must be accessible by automated
>>> scripts. Make it so!
>>>
>>> By the way, the version number is a piece of metadata that is not in
>>> the downloaded files or the gateway's search criteria. It appears in
>>> the wget script as part of the path in the file's http location, but
>>> the path is not preserved after the wget is complete, so it is
>>> effectively lost after the download is done. I guess the file's date
>>> stamp would be the only way to know if the version number of the
>>> data file in question has been changed, but I'm not going to write
>>> that check into my filtering scripts.
>>>
>>> --Jennifer
>>>
>>>
>>> --
>>> Jennifer M. Adams
>>> IGES/COLA
>>> 4041 Powder Mill Road, Suite 302
>>> Calverton, MD 20705
>>> jma at cola.iges.org <mailto:jma at cola.iges.org>
>>>
>>>
>>>
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu <mailto:GO-ESSP-TECH at ucar.edu>
>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>
>
> --
> Jennifer M. Adams
> IGES/COLA
> 4041 Powder Mill Road, Suite 302
> Calverton, MD 20705
> jma at cola.iges.org <mailto:jma at cola.iges.org>
>
>
>
>
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
--
Sébastien Denvil
IPSL, Pôle de modélisation du climat
UPMC, Case 101, 4 place Jussieu,
75252 Paris Cedex 5
Tour 45-55 2ème étage Bureau 209
Tel: 33 1 44 27 21 10
Fax: 33 1 44 27 39 02
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20111214/ce4dfa32/attachment.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4172 bytes
Desc: S/MIME Cryptographic Signature
Url : http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20111214/ce4dfa32/attachment.bin
More information about the GO-ESSP-TECH
mailing list