[Go-essp-tech] Status of Gateway 2.0 (another use case)
Sébastien Denvil
sebastien.denvil at ipsl.jussieu.fr
Wed Dec 14 14:25:50 MST 2011
On 14/12/2011 21:04, Jennifer Adams wrote:
>
> On Dec 14, 2011, at 12:51 PM, Steve Hankin wrote:
>
>> Hi Jennifer,
>>
>> I imagine that I am speaking for everyone in ESG in saying that your
>> clear and constructive comments have been MOST HELPFUL!
>>
>> A question for you: What do you see as the potential contributions
>> of OPeNDAP access to CMIP datasets? -- both for yourself and the
>> users that you represent?
>
> I don't see OPeNDAP access solving any of the problems we're facing
> now. COLA users need this data on our local high-speed file system,
> where we can carve it up and aggregate it and throw all our analysis
> methods at it and compare it to other data sets without worrying about
> authenitcation, internet bandwidth, or the status of all the
> distributed servers.
> --Jennifer
>
We have the same requirements as above. Jennifer is right stating that
opendap won't help on the production datanode side (except for some
quick look here and there).
But I just want to emphasis that opendap aggregation is a key feature,
because not all groups followed the same logic to "chunk" their files.
Some did it per year, others per 2 years, others per 13 years, other cut
their files in November!
What we do is to *locally* aggregate those files (thredds aggregation),
make them available from an opendap endpoint, mount an ethernet
interface over infiniband for this endpoint.
Result is a filesystem performance access level to an opendap
aggregation endpoint, and not *just* a network performance access level.
You don't have to care about chunking, you don't have to create physical
aggregated files (with nco and such) and you have high performance ===>
heavy time saving.
Regards.
Sébastien
>
>
>>
>> - Steve
>>
>> ==============================
>>
>> On 12/14/2011 9:38 AM, Jennifer Adams wrote:
>>> Well, after working from the client side to get CMIP3 and CMIP5
>>> data, I can say that wget is a fine tool to rely on at the core of
>>> the workflow. Unfortunately, the step up in complexity from CMIP3 to
>>> CMIP5 and the switch from FTP to HTTP trashed the elegant use of
>>> wget. No amount of customized wrapper software, browser interfaces,
>>> or pre-packaged tools like DML fixes that problem.
>>>
>>> At the moment, the burden on the user is embarrassingly high. It's
>>> so easy to suggest that the user should "filter to remove what is
>>> not required" from a downloaded script, but the actual pratice of
>>> doing that in a timely and automated and distributed way is NOT
>>> simple! And if the solution to my problem of filling in the gaps in
>>> my incomplete collection is to go back to clicking in my browser and
>>> do the whole thing over again but make my filters smarter by looking
>>> for what's already been acquired or what has a new version number …
>>> this is unacceptable. The filtering must be a server-side
>>> responsibility and the interface must be accessible by automated
>>> scripts. Make it so!
>>>
>>> By the way, the version number is a piece of metadata that is not in
>>> the downloaded files or the gateway's search criteria. It appears in
>>> the wget script as part of the path in the file's http location, but
>>> the path is not preserved after the wget is complete, so it is
>>> effectively lost after the download is done. I guess the file's date
>>> stamp would be the only way to know if the version number of the
>>> data file in question has been changed, but I'm not going to write
>>> that check into my filtering scripts.
>>>
>>> --Jennifer
>>>
>>>
>>> --
>>> Jennifer M. Adams
>>> IGES/COLA
>>> 4041 Powder Mill Road, Suite 302
>>> Calverton, MD 20705
>>> jma at cola.iges.org <mailto:jma at cola.iges.org>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
> --
> Jennifer M. Adams
> IGES/COLA
> 4041 Powder Mill Road, Suite 302
> Calverton, MD 20705
> jma at cola.iges.org <mailto:jma at cola.iges.org>
>
>
>
>
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
--
Sébastien Denvil
IPSL, Pôle de modélisation du climat
UPMC, Case 101, 4 place Jussieu,
75252 Paris Cedex 5
Tour 45-55 2ème étage Bureau 209
Tel: 33 1 44 27 21 10
Fax: 33 1 44 27 39 02
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20111214/4fc783bc/attachment-0001.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4172 bytes
Desc: S/MIME Cryptographic Signature
Url : http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20111214/4fc783bc/attachment-0001.bin
More information about the GO-ESSP-TECH
mailing list