[Go-essp-tech] Status of Gateway 2.0 (another use case)

Wed Dec 14 14:25:50 MST 2011

  On 14/12/2011 21:04, Jennifer Adams wrote:
>
> On Dec 14, 2011, at 12:51 PM, Steve Hankin wrote:
>
>> Hi Jennifer,
>>
>> I imagine that I am speaking for everyone in ESG in saying that your 
>> clear and constructive comments have been MOST HELPFUL!
>>
>> A question for you:  What do you see as the potential contributions 
>> of OPeNDAP access to CMIP datasets? -- both for yourself and the 
>> users that you represent?
>
> I don't see OPeNDAP access solving any of the problems we're facing 
> now. COLA users need this data on our local high-speed file system, 
> where we can carve it up and aggregate it and throw all our analysis 
> methods at it and compare it to other data sets without worrying about 
> authenitcation, internet bandwidth, or the status of all the 
> distributed servers.
> --Jennifer
>

We have the same requirements as above. Jennifer is right stating that 
opendap won't help on the production datanode side (except for some 
quick look here and there).

But I just want to emphasis that opendap aggregation is a key feature, 
because not all groups followed the same logic to "chunk" their files. 
Some did it per year, others per 2 years, others per 13 years, other cut 
their files in November!

What we do is to *locally* aggregate those files (thredds aggregation), 
make them available from an opendap endpoint, mount an ethernet 
interface over infiniband for this endpoint.

Result is a filesystem performance access level to an opendap 
aggregation endpoint, and not *just* a network performance access level. 
You don't have to care about chunking, you don't have to create physical 
aggregated files (with nco and such) and you have high performance ===> 
heavy time saving.

Regards.
Sébastien

>
>
>>
>>     - Steve
>>
>> ==============================
>>
>> On 12/14/2011 9:38 AM, Jennifer Adams wrote:
>>> Well, after working from the client side to get CMIP3 and CMIP5 
>>> data, I can say that wget is a fine tool to rely on at the core of 
>>> the workflow. Unfortunately, the step up in complexity from CMIP3 to 
>>> CMIP5 and the switch from FTP to HTTP trashed the elegant use of 
>>> wget. No amount of customized wrapper software, browser interfaces, 
>>> or pre-packaged tools like DML fixes that problem.
>>>
>>> At the moment, the burden on the user is embarrassingly high. It's 
>>> so easy to suggest that the user should "filter to remove what is 
>>> not required" from a downloaded script, but the actual pratice of 
>>> doing that in a timely and automated and distributed way is NOT 
>>> simple! And if the solution to my problem of filling in the gaps in 
>>> my incomplete collection is to go back to clicking in my browser and 
>>> do the whole thing over again but make my filters smarter by looking 
>>> for what's already been acquired or what has a new version number … 
>>> this is unacceptable. The filtering must be a server-side 
>>> responsibility and the interface must be accessible by automated 
>>> scripts. Make it so!
>>>
>>> By the way, the version number is a piece of metadata that is not in 
>>> the downloaded files or the gateway's search criteria. It appears in 
>>> the wget script as part of the path in the file's http location, but 
>>> the path is not preserved after the wget is complete, so it is 
>>> effectively lost after the download is done. I guess the file's date 
>>> stamp would be the only way to know if the version number of the 
>>> data file in question has been changed, but I'm not going to write 
>>> that check into my filtering scripts.
>>>
>>> --Jennifer
>>>
>>>
>>> --
>>> Jennifer M. Adams
>>> IGES/COLA
>>> 4041 Powder Mill Road, Suite 302
>>> Calverton, MD 20705
>>> jma at cola.iges.org <mailto:jma at cola.iges.org>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
> --
> Jennifer M. Adams
> IGES/COLA
> 4041 Powder Mill Road, Suite 302
> Calverton, MD 20705
> jma at cola.iges.org <mailto:jma at cola.iges.org>
>
>
>
>
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

-- 
Sébastien Denvil
IPSL, Pôle de modélisation du climat
UPMC, Case 101, 4 place Jussieu,
75252 Paris Cedex 5

Tour 45-55 2ème étage Bureau 209
Tel: 33 1 44 27 21 10
Fax: 33 1 44 27 39 02

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20111214/4fc783bc/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4172 bytes
Desc: S/MIME Cryptographic Signature
Url : http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20111214/4fc783bc/attachment-0001.bin 