[Go-essp-tech] Status of Gateway 2.0 (another use case)

[Cinquini, Luca (3880)]

So Jennifer, would having the capability of doing negative searches (model=!CCSM), and generate the corresponding wget scripts, help you ?
thanks, Luca

On Dec 14, 2011, at 10:38 AM, Jennifer Adams wrote:

Well, after working from the client side to get CMIP3 and CMIP5 data, I can say that wget is a fine tool to rely on at the core of the workflow. Unfortunately, the step up in complexity from CMIP3 to CMIP5 and the switch from FTP to HTTP trashed the elegant use of wget. No amount of customized wrapper software, browser interfaces, or pre-packaged tools like DML fixes that problem.

At the moment, the burden on the user is embarrassingly high. It's so easy to suggest that the user should "filter to remove what is not required" from a downloaded script, but the actual pratice of doing that in a timely and automated and distributed way is NOT simple! And if the solution to my problem of filling in the gaps in my incomplete collection is to go back to clicking in my browser and do the whole thing over again but make my filters smarter by looking for what's already been acquired or what has a new version number … this is unacceptable. The filtering must be a server-side responsibility and the interface must be accessible by automated scripts. Make it so!

By the way, the version number is a piece of metadata that is not in the downloaded files or the gateway's search criteria. It appears in the wget script as part of the path in the file's http location, but the path is not preserved after the wget is complete, so it is effectively lost after the download is done. I guess the file's date stamp would be the only way to know if the version number of the data file in question has been changed, but I'm not going to write that check into my filtering scripts.

--Jennifer


--
Jennifer M. Adams
IGES/COLA
4041 Powder Mill Road, Suite 302
Calverton, MD 20705
jma at cola.iges.org<mailto:jma at cola.iges.org>



_______________________________________________
GO-ESSP-TECH mailing list
GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20111214/211aa115/attachment-0001.html