[Go-essp-tech] Status of Gateway 2.0

[Jennifer Adams]

On Dec 5, 2011, at 5:35 PM, Eric Nienhouse wrote:

> Hi Jennifer,
> 
> Jennifer Adams wrote:
>> Dear Colleagues, I have been searching for a way to automate the CMIP5 download process -- as many have previously noted, the current gateway interface is fine for testing and poking, but for high-volume downloading it scales to impossible amounts of time and effort. I was encouraged by the Gateway 2.0 beta site at NCAR:      http://search-esg.prototype.ucar.edu <http://search-esg.prototype.ucar.edu/>
>> This quickly gives me what looks like a single wget script to grab all models and ensembles for a specified experiment/realm/frequency/variable. Excellent!
> Great to hear and thanks for the feedback.  Yes, a single wget script can be generated for a specified experiment/realm/frequency/variable.  Does this model work for your CMIP5 download use case(s)?
Yes, I am fielding requests for data sets based on experiment, realm, frequency and variable. We may also want to narrow down the number of models, but being able to lump all members into one wget script is a big time saver.

>  Are there other use scenarios you need or would like to see?
I will think about that, poke at the prototype some more, and let you know if I come up with anything.

>> 
>> Q1: Will the version 2.0 wget scripts be using certificates so they don't expire?
> Using certificates should be easier in the future due to improvements in the script.  The certificates will still expire, though, typically after 72 hours.  We're working on getting all certificates in the ESG federation to be a lifetime of 72 hours which will help.  There are still a few services creating them that issue 12 hour lifetimes.
Renewing the certificates is no problem. What I never want to see again are the authorization tokens that expire after a couple of hours and render the entire wget script useless. In most cases, my tokens would expire before the wget script could even finish the download, very frustrating when the darned thing was so costly to acquire in the first place. 

>> 
>> Q2: When is the version 2.0 gateway going to become operational?
> We're planning to deploy the 2.0 gateway in production at NCAR as soon as possible and we're testing the 2.0 version heavily right now.  Getting this version ready for use is our top priority.  I'll let you know once it's deployed and you can try it out.
OK. Thanks for the encouraging update. Will it work for all CMIP5 data sets? Right now the prototype finds 20115 results for a search with  'project=cmip5'  and the operational gateway finds 25463. 

--Jennifer

> 
> Thanks,
> 
> -Eric
>> 
>> There's no point in wasting time with the current Gateway if the new one is around the corner.
>> Jennifer
>> 
>> --
>> Jennifer M. Adams
>> IGES/COLA
>> 4041 Powder Mill Road, Suite 302
>> Calverton, MD 20705
>> jma at cola.iges.org <mailto:jma at cola.iges.org>
>> 
>> 
>> 
>> 
>> ------------------------------------------------------------------------
>> 
>> _______________________________________________
>> GO-ESSP-TECH mailing list
>> GO-ESSP-TECH at ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>  
> 

--
Jennifer M. Adams
IGES/COLA
4041 Powder Mill Road, Suite 302
Calverton, MD 20705
jma at cola.iges.org



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20111205/7577a514/attachment.html