[Go-essp-tech] Status of Gateway 2.0

Tue Dec 13 08:54:57 MST 2011

Karl, thanks for following up on this. Jennifer, Eric spoke correctly last week: at NCAR we are in the process of moving 2.0 into local production as a priority next step. I think this will be a big improvement for the scientific community, and I hope we can deploy it more broadly for the same reason.

There was discussion at the end of last week on having a PI level meeting but I haven't heard any more on that so far. It would helpful to have some more open, community discussion of how we move forward; across the federation there are a lot of investments, stakeholders, and resource issues to consider. We would also like to offer a walk-through of 2.0. There's not a lot of time left before the holidays, but we do have next Tuesday's GO-ESSP slot. If you're available and interested, please let us know, and Eric & Nate will put together a tour.

cheers - don

On Dec 7, 2011, at 1:01 PM, Karl Taylor wrote:

> Dear Jennifer and Jean-Yves,
> 
> I apologize that there has been some confusion perpetrated concerning the planned paths for ESG improvement.  For CMIP5 we expect to clear this up by the end of next week when we will post a clear statement outlining our plans.  
> 
> Thanks again for providing feedback that will improve the system.
> 
> Best regards,
> Karl
> 
> On 12/7/11 9:10 AM, Jennifer Adams wrote:
>> 
>> Thank you, Karl, for a very helpful straight answer. One thing I don't understand: Eric Neinhouse's statement that 'getting Gateway 2.0 in production at NCAR is a top priority' seems to be in direct conflict with what you wrote. Will Gateway 2.0 be for non-CMIP5 ESG datasets, or are NCAR and PCMDI taking different approaches? 
>> --Jennifer
>> 
>> On Dec 7, 2011, at 11:29 AM, Karl Taylor wrote:
>> 
>>> Dear Jennifer,
>>> 
>>> Thank you for your valuable input on how to improve the search and scripting capabilities of both the ESG Gateway 2.0 and the ESGF P2P system.  
>>> 
>>> Before you spend too much time attempting to optimize your download of CMIP5 data, you should know that we are very unlikely to actually make Gateway 2.0 operational for CMIP5 model output. Instead we will move directly to P2P.  We believe that with currently available resources PCMDI and the ESG federation cannot support two new systems.  Instead we will keep our current operational system running and bring on line the P2P.  The testing phase of this should happen by the end of the month.  At some point the currently operational system will be turned off and the P2P will become operational.  Therefore, if as a CMIP5 user you plan to further explore the new systems, I suggest your time might be best spent focusing on P2P.
>>> 
>>> P2P and Gateway 2.0 may have similar capabilities at this time, and both are substantially improved over previous versions of the Gateway, but we're convinced that P2P provides a more robust and flexible framework for adding future improvements.   By going directly to P2P, we'll also minimize the inconvenience for users already on the system.
>>> 
>>> Please let me know if you have any concerns about our plans.  We realize the current system continues to be more difficult to use than it should be,  so your feedback and suggestions are invaluable.   Our focus is on getting the improvements offered by P2P in place as soon as possible.  I will let you know if our planned path to a better ESGF delivery system for CMIP5 data changes.  Please feel free to write me directly at taylor13 at llnl.gov
>>> 
>>> best regards,
>>> Karl
>>> 
>>> On 12/7/11 6:14 AM, Jennifer Adams wrote:
>>>> 
>>>> Dear Colleagues,
>>>> Here is more feedback on the prototype of Gateway 2.0. My current use case is searching with a fixed experiment, realm, frequency, and a selected subset of variables. I'm delighted that you can select more than one variable and it will keep adding them to the list of selected results. THANK YOU for listing the variables alphabetically by short name ("Variable Name"); the listing by long name ("CF Variable") is not useful to me, it is not alphabetized, and some names are so long they wrap so it looks like there are three variables called "Flux in the Atmosphere", "Due to Natural", and "Sources". Oops, my bad, it's actually "Surface Carbon Mass Flux into the Atmosphere Due to Natural Sources". Heh. QED. 
>>>> 
>>>> When I'm done clicking on all my desired constraints, I get a URL that looks like this :
>>>> http://search-esg.prototype.ucar.edu/search?Type=Dataset&Experiment=decadal1960&Frequency=mon&Realm=atmos&Variable+Name=clt&Variable+Name=prw
>>>> That is great, because it means I can take a short cut from all the mouse clicking by building a URL that will create my desired collection. 
>>>> 
>>>> I am happy with the search results, so I click on the "Download Selected" button, and get a wget script called "script.wget". 
>>>> 
>>>> 1. It would be great if you could give the script a name that reflects the search constraints used to build that script -- something like
>>>>     wget.cmip5.decadal1960.mon.atmos.clt+prw.sh
>>>> Every time I download a new wget script, I have to rename it to something meaningful (such as the above) because the very next search result will overwrite script.wget, or give me "script(1).wget" instead, and I'll have no simple way to know what data set that wget script is configured to bring back. Can you PLEASE give the wget script a relevant name for me so I can skip this time-consuming step?
>>>> 
>>>> 2. Is there a way to use our favorite tool (wget) or curl to download the wget script so I can bypass the Gateway 2.0 browser interface altogether? My guess is the answer is no, because of the elaborate login process with OpenID and username/password, but it's worth asking… 
>>>> 
>>>> -----------
>>>> 
>>>> As an alternative, I have looked carefully at the wiki page (and its embedded links) on automated generation of wget scripts, http://www.esgf.org/wiki/ESGF_scripting. This is very promising, and Luca assures me that the P2P service will be working in a matter of weeks. 
>>>> 
>>>> I think the syntax of the URL for the same search as my example above would be:
>>>> http://esg-datanode.jpl.nasa.gov/esg-search/wget?experiment=decadal1960&realm=atmos&time_frequency=month&variable=clt,prw
>>>> 
>>>> 1. Is the comma-delimited list of desired variables legitimate syntax? 
>>>> 2. Will it work the way the Gateway 2.0 does to deliver my subset of variables?  
>>>> 3. Will "http://esg-datanode.jpl.nasa.gov" be the only location providing this service? 
>>>> 4. Will the delivered wget script use certificates and checksums?
>>>> 
>>>> Since this isn't fully operational at the moment, I can't say for sure whether this will be the better route for reaching my goal of acquiring CMIP5 data using only scripts and no browser. 
>>>> 
>>>> Respectfully submitted,
>>>> Jennifer
>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Jennifer M. Adams
>>>> IGES/COLA
>>>> 4041 Powder Mill Road, Suite 302
>>>> Calverton, MD 20705
>>>> jma at cola.iges.org
>>>> 
>>>> 
>>>> 
>> 
>> --
>> Jennifer M. Adams
>> IGES/COLA
>> 4041 Powder Mill Road, Suite 302
>> Calverton, MD 20705
>> jma at cola.iges.org
>> 
>> 
>> 
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20111213/afd7eb7b/attachment.html 