[Go-essp-tech] GO-ESSP call schedule / Status of Gateway 2.0
Eric Nienhouse
ejn at ucar.edu
Tue Dec 20 08:44:20 MST 2011
Hi Dean, All,
We'd like to offer a walk-through of the Gateway 2.0 once our GO-ESSP
call schedule resumes after the new year.
I propose Tuesday Jan 10th for this topic.
We could likely fit this into 30 minutes and have time to address other
topics if needed.
How does that sound?
Best regards,
-Eric
Williams, Dean N. wrote:
> Hi Don,
>
> We have something already in the works for this coming week. With the
> holiday's coming in the EU and the US, perhaps it might not be best to
> have this the first week in January. Looking forward to seeing it.
>
> Best regards,
> Dean
>
> From: Don Middleton <don at ucar.edu <mailto:don at ucar.edu>>
> Date: Tue, 13 Dec 2011 07:54:57 -0800
> To: "Taylor, Karl Taylor" <taylor13 at llnl.gov
> <mailto:taylor13 at llnl.gov>>, Adams Jennifer <jma at cola.iges.org
> <mailto:jma at cola.iges.org>>
> Cc: "go-essp-tech at ucar.edu <mailto:go-essp-tech at ucar.edu>"
> <go-essp-tech at ucar.edu <mailto:go-essp-tech at ucar.edu>>
> Subject: Re: [Go-essp-tech] Status of Gateway 2.0
>
> Karl, thanks for following up on this. Jennifer, Eric spoke correctly
> last week: at NCAR we are in the process of moving 2.0 into local
> production as a priority next step. I think this will be a big
> improvement for the scientific community, and I hope we can deploy it
> more broadly for the same reason.
>
> There was discussion at the end of last week on having a PI level
> meeting but I haven't heard any more on that so far. It would helpful
> to have some more open, community discussion of how we move forward;
> across the federation there are a lot of investments, stakeholders,
> and resource issues to consider. We would also like to offer a
> walk-through of 2.0. There's not a lot of time left before the
> holidays, but we do have next Tuesday's GO-ESSP slot. If you're
> available and interested, please let us know, and Eric & Nate will put
> together a tour.
>
> cheers - don
>
>
> On Dec 7, 2011, at 1:01 PM, Karl Taylor wrote:
>
>> Dear Jennifer and Jean-Yves,
>>
>> I apologize that there has been some confusion perpetrated concerning
>> the planned paths for ESG improvement. For CMIP5 we expect to clear
>> this up by the end of next week when we will post a clear statement
>> outlining our plans.
>>
>> Thanks again for providing feedback that will improve the system.
>>
>> Best regards,
>> Karl
>>
>> On 12/7/11 9:10 AM, Jennifer Adams wrote:
>>> Thank you, Karl, for a very helpful straight answer. One thing I
>>> don't understand: Eric Neinhouse's statement that 'getting Gateway
>>> 2.0 in production at NCAR is a top priority' seems to be in direct
>>> conflict with what you wrote. Will Gateway 2.0 be for non-CMIP5 ESG
>>> datasets, or are NCAR and PCMDI taking different approaches?
>>> --Jennifer
>>>
>>> On Dec 7, 2011, at 11:29 AM, Karl Taylor wrote:
>>>
>>>> Dear Jennifer,
>>>>
>>>> Thank you for your valuable input on how to improve the search and
>>>> scripting capabilities of both the ESG Gateway 2.0 and the ESGF P2P
>>>> system.
>>>>
>>>> Before you spend too much time attempting to optimize your download
>>>> of CMIP5 data, you should know that we are very unlikely to
>>>> actually make Gateway 2.0 operational for CMIP5 model output.
>>>> Instead we will move directly to P2P. We believe that with
>>>> currently available resources PCMDI and the ESG federation cannot
>>>> support two new systems. Instead we will keep our current
>>>> operational system running and bring on line the P2P. The testing
>>>> phase of this should happen by the end of the month. At some point
>>>> the currently operational system will be turned off and the P2P
>>>> will become operational. Therefore, if as a CMIP5 user you plan to
>>>> further explore the new systems, I suggest your time might be best
>>>> spent focusing on P2P.
>>>>
>>>> P2P and Gateway 2.0 may have similar capabilities at this time, and
>>>> both are substantially improved over previous versions of the
>>>> Gateway, but we're convinced that P2P provides a more robust and
>>>> flexible framework for adding future improvements. By going
>>>> directly to P2P, we'll also minimize the inconvenience for users
>>>> already on the system.
>>>>
>>>> Please let me know if you have any concerns about our plans. We
>>>> realize the current system continues to be more difficult to use
>>>> than it should be, so your feedback and suggestions are
>>>> invaluable. Our focus is on getting the improvements offered by
>>>> P2P in place as soon as possible. I will let you know if our
>>>> planned path to a better ESGF delivery system for CMIP5 data
>>>> changes. Please feel free to write me directly at taylor13 at llnl.gov
>>>>
>>>> best regards,
>>>> Karl
>>>>
>>>> On 12/7/11 6:14 AM, Jennifer Adams wrote:
>>>>> Dear Colleagues,
>>>>> Here is more feedback on the prototype of Gateway 2.0. My current
>>>>> use case is searching with a fixed experiment, realm, frequency,
>>>>> and a selected subset of variables. I'm delighted that you can
>>>>> select more than one variable and it will keep adding them to the
>>>>> list of selected results. THANK YOU for listing the variables
>>>>> alphabetically by short name ("Variable Name"); the listing by
>>>>> long name ("CF Variable") is not useful to me, it is not
>>>>> alphabetized, and some names are so long they wrap so it looks
>>>>> like there are three variables called "Flux in the Atmosphere",
>>>>> "Due to Natural", and "Sources". Oops, my bad, it's actually
>>>>> "Surface Carbon Mass Flux into the Atmosphere Due to Natural
>>>>> Sources". Heh. QED.
>>>>>
>>>>> When I'm done clicking on all my desired constraints, I get a URL
>>>>> that looks like this :
>>>>> http://search-esg.prototype.ucar.edu/search?Type=Dataset&Experiment=decadal1960&Frequency=mon&Realm=atmos&Variable+Name=clt&Variable+Name=prw
>>>>> <http://search-esg.prototype.ucar.edu/search?Type=Dataset&Experiment=decadal1960&Frequency=mon&Realm=atmos&Variable+Name=clt&Variable+Name=prw>
>>>>> That is great, because it means I can take a short cut from all
>>>>> the mouse clicking by building a URL that will create my desired
>>>>> collection.
>>>>>
>>>>> I am happy with the search results, so I click on the "Download
>>>>> Selected" button, and get a wget script called "script.wget".
>>>>>
>>>>> 1. It would be great if you could give the script a name that
>>>>> reflects the search constraints used to build that script --
>>>>> something like
>>>>> wget.cmip5.decadal1960.mon.atmos.clt+prw.sh
>>>>> Every time I download a new wget script, I have to rename it to
>>>>> something meaningful (such as the above) because the very next
>>>>> search result will overwrite script.wget, or give me
>>>>> "script(1).wget" instead, and I'll have no simple way to know what
>>>>> data set that wget script is configured to bring back. Can you
>>>>> PLEASE give the wget script a relevant name for me so I can skip
>>>>> this time-consuming step?
>>>>>
>>>>> 2. Is there a way to use our favorite tool (wget) or curl to
>>>>> download the wget script so I can bypass the Gateway 2.0 browser
>>>>> interface altogether? My guess is the answer is no, because of the
>>>>> elaborate login process with OpenID and username/password, but
>>>>> it's worth asking…
>>>>>
>>>>> -----------
>>>>>
>>>>> As an alternative, I have looked carefully at the wiki page (and
>>>>> its embedded links) on automated generation of wget scripts,
>>>>> http://www.esgf.org/wiki/ESGF_scripting. This is very promising,
>>>>> and Luca assures me that the P2P service will be working in a
>>>>> matter of weeks.
>>>>>
>>>>> I think the syntax of the URL for the same search as my example
>>>>> above would be:
>>>>> http://esg-datanode.jpl.nasa.gov/esg-search/wget?experiment=decadal1960&realm=atmos&time_frequency=month&variable=clt,prw
>>>>> <http://esg-datanode.jpl.nasa.gov/esg-search/wget?experiment=decadal1960&realm=atmos&time_frequency=month&variable=clt,prw>
>>>>>
>>>>> 1. Is the comma-delimited list of desired variables legitimate
>>>>> syntax?
>>>>> 2. Will it work the way the Gateway 2.0 does to deliver my subset
>>>>> of variables?
>>>>> 3. Will "http://esg-datanode.jpl.nasa.gov
>>>>> <http://esg-datanode.jpl.nasa.gov/>" be the only location
>>>>> providing this service?
>>>>> 4. Will the delivered wget script use certificates and checksums?
>>>>>
>>>>> Since this isn't fully operational at the moment, I can't say for
>>>>> sure whether this will be the better route for reaching my goal of
>>>>> acquiring CMIP5 data using only scripts and no browser.
>>>>>
>>>>> Respectfully submitted,
>>>>> Jennifer
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Jennifer M. Adams
>>>>> IGES/COLA
>>>>> 4041 Powder Mill Road, Suite 302
>>>>> Calverton, MD 20705
>>>>> jma at cola.iges.org <mailto:jma at cola.iges.org>
>>>>>
>>>>>
>>>>>
>>>
>>> --
>>> Jennifer M. Adams
>>> IGES/COLA
>>> 4041 Powder Mill Road, Suite 302
>>> Calverton, MD 20705
>>> jma at cola.iges.org <mailto:jma at cola.iges.org>
>>>
>>>
>>>
>> _______________________________________________
>> GO-ESSP-TECH mailing list
>> GO-ESSP-TECH at ucar.edu <mailto:GO-ESSP-TECH at ucar.edu>
>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
More information about the GO-ESSP-TECH
mailing list