[Go-essp-tech] GO-ESSP call Jan 10, 2011

Eric Nienhouse ejn at ucar.edu
Mon Jan 9 14:25:18 MST 2012 

Hi All,

We plan to hold a GO-ESSP tech call tomorrow.  Following is our 
tentative agenda:

* Overview and demo of Gateway 2.0.
* Near and mid term technology roadmap for CMIP5 operations (open 
discussion.)

We'll be using ReadyTalk web sharing for the Gateway 2.0 demo.  I'll 
send a follow-up email with connection details.

Best regards,

-Eric

Eric Nienhouse wrote:
> Hi Micheal,
>
> Thank you.  I agree discussing the future path of the running CMIP5 
> archive and the ESGF efforts openly is extremely important as we all 
> plan and work to meet the needs of our user community.  We should 
> devote the time necessary to discuss this topic (or at least begin to) 
> in our upcoming call on Jan 10th as you suggest.
>
> Best regards,
>
> -Eric
>
> Michael Lautenschlager wrote:
>> Dear Eric,
>>
>> I support your suggestion to postpone the gateway discussion to 
>> January 10th. Then most of us should be back from vacation between 
>> the years. But I suggest that we should plan for more than 30 minutes 
>> discussion on this topic because I think we should discuss the short 
>> term development for the running CMIP5 archive and should also start 
>> discussion on the mid term development within ESGF. Both aspects are 
>> important for us at DKRZ because we would like to contribute to ESGF, 
>> share work between ESGF partners and avoid double work.
>>
>> Best wishes, Michael
>>
>> ---------------
>> Dr. Michael Lautenschlager
>> Head of DKRZ Department Data Management
>> Director World Data Center Climate
>>
>> German Climate Computing Centre (DKRZ)
>> ADDRESS: Bundesstrasse 45a, D-20146 Hamburg, Germany
>> PHONE:   +4940-460094-118
>> E-Mail:  lautenschlager at dkrz.de
>>
>> URL:    http://www.dkrz.de/
>>         http://www.wdc-climate.de/
>>
>>
>> Geschäftsführer: Prof. Dr. Thomas Ludwig
>> Sitz der Gesellschaft: Hamburg
>> Amtsgericht Hamburg HRB 39784
>>
>>
>> Am 20.12.2011 16:44, schrieb Eric Nienhouse:
>>> Hi Dean, All,
>>>
>>> We'd like to offer a walk-through of the Gateway 2.0 once our GO-ESSP
>>> call schedule resumes after the new year.
>>>
>>> I propose Tuesday Jan 10th for this topic.
>>>
>>> We could likely fit this into 30 minutes and have time to address other
>>> topics if needed.
>>>
>>> How does that sound?
>>>
>>> Best regards,
>>>
>>> -Eric
>>>
>>> Williams, Dean N. wrote:
>>>> Hi Don,
>>>>
>>>> We have something already in the works for this coming week. With the
>>>> holiday's coming in the EU and the US, perhaps it might not be best to
>>>> have this the first week in January. Looking forward to seeing it.
>>>>
>>>> Best regards,
>>>> Dean
>>>>
>>>> From: Don Middleton<don at ucar.edu<mailto:don at ucar.edu>>
>>>> Date: Tue, 13 Dec 2011 07:54:57 -0800
>>>> To: "Taylor, Karl Taylor"<taylor13 at llnl.gov
>>>> <mailto:taylor13 at llnl.gov>>, Adams Jennifer<jma at cola.iges.org
>>>> <mailto:jma at cola.iges.org>>
>>>> Cc: "go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>"
>>>> <go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>>
>>>> Subject: Re: [Go-essp-tech] Status of Gateway 2.0
>>>>
>>>> Karl, thanks for following up on this. Jennifer, Eric spoke correctly
>>>> last week: at NCAR we are in the process of moving 2.0 into local
>>>> production as a priority next step. I think this will be a big
>>>> improvement for the scientific community, and I hope we can deploy it
>>>> more broadly for the same reason.
>>>>
>>>> There was discussion at the end of last week on having a PI level
>>>> meeting but I haven't heard any more on that so far. It would helpful
>>>> to have some more open, community discussion of how we move forward;
>>>> across the federation there are a lot of investments, stakeholders,
>>>> and resource issues to consider. We would also like to offer a
>>>> walk-through of 2.0. There's not a lot of time left before the
>>>> holidays, but we do have next Tuesday's GO-ESSP slot. If you're
>>>> available and interested, please let us know, and Eric&  Nate will put
>>>> together a tour.
>>>>
>>>> cheers - don
>>>>
>>>>
>>>> On Dec 7, 2011, at 1:01 PM, Karl Taylor wrote:
>>>>
>>>>> Dear Jennifer and Jean-Yves,
>>>>>
>>>>> I apologize that there has been some confusion perpetrated concerning
>>>>> the planned paths for ESG improvement.  For CMIP5 we expect to clear
>>>>> this up by the end of next week when we will post a clear statement
>>>>> outlining our plans.
>>>>>
>>>>> Thanks again for providing feedback that will improve the system.
>>>>>
>>>>> Best regards,
>>>>> Karl
>>>>>
>>>>> On 12/7/11 9:10 AM, Jennifer Adams wrote:
>>>>>> Thank you, Karl, for a very helpful straight answer. One thing I
>>>>>> don't understand: Eric Neinhouse's statement that 'getting Gateway
>>>>>> 2.0 in production at NCAR is a top priority' seems to be in direct
>>>>>> conflict with what you wrote. Will Gateway 2.0 be for non-CMIP5 ESG
>>>>>> datasets, or are NCAR and PCMDI taking different approaches?
>>>>>> --Jennifer
>>>>>>
>>>>>> On Dec 7, 2011, at 11:29 AM, Karl Taylor wrote:
>>>>>>
>>>>>>> Dear Jennifer,
>>>>>>>
>>>>>>> Thank you for your valuable input on how to improve the search and
>>>>>>> scripting capabilities of both the ESG Gateway 2.0 and the ESGF P2P
>>>>>>> system.
>>>>>>>
>>>>>>> Before you spend too much time attempting to optimize your download
>>>>>>> of CMIP5 data, you should know that we are very unlikely to
>>>>>>> actually make Gateway 2.0 operational for CMIP5 model output.
>>>>>>> Instead we will move directly to P2P.  We believe that with
>>>>>>> currently available resources PCMDI and the ESG federation cannot
>>>>>>> support two new systems.  Instead we will keep our current
>>>>>>> operational system running and bring on line the P2P.  The testing
>>>>>>> phase of this should happen by the end of the month.  At some point
>>>>>>> the currently operational system will be turned off and the P2P
>>>>>>> will become operational.  Therefore, if as a CMIP5 user you plan to
>>>>>>> further explore the new systems, I suggest your time might be best
>>>>>>> spent focusing on P2P.
>>>>>>>
>>>>>>> P2P and Gateway 2.0 may have similar capabilities at this time, and
>>>>>>> both are substantially improved over previous versions of the
>>>>>>> Gateway, but we're convinced that P2P provides a more robust and
>>>>>>> flexible framework for adding future improvements.   By going
>>>>>>> directly to P2P, we'll also minimize the inconvenience for users
>>>>>>> already on the system.
>>>>>>>
>>>>>>> Please let me know if you have any concerns about our plans.  We
>>>>>>> realize the current system continues to be more difficult to use
>>>>>>> than it should be,  so your feedback and suggestions are
>>>>>>> invaluable.   Our focus is on getting the improvements offered by
>>>>>>> P2P in place as soon as possible.  I will let you know if our
>>>>>>> planned path to a better ESGF delivery system for CMIP5 data
>>>>>>> changes.  Please feel free to write me directly at 
>>>>>>> taylor13 at llnl.gov
>>>>>>>
>>>>>>> best regards,
>>>>>>> Karl
>>>>>>>
>>>>>>> On 12/7/11 6:14 AM, Jennifer Adams wrote:
>>>>>>>> Dear Colleagues,
>>>>>>>> Here is more feedback on the prototype of Gateway 2.0. My current
>>>>>>>> use case is searching with a fixed experiment, realm, frequency,
>>>>>>>> and a selected subset of variables. I'm delighted that you can
>>>>>>>> select more than one variable and it will keep adding them to the
>>>>>>>> list of selected results. THANK YOU for listing the variables
>>>>>>>> alphabetically by short name ("Variable Name"); the listing by
>>>>>>>> long name ("CF Variable") is not useful to me, it is not
>>>>>>>> alphabetized, and some names are so long they wrap so it looks
>>>>>>>> like there are three variables called "Flux in the Atmosphere",
>>>>>>>> "Due to Natural", and "Sources". Oops, my bad, it's actually
>>>>>>>> "Surface Carbon Mass Flux into the Atmosphere Due to Natural
>>>>>>>> Sources". Heh. QED.
>>>>>>>>
>>>>>>>> When I'm done clicking on all my desired constraints, I get a URL
>>>>>>>> that looks like this :
>>>>>>>> http://search-esg.prototype.ucar.edu/search?Type=Dataset&Experiment=decadal1960&Frequency=mon&Realm=atmos&Variable+Name=clt&Variable+Name=prw 
>>>>>>>>
>>>>>>>> <http://search-esg.prototype.ucar.edu/search?Type=Dataset&Experiment=decadal1960&Frequency=mon&Realm=atmos&Variable+Name=clt&Variable+Name=prw> 
>>>>>>>>
>>>>>>>> That is great, because it means I can take a short cut from all
>>>>>>>> the mouse clicking by building a URL that will create my desired
>>>>>>>> collection.
>>>>>>>>
>>>>>>>> I am happy with the search results, so I click on the "Download
>>>>>>>> Selected" button, and get a wget script called "script.wget".
>>>>>>>>
>>>>>>>> 1. It would be great if you could give the script a name that
>>>>>>>> reflects the search constraints used to build that script --
>>>>>>>> something like
>>>>>>>>      wget.cmip5.decadal1960.mon.atmos.clt+prw.sh
>>>>>>>> Every time I download a new wget script, I have to rename it to
>>>>>>>> something meaningful (such as the above) because the very next
>>>>>>>> search result will overwrite script.wget, or give me
>>>>>>>> "script(1).wget" instead, and I'll have no simple way to know what
>>>>>>>> data set that wget script is configured to bring back. Can you
>>>>>>>> PLEASE give the wget script a relevant name for me so I can skip
>>>>>>>> this time-consuming step?
>>>>>>>>
>>>>>>>> 2. Is there a way to use our favorite tool (wget) or curl to
>>>>>>>> download the wget script so I can bypass the Gateway 2.0 browser
>>>>>>>> interface altogether? My guess is the answer is no, because of the
>>>>>>>> elaborate login process with OpenID and username/password, but
>>>>>>>> it's worth asking…
>>>>>>>>
>>>>>>>> -----------
>>>>>>>>
>>>>>>>> As an alternative, I have looked carefully at the wiki page (and
>>>>>>>> its embedded links) on automated generation of wget scripts,
>>>>>>>> http://www.esgf.org/wiki/ESGF_scripting. This is very promising,
>>>>>>>> and Luca assures me that the P2P service will be working in a
>>>>>>>> matter of weeks.
>>>>>>>>
>>>>>>>> I think the syntax of the URL for the same search as my example
>>>>>>>> above would be:
>>>>>>>> http://esg-datanode.jpl.nasa.gov/esg-search/wget?experiment=decadal1960&realm=atmos&time_frequency=month&variable=clt,prw 
>>>>>>>>
>>>>>>>> <http://esg-datanode.jpl.nasa.gov/esg-search/wget?experiment=decadal1960&realm=atmos&time_frequency=month&variable=clt,prw> 
>>>>>>>>
>>>>>>>>
>>>>>>>> 1. Is the comma-delimited list of desired variables legitimate
>>>>>>>> syntax?
>>>>>>>> 2. Will it work the way the Gateway 2.0 does to deliver my subset
>>>>>>>> of variables?
>>>>>>>> 3. Will "http://esg-datanode.jpl.nasa.gov
>>>>>>>> <http://esg-datanode.jpl.nasa.gov/>" be the only location
>>>>>>>> providing this service?
>>>>>>>> 4. Will the delivered wget script use certificates and checksums?
>>>>>>>>
>>>>>>>> Since this isn't fully operational at the moment, I can't say for
>>>>>>>> sure whether this will be the better route for reaching my goal of
>>>>>>>> acquiring CMIP5 data using only scripts and no browser.
>>>>>>>>
>>>>>>>> Respectfully submitted,
>>>>>>>> Jennifer
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -- 
>>>>>>>> Jennifer M. Adams
>>>>>>>> IGES/COLA
>>>>>>>> 4041 Powder Mill Road, Suite 302
>>>>>>>> Calverton, MD 20705
>>>>>>>> jma at cola.iges.org<mailto:jma at cola.iges.org>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>> -- 
>>>>>> Jennifer M. Adams
>>>>>> IGES/COLA
>>>>>> 4041 Powder Mill Road, Suite 302
>>>>>> Calverton, MD 20705
>>>>>> jma at cola.iges.org<mailto:jma at cola.iges.org>
>>>>>>
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> GO-ESSP-TECH mailing list
>>>>> GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
>>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>> ------------------------------------------------------------------------ 
>>>>
>>>>
>>>> _______________________________________________
>>>> GO-ESSP-TECH mailing list
>>>> GO-ESSP-TECH at ucar.edu
>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>
>
>

More information about the GO-ESSP-TECH mailing list