[Go-essp-tech] GO-ESSP call Jan 10, 2012
Don Middleton
don at ucar.edu
Wed Jan 11 16:43:52 MST 2012
Thanks again, Eric & team, for the nice demo on Tuesday and thanks to all of you who attended! As noted, we did the demonstration from our staging server rather than our production system, which had been the original plan. There are a few things to explain here.
As Eric indicated, we put 2.0 into production at our site on 27 December. It went very smoothly and we have heard positive feedback from the user community. Just recently, Karl contacted us and asked us to remove all of the federated CMIP5 datasets from our gateway. We complied with his request and, on Monday night before the demo, the team here unpublished 30,000+ datasets. We are still running our production gateway on 2.0-RC1, just without the federated datasets.
So this is a heads up in case this affects you and/or anything related arises on the support desk. The new CIM<->Data cross-links were all to federated data, so those were also removed as part of the unpublishing operation and no longer appear. Lastly, federation OpenID's and data access are not enabled on the staging instance, and we're putting up a banner that mentions this so people aren't confused or frustrated. Please let us know if anything else needs attention.
cheers - don
On Jan 10, 2012, at 10:26 AM, Eric Nienhouse wrote:
> Hi All,
>
> Thanks for attending and contributing to the the 2.0 Gateway Demo today.
>
> The 2.0 site we walked through can be found at:
>
> http://esg.prototype.ucar.edu
>
> Best regards,
>
> -Eric
>
>
> Eric Nienhouse wrote:
>> Hi All,
>>
>> If anyone has trouble accessing ready talk for today's Gateway demo,
>> you can follow along by going directly to the NCAR federation testing
>> 2.0 Gateway:
>>
>> http://esg.prototype.ucar.edu
>>
>> Thank you Stephen for the CMIP5 Web interface requirements matrix. We
>> should have time to introduce this during today's call.
>>
>> Thanks,
>>
>> -Eric
>>
>> Michael Lautenschlager wrote:
>>> Hi Eric,
>>> thanks for the information. I will be on the call at least for one
>>> hour together with Estani.
>>> Best wishes, Michael
>>>
>>> ---------------
>>> Dr. Michael Lautenschlager
>>> Head of DKRZ Department Data Management
>>> Director World Data Center Climate
>>>
>>> German Climate Computing Centre (DKRZ)
>>> ADDRESS: Bundesstrasse 45a, D-20146 Hamburg, Germany
>>> PHONE: +4940-460094-118
>>> E-Mail: lautenschlager at dkrz.de
>>>
>>> URL: http://www.dkrz.de/
>>> http://www.wdc-climate.de/
>>>
>>>
>>> Geschäftsführer: Prof. Dr. Thomas Ludwig
>>> Sitz der Gesellschaft: Hamburg
>>> Amtsgericht Hamburg HRB 39784
>>>
>>>
>>> Am 10.01.2012 00:53, schrieb Eric Nienhouse:
>>>> Hi All,
>>>>
>>>> We plan to hold a GO-ESSP tech call tomorrow (Jan 10, 2012.) Agenda
>>>> follows:
>>>>
>>>> * Overview and demo of Gateway 2.0.
>>>> * Near and mid term technology roadmap for CMIP5 operations (open
>>>> discussion.)
>>>>
>>>> We'll be using ReadyTalk web sharing for the Gateway 2.0 demo.
>>>> There is
>>>> a brief registration page to join in. The link follows:
>>>>
>>>> https://cc.readytalk.com/r/6icl9p8fqbey
>>>>
>>>> Please access the telco number for the audio portion:
>>>>
>>>> +01 (925) 424-8105 access code 305757#
>>>>
>>>> *Telco details:
>>>> *16:00 BST, 17:00 CEST, 8:00 PDT, 9:00 MDT, 11:00 EDT.
>>>>
>>>> Apologies - the previous email subject noted the telco is in
>>>> "2011". It
>>>> is, of course, scheduled for Jan 10, 2012. Happy New Year :-)
>>>>
>>>> Best regards,
>>>>
>>>> -Eric
>>>>
>>>>
>>>> Eric Nienhouse wrote:
>>>>> Hi All,
>>>>>
>>>>> We plan to hold a GO-ESSP tech call tomorrow. Following is our
>>>>> tentative agenda:
>>>>>
>>>>> * Overview and demo of Gateway 2.0.
>>>>> * Near and mid term technology roadmap for CMIP5 operations (open
>>>>> discussion.)
>>>>>
>>>>> We'll be using ReadyTalk web sharing for the Gateway 2.0 demo. I'll
>>>>> send a follow-up email with connection details.
>>>>>
>>>>> Best regards,
>>>>>
>>>>> -Eric
>>>>>
>>>>> Eric Nienhouse wrote:
>>>>>
>>>>>> Hi Micheal,
>>>>>>
>>>>>> Thank you. I agree discussing the future path of the running CMIP5
>>>>>> archive and the ESGF efforts openly is extremely important as we all
>>>>>> plan and work to meet the needs of our user community. We should
>>>>>> devote the time necessary to discuss this topic (or at least begin
>>>>>> to)
>>>>>> in our upcoming call on Jan 10th as you suggest.
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> -Eric
>>>>>>
>>>>>> Michael Lautenschlager wrote:
>>>>>>
>>>>>>> Dear Eric,
>>>>>>>
>>>>>>> I support your suggestion to postpone the gateway discussion to
>>>>>>> January 10th. Then most of us should be back from vacation between
>>>>>>> the years. But I suggest that we should plan for more than 30
>>>>>>> minutes
>>>>>>> discussion on this topic because I think we should discuss the short
>>>>>>> term development for the running CMIP5 archive and should also start
>>>>>>> discussion on the mid term development within ESGF. Both aspects are
>>>>>>> important for us at DKRZ because we would like to contribute to
>>>>>>> ESGF,
>>>>>>> share work between ESGF partners and avoid double work.
>>>>>>>
>>>>>>> Best wishes, Michael
>>>>>>>
>>>>>>> ---------------
>>>>>>> Dr. Michael Lautenschlager
>>>>>>> Head of DKRZ Department Data Management
>>>>>>> Director World Data Center Climate
>>>>>>>
>>>>>>> German Climate Computing Centre (DKRZ)
>>>>>>> ADDRESS: Bundesstrasse 45a, D-20146 Hamburg, Germany
>>>>>>> PHONE: +4940-460094-118
>>>>>>> E-Mail: lautenschlager at dkrz.de
>>>>>>>
>>>>>>> URL: http://www.dkrz.de/
>>>>>>> http://www.wdc-climate.de/
>>>>>>>
>>>>>>>
>>>>>>> Geschäftsführer: Prof. Dr. Thomas Ludwig
>>>>>>> Sitz der Gesellschaft: Hamburg
>>>>>>> Amtsgericht Hamburg HRB 39784
>>>>>>>
>>>>>>>
>>>>>>> Am 20.12.2011 16:44, schrieb Eric Nienhouse:
>>>>>>>
>>>>>>>> Hi Dean, All,
>>>>>>>>
>>>>>>>> We'd like to offer a walk-through of the Gateway 2.0 once our
>>>>>>>> GO-ESSP
>>>>>>>> call schedule resumes after the new year.
>>>>>>>>
>>>>>>>> I propose Tuesday Jan 10th for this topic.
>>>>>>>>
>>>>>>>> We could likely fit this into 30 minutes and have time to
>>>>>>>> address other
>>>>>>>> topics if needed.
>>>>>>>>
>>>>>>>> How does that sound?
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>> -Eric
>>>>>>>>
>>>>>>>> Williams, Dean N. wrote:
>>>>>>>>
>>>>>>>>> Hi Don,
>>>>>>>>>
>>>>>>>>> We have something already in the works for this coming week.
>>>>>>>>> With the
>>>>>>>>> holiday's coming in the EU and the US, perhaps it might not be
>>>>>>>>> best to
>>>>>>>>> have this the first week in January. Looking forward to seeing it.
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>> Dean
>>>>>>>>>
>>>>>>>>> From: Don Middleton<don at ucar.edu<mailto:don at ucar.edu>>
>>>>>>>>> Date: Tue, 13 Dec 2011 07:54:57 -0800
>>>>>>>>> To: "Taylor, Karl Taylor"<taylor13 at llnl.gov
>>>>>>>>> <mailto:taylor13 at llnl.gov>>, Adams Jennifer<jma at cola.iges.org
>>>>>>>>> <mailto:jma at cola.iges.org>>
>>>>>>>>> Cc: "go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>"
>>>>>>>>> <go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>>
>>>>>>>>> Subject: Re: [Go-essp-tech] Status of Gateway 2.0
>>>>>>>>>
>>>>>>>>> Karl, thanks for following up on this. Jennifer, Eric spoke
>>>>>>>>> correctly
>>>>>>>>> last week: at NCAR we are in the process of moving 2.0 into local
>>>>>>>>> production as a priority next step. I think this will be a big
>>>>>>>>> improvement for the scientific community, and I hope we can
>>>>>>>>> deploy it
>>>>>>>>> more broadly for the same reason.
>>>>>>>>>
>>>>>>>>> There was discussion at the end of last week on having a PI level
>>>>>>>>> meeting but I haven't heard any more on that so far. It would
>>>>>>>>> helpful
>>>>>>>>> to have some more open, community discussion of how we move
>>>>>>>>> forward;
>>>>>>>>> across the federation there are a lot of investments,
>>>>>>>>> stakeholders,
>>>>>>>>> and resource issues to consider. We would also like to offer a
>>>>>>>>> walk-through of 2.0. There's not a lot of time left before the
>>>>>>>>> holidays, but we do have next Tuesday's GO-ESSP slot. If you're
>>>>>>>>> available and interested, please let us know, and Eric& Nate
>>>>>>>>> will put
>>>>>>>>> together a tour.
>>>>>>>>>
>>>>>>>>> cheers - don
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Dec 7, 2011, at 1:01 PM, Karl Taylor wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Dear Jennifer and Jean-Yves,
>>>>>>>>>>
>>>>>>>>>> I apologize that there has been some confusion perpetrated
>>>>>>>>>> concerning
>>>>>>>>>> the planned paths for ESG improvement. For CMIP5 we expect to
>>>>>>>>>> clear
>>>>>>>>>> this up by the end of next week when we will post a clear
>>>>>>>>>> statement
>>>>>>>>>> outlining our plans.
>>>>>>>>>>
>>>>>>>>>> Thanks again for providing feedback that will improve the system.
>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>> Karl
>>>>>>>>>>
>>>>>>>>>> On 12/7/11 9:10 AM, Jennifer Adams wrote:
>>>>>>>>>>
>>>>>>>>>>> Thank you, Karl, for a very helpful straight answer. One thing I
>>>>>>>>>>> don't understand: Eric Neinhouse's statement that 'getting
>>>>>>>>>>> Gateway
>>>>>>>>>>> 2.0 in production at NCAR is a top priority' seems to be in
>>>>>>>>>>> direct
>>>>>>>>>>> conflict with what you wrote. Will Gateway 2.0 be for
>>>>>>>>>>> non-CMIP5 ESG
>>>>>>>>>>> datasets, or are NCAR and PCMDI taking different approaches?
>>>>>>>>>>> --Jennifer
>>>>>>>>>>>
>>>>>>>>>>> On Dec 7, 2011, at 11:29 AM, Karl Taylor wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Dear Jennifer,
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you for your valuable input on how to improve the
>>>>>>>>>>>> search and
>>>>>>>>>>>> scripting capabilities of both the ESG Gateway 2.0 and the
>>>>>>>>>>>> ESGF P2P
>>>>>>>>>>>> system.
>>>>>>>>>>>>
>>>>>>>>>>>> Before you spend too much time attempting to optimize your
>>>>>>>>>>>> download
>>>>>>>>>>>> of CMIP5 data, you should know that we are very unlikely to
>>>>>>>>>>>> actually make Gateway 2.0 operational for CMIP5 model output.
>>>>>>>>>>>> Instead we will move directly to P2P. We believe that with
>>>>>>>>>>>> currently available resources PCMDI and the ESG federation
>>>>>>>>>>>> cannot
>>>>>>>>>>>> support two new systems. Instead we will keep our current
>>>>>>>>>>>> operational system running and bring on line the P2P. The
>>>>>>>>>>>> testing
>>>>>>>>>>>> phase of this should happen by the end of the month. At
>>>>>>>>>>>> some point
>>>>>>>>>>>> the currently operational system will be turned off and the P2P
>>>>>>>>>>>> will become operational. Therefore, if as a CMIP5 user you
>>>>>>>>>>>> plan to
>>>>>>>>>>>> further explore the new systems, I suggest your time might
>>>>>>>>>>>> be best
>>>>>>>>>>>> spent focusing on P2P.
>>>>>>>>>>>>
>>>>>>>>>>>> P2P and Gateway 2.0 may have similar capabilities at this
>>>>>>>>>>>> time, and
>>>>>>>>>>>> both are substantially improved over previous versions of the
>>>>>>>>>>>> Gateway, but we're convinced that P2P provides a more robust
>>>>>>>>>>>> and
>>>>>>>>>>>> flexible framework for adding future improvements. By going
>>>>>>>>>>>> directly to P2P, we'll also minimize the inconvenience for
>>>>>>>>>>>> users
>>>>>>>>>>>> already on the system.
>>>>>>>>>>>>
>>>>>>>>>>>> Please let me know if you have any concerns about our
>>>>>>>>>>>> plans. We
>>>>>>>>>>>> realize the current system continues to be more difficult to
>>>>>>>>>>>> use
>>>>>>>>>>>> than it should be, so your feedback and suggestions are
>>>>>>>>>>>> invaluable. Our focus is on getting the improvements
>>>>>>>>>>>> offered by
>>>>>>>>>>>> P2P in place as soon as possible. I will let you know if our
>>>>>>>>>>>> planned path to a better ESGF delivery system for CMIP5 data
>>>>>>>>>>>> changes. Please feel free to write me directly at
>>>>>>>>>>>> taylor13 at llnl.gov
>>>>>>>>>>>>
>>>>>>>>>>>> best regards,
>>>>>>>>>>>> Karl
>>>>>>>>>>>>
>>>>>>>>>>>> On 12/7/11 6:14 AM, Jennifer Adams wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Dear Colleagues,
>>>>>>>>>>>>> Here is more feedback on the prototype of Gateway 2.0. My
>>>>>>>>>>>>> current
>>>>>>>>>>>>> use case is searching with a fixed experiment, realm,
>>>>>>>>>>>>> frequency,
>>>>>>>>>>>>> and a selected subset of variables. I'm delighted that you can
>>>>>>>>>>>>> select more than one variable and it will keep adding them
>>>>>>>>>>>>> to the
>>>>>>>>>>>>> list of selected results. THANK YOU for listing the variables
>>>>>>>>>>>>> alphabetically by short name ("Variable Name"); the listing by
>>>>>>>>>>>>> long name ("CF Variable") is not useful to me, it is not
>>>>>>>>>>>>> alphabetized, and some names are so long they wrap so it looks
>>>>>>>>>>>>> like there are three variables called "Flux in the
>>>>>>>>>>>>> Atmosphere",
>>>>>>>>>>>>> "Due to Natural", and "Sources". Oops, my bad, it's actually
>>>>>>>>>>>>> "Surface Carbon Mass Flux into the Atmosphere Due to Natural
>>>>>>>>>>>>> Sources". Heh. QED.
>>>>>>>>>>>>>
>>>>>>>>>>>>> When I'm done clicking on all my desired constraints, I get
>>>>>>>>>>>>> a URL
>>>>>>>>>>>>> that looks like this :
>>>>>>>>>>>>> http://search-esg.prototype.ucar.edu/search?Type=Dataset&Experiment=decadal1960&Frequency=mon&Realm=atmos&Variable+Name=clt&Variable+Name=prw
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> <http://search-esg.prototype.ucar.edu/search?Type=Dataset&Experiment=decadal1960&Frequency=mon&Realm=atmos&Variable+Name=clt&Variable+Name=prw>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> That is great, because it means I can take a short cut from
>>>>>>>>>>>>> all
>>>>>>>>>>>>> the mouse clicking by building a URL that will create my
>>>>>>>>>>>>> desired
>>>>>>>>>>>>> collection.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am happy with the search results, so I click on the
>>>>>>>>>>>>> "Download
>>>>>>>>>>>>> Selected" button, and get a wget script called "script.wget".
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. It would be great if you could give the script a name that
>>>>>>>>>>>>> reflects the search constraints used to build that script --
>>>>>>>>>>>>> something like
>>>>>>>>>>>>> wget.cmip5.decadal1960.mon.atmos.clt+prw.sh
>>>>>>>>>>>>> Every time I download a new wget script, I have to rename
>>>>>>>>>>>>> it to
>>>>>>>>>>>>> something meaningful (such as the above) because the very next
>>>>>>>>>>>>> search result will overwrite script.wget, or give me
>>>>>>>>>>>>> "script(1).wget" instead, and I'll have no simple way to
>>>>>>>>>>>>> know what
>>>>>>>>>>>>> data set that wget script is configured to bring back. Can you
>>>>>>>>>>>>> PLEASE give the wget script a relevant name for me so I can
>>>>>>>>>>>>> skip
>>>>>>>>>>>>> this time-consuming step?
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2. Is there a way to use our favorite tool (wget) or curl to
>>>>>>>>>>>>> download the wget script so I can bypass the Gateway 2.0
>>>>>>>>>>>>> browser
>>>>>>>>>>>>> interface altogether? My guess is the answer is no, because
>>>>>>>>>>>>> of the
>>>>>>>>>>>>> elaborate login process with OpenID and username/password, but
>>>>>>>>>>>>> it's worth asking…
>>>>>>>>>>>>>
>>>>>>>>>>>>> -----------
>>>>>>>>>>>>>
>>>>>>>>>>>>> As an alternative, I have looked carefully at the wiki page
>>>>>>>>>>>>> (and
>>>>>>>>>>>>> its embedded links) on automated generation of wget scripts,
>>>>>>>>>>>>> http://www.esgf.org/wiki/ESGF_scripting. This is very
>>>>>>>>>>>>> promising,
>>>>>>>>>>>>> and Luca assures me that the P2P service will be working in a
>>>>>>>>>>>>> matter of weeks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think the syntax of the URL for the same search as my
>>>>>>>>>>>>> example
>>>>>>>>>>>>> above would be:
>>>>>>>>>>>>> http://esg-datanode.jpl.nasa.gov/esg-search/wget?experiment=decadal1960&realm=atmos&time_frequency=month&variable=clt,prw
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> <http://esg-datanode.jpl.nasa.gov/esg-search/wget?experiment=decadal1960&realm=atmos&time_frequency=month&variable=clt,prw>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. Is the comma-delimited list of desired variables legitimate
>>>>>>>>>>>>> syntax?
>>>>>>>>>>>>> 2. Will it work the way the Gateway 2.0 does to deliver my
>>>>>>>>>>>>> subset
>>>>>>>>>>>>> of variables?
>>>>>>>>>>>>> 3. Will "http://esg-datanode.jpl.nasa.gov
>>>>>>>>>>>>> <http://esg-datanode.jpl.nasa.gov/>" be the only location
>>>>>>>>>>>>> providing this service?
>>>>>>>>>>>>> 4. Will the delivered wget script use certificates and
>>>>>>>>>>>>> checksums?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Since this isn't fully operational at the moment, I can't
>>>>>>>>>>>>> say for
>>>>>>>>>>>>> sure whether this will be the better route for reaching my
>>>>>>>>>>>>> goal of
>>>>>>>>>>>>> acquiring CMIP5 data using only scripts and no browser.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Respectfully submitted,
>>>>>>>>>>>>> Jennifer
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Jennifer M. Adams
>>>>>>>>>>>>> IGES/COLA
>>>>>>>>>>>>> 4041 Powder Mill Road, Suite 302
>>>>>>>>>>>>> Calverton, MD 20705
>>>>>>>>>>>>> jma at cola.iges.org<mailto:jma at cola.iges.org>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Jennifer M. Adams
>>>>>>>>>>> IGES/COLA
>>>>>>>>>>> 4041 Powder Mill Road, Suite 302
>>>>>>>>>>> Calverton, MD 20705
>>>>>>>>>>> jma at cola.iges.org<mailto:jma at cola.iges.org>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> GO-ESSP-TECH mailing list
>>>>>>>>>> GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
>>>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>>>>>>>
>>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> GO-ESSP-TECH mailing list
>>>>>>>>> GO-ESSP-TECH at ucar.edu
>>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>>>>>>
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> GO-ESSP-TECH mailing list
>>>>>>>> GO-ESSP-TECH at ucar.edu
>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> GO-ESSP-TECH mailing list
>>>>> GO-ESSP-TECH at ucar.edu
>>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>>
>>>> _______________________________________________
>>>> GO-ESSP-TECH mailing list
>>>> GO-ESSP-TECH at ucar.edu
>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>
>>
>>
>
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
More information about the GO-ESSP-TECH
mailing list