[Go-essp-tech] GO-ESSP-TECH Digest, Vol 35, Issue 59
Mike Berkley
mike.berkley at gmail.com
Sun Jan 22 21:54:06 MST 2012
The ESG gateway and data nodes, are amazing software accomplishments,
so I very much hate to make negative comments.
However, it is my guess that, given a choice, "my" scientists would
ask for a dead simple http or better yet, ftp, interface to the CMIP5
data. The ESG approach offers a great deal, but as currently
implemented, clients find it neither simple nor robust, when compared
to the tools with which they are already familiar.
Therefore, I would advise caution, when considering any fundamental
changes to the ESG software. Making it less robust, even in the short
term, will be very unpopular.
On Sat, Jan 21, 2012 at 03:24, Mark Elkington
<mark.elkington at blueyonder.co.uk> wrote:
>
> Finally . . .
>
> "Scientists and data managers at the CMIP5 modelling centres are having to struggle with the current
> system and feel the ground shifting under their feet without any say in what's going on. If PCMDI
> want to deploy the P2P system within weeks then the wider CMIP5 users, not just those at PCMDI, need
> to be involved. Their work is critically dependent on what system we put in place".
>
> + a few thousand on that. As a minimum all the groups submitting data to the system should be
> invited to have some representation on this user panel, and there should be some representatives
> from the wider community too. How can you be responsive to the international community need without
> at least seeking their views as a group.
>
> Regards
>
> Mark
>
> --------------------------------------------------------------
> Dr Mark Elkington
> Elmswood Barn, Ogwell, Newton Abbot, Devon TQ12 6AF
> tel: 01626 212545 mob: 07804 795826
> email: mark.elkington at blueyonder.co.uk
>
>
> -----Original Message-----
> From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of
> go-essp-tech-request at ucar.edu
> Sent: 21 January 2012 10:32
> To: go-essp-tech at ucar.edu
> Subject: GO-ESSP-TECH Digest, Vol 35, Issue 59
>
> Send GO-ESSP-TECH mailing list submissions to
> go-essp-tech at ucar.edu
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> or, via email, send a message with subject or body 'help' to
> go-essp-tech-request at ucar.edu
>
> You can reach the person managing the list at
> go-essp-tech-owner at ucar.edu
>
> When replying, please edit your Subject line so it is more specific than "Re: Contents of
> GO-ESSP-TECH digest..."
>
>
> Today's Topics:
>
> 1. Re: Plan (stephen.pascoe at stfc.ac.uk)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sat, 21 Jan 2012 10:31:12 +0000
> From: <stephen.pascoe at stfc.ac.uk>
> Subject: Re: [Go-essp-tech] Plan
> To: <gavin at llnl.gov>, <cecelia.deluca at noaa.gov>
> Cc: Luca.Cinquini at jpl.nasa.gov, go-essp-tech at ucar.edu
> Message-ID:
> <4C353E6E4A08AE4792B350DAA392B5211C980313 at EXCHMBX01.fed.cclrc.ac.uk>
> Content-Type: text/plain; charset="us-ascii"
>
> Briefly getting back to the Agenda for Tuesday. I can make an extended 2 hour telco but the order
> we do things in will depend on who can attend 9-10 PTD. That is 6-7pm CET so people from DKRZ,
> IPSL, SMI etc. may not be able to make it. As well as the requirements matrix I'd like to cover
> setting up a scientific feedback panel as suggested by Cecelia and supported by Balaji.
>
> Inline comments follow, reordered for relevance.
>
>
>> 2. Have the metrics discussion from 9-10.
>> What kind of metrics?
> I think Cecelia means the requirements matrix aka spreadsheet I circulated
>
>
>> Bonus points:
>
>> 8. Get together a management body with key stakeholders so that
>> decisions are backed up and can be
>
>> communicated more easily.
>
>>
>> Nod, the governance structure will be formalized and posted.... among
>> other things
> Gavin, who are the stakeholders going to be? This decision should be the result of a dialog under
> the auspices of GO-ESSP not just "posted".
>
>
>
>> 3. Start putting together a list of users whose input you think would
>> be useful. This is a good idea even
>
>> if they only ever end up reviewing the p2p system.
>> Here at PCMDI we have had a handful of key scientists looking at,
>> using and critiquing the P2P system we are participating in building.
>> The feedback they provided has been driving most of the changes that have been made and the
> overall direction of the development, as they should. They continue to review and give feedback
> that we triage and address.
> Well, clearly some "key" CMIP5 scientists are feeling left out of this process. I think Cecelia's
> idea is excellent and we should act on it. The requirements matrix has had input from the UK Met
> Office and I'm confident they would like to be involved in something like a feedback panel.
>
> We should give credit to the P2P team that they are putting together plenty of documentation on the
> ESGF wiki and the bug tracker is working well for developers -- I doubt many users know about it
> though! Gavin is right that keeping the federation status up to date manually is difficult and
> arduous. The P2P status system will make a big difference.
>
> However the content on esgf.org is developer focussed. Scientists and data managers at the CMIP5
> modelling centres are having to struggle with the current system and feel the ground shifting under
> their feet without any say in what's going on. If PCMDI want to deploy the P2P system within weeks
> then the wider CMIP5 users, not just those at PCMDI, need to be involved. Their work is critically
> dependent on what system we put in place.
>
>
>> 4. Update the PCMDI gateway so that it is at least to 1.3.4
>
>> and is in the best shape it can be given the circumstances.
>> I think this has been done.
> I thought it had too but it says "1.3.2" on pcmdi3 right now.
>
>
>> We will also be using the website, our blog, twitter page (I know...
>> I know...), and RSS feed notification as conduits for disseminating information (news, updates,
> etc.) to the community.
> What Gavin is hinting at here is that the ESGF blog has been a shell since the beginning. It's
> been there some 2 years but never had more than a couple of posts on it. I know where he's coming
> from here: for developers it's a lot more fun to setup a blog than keep it current and Gavin has a
> lot on his shoulders. I think this illustrates that organising ESGF should involve a broader range
> of people than the core developers.
>
>
> Thanks for your attention,
> Stephen.
>
> ---
> Stephen Pascoe +44 (0)1235 445980
> Centre of Environmental Data Archival
> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
>
> From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of Gavin M.
> Bell
> Sent: 21 January 2012 09:00
> To: Cecelia DeLuca
> Cc: Cinquini, Luca (3880); go-essp-tech at ucar.edu
> Subject: Re: [Go-essp-tech] Plan
>
> Hi Cecilia et. al.
>
> On 1/20/12 1:56 PM, Cecelia DeLuca wrote:
>
> Thanks Luca.
>
>
>
> Here is a plan. I recognize that it is stupid and probably
>
> counterproductive to continue making suggestions. Unfortunately
>
> that does not seem to stop me. I think these things should
>
> not be put off and would not take very long.
> No no no... not at all... certainly it is never stupid and not counterproductive (especially coming
> from you).
> Your plan is clearly cogent and sound - the devil is in the details. I tried to not mince too finely
> on points in the interleaved response below. Many of the items in this plan are well underway and
> others are burgeoning. It is always good to have a plan. :-)
>
>
>
>
> 1. Do the ESGF demo next Tues from 8-9.
> nod
>
>
> 2. Have the metrics discussion from 9-10.
> What kind of metrics? (pardon if this is a silly question - just want clarification "metrics" means
> many things to many people) Or, do you mean matrix?
>
>
> 3. Start putting together a list of users whose input
>
> you think would be useful. This is a good idea even if they
>
> only ever end up reviewing the p2p system.
> Here at PCMDI we have had a handful of key scientists looking at, using and critiquing the P2P
> system we are participating in building. The feedback they provided has been driving most of the
> changes that have been made and the overall direction of the development, as they should. They
> continue to review and give feedback that we triage and address.
>
>
> 4. Update the PCMDI gateway so that it is at least to 1.3.4
>
> and is in the best shape it can be given the circumstances.
> I think this has been done.
>
>
> 5. Update the wiki on the ESGF site so that the
>
> status of all gateways and nodes is current.
> Yes, we should have a way to view the overall federation status, but I am not convinced that the
> manual task of updating a wiki is quite the best use of resources and time. In terms of the P2P
> system, the mechanics of the system itself would obviate this kind of manual reporting. Not to
> mince things too finely... yes, indeed having this information would be useful.
>
>
> 6. Identify people and a schedule for periodically testing the
>
> status of the federation and keeping that status on the
>
> wiki up to date. Create a minimum set of federation-level
>
> tests and post them. This can be really minimal but it
>
> is important to start and create a routine. Useful both
>
> now and later.
> As mentioned, there is a small group of testers, both programmers and scientists - certainly we can
> and will expand the group of test (vanguard) users. Luca has put together documentation on the
> minimal set of federation tests as you mentioned. Also, there will be a series of automated tests
> that should provide a baseline for the system mechanics. John has been emphasizing testing and is
> establishing a testing working group to focus on addressing this issue.
>
> The idea is that everyone should use the system and post bugs when they find them. And even better
> yet fix them ;-). IMHO - First things first... it is key to have a core from which to grow. This
> core is what has been established and to be demonstrated (the ESGF P2P Node Stack).
>
>
> 7. Create a website where known bugs of the production system
>
> that affect users are listed, in terms that users
>
> can understand. Make users aware of it.
> nod
> Currently we are using our bug tracking system (http://esgf.org/bugzilla/) and mailing lists
> (esgf-node-dev, esgf-node-user) to get feedback from users.
> We will also be using the website, our blog, twitter page (I know... I know...), and RSS feed
> notification as conduits for disseminating information (news, updates, etc.) to the community.
>
>
>
>
> Bonus points:
>
> 8. Get together a management body with key stakeholders so
>
> that decisions are backed up and can be communicated more
>
> easily.
>
>
> Nod, the governance structure will be formalized and posted.... among other things.
>
>
> Maybe some of this is already done. (Going to hide now.)
>
>
> nod
>
>
> Cecelia
>
>
>
> ps. I looked up polemicize and tried not to.
>
>
> :-)
> I personally don't eschew polemics, such engagements can lead to quite fruitful ends - though they
> do require a thick skin.
>
> As I mentioned... first things first... and that is to meet on Tuesday and introduce the P2P system
> to the go-essp community (again).... then we go from there. IMHO
>
>
> On 1/20/2012 11:37 AM, Cinquini, Luca (3880) wrote:
>
> Hi Cecilia,
>
> certainly, as we discussed many times, feedback from users if of the outmost important, and
> being responsive to the that feedback is even more important:)
>
>
>
> This can take the form of scientists that use the system and offer their suggestions freely, or of a
> formal users review board for ESGF. We probably need both. And off course developers
>
> need to estimate what is possible and what's not, and in which time frame. So we should devote some
> of the future GO-ESSP calls to put something like this in place. Looks like a busy few months
> ahead..
>
>
>
> thanks, Luca
>
>
>
> On Jan 20, 2012, at 11:26 AM, Cecelia DeLuca wrote:
>
>
>
> Luca, maybe I'm wrong but this list seems overwhelmingly populated by
>
> developers,
>
> not by scientist users.
>
>
>
> It would be good to get an "external" scientist user group together.
>
> Jennifer and
>
> Gary are a start, but it also seems important to include some scientists
>
> who are not as
>
> closely tied to infrastructure as the folks who would normally show up here.
>
>
>
> I do think that, as tough as it might be, Stephen's list also needs to
>
> be reviewed by
>
> developers. That discussion must be concerned with not just the current
>
> state of the system, which
>
> is what users see, but also the future of the system, and developers are
>
> best positioned
>
> to comment on that.
>
>
>
> -- Cecelia
>
>
>
>
>
> On 1/20/2012 10:12 AM, Cinquini, Luca (3880) wrote:
>
> Hi Gary,
>
> I think everybody on this list is invited - just like it happened for the previous one. I hope
> you can be there :).
>
> thanks, Luca
>
>
>
> On Jan 20, 2012, at 9:58 AM, Gary Strand wrote:
>
>
>
> How many users of the overall system, outside of the developers, have been invited to these demos?
>
>
>
> On Fri Jan 20, 2012, at 9:57 AM, Sylvia Murphy wrote:
>
>
>
> Hi Everyone,
>
>
>
> It seems to me that this decsion is not going to be made through demos, but by a careful analysis of
> the criteria for CMIP5 and how each software stack stands up to that criteria.
>
>
>
> We need an evaluation and testing strategy, preferably involving key stakeholders who can safely
> examine both packages.
>
>
>
> Deciding on how to do this, the timelines etc seems to me to be our highest priority.
>
>
>
> 2 cents
>
>
>
>
>
> Sylvia
>
> On Fri, Jan 20, 2012 at 9:49 AM, Cinquini, Luca
> (3880)<Luca.Cinquini at jpl.nasa.gov><mailto:Luca.Cinquini at jpl.nasa.gov> wrote:
>
> Hi Stephen,
>
> I think it will be extremely important to give P2P and Gateway 2.0 the exact same amount of
> time to showcase their capabilities, otherwise how is CMIP5 going to make an informed decision ?
>
> Yes, P2P has been demonstrated one time before, but the gateway has been demoed many more times. So
> I beg to disagree with you... off course, ultimately the decision of what the agenda should be lies
> with the GO-ESSP PIs.
>
> thanks, Luca
>
>
>
> On Jan 20, 2012, at 9:41 AM,<stephen.pascoe at stfc.ac.uk><mailto:stephen.pascoe at stfc.ac.uk> wrote:
>
>
>
> Hi Luca,
>
>
>
> I really don't think this is a good use of time. I know P2P is improving all the time but I've seen
> a number of demos now. It would be more effective to go through the matrix and you describe how P2P
> meets the requirements or how you plan to meet them.
>
>
>
> Depending on who's on the call, we may be in a position to discuss the GW2/P2P deployment strategy
> for CMIP5 in more detail. Therefore, I wouldn't want to schedule something that is likely to take
> up most of the hour.
>
>
>
> Stephen.
>
>
>
> ---
>
> Stephen Pascoe +44 (0)1235 445980
>
> Centre of Environmental Data Archival
>
> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
>
>
>
>
>
> -----Original Message-----
>
> From: Cinquini, Luca (3880) [mailto:Luca.Cinquini at jpl.nasa.gov]
>
> Sent: 20 January 2012 14:29
>
> To: Eric Nienhouse
>
> Cc: Pascoe, Stephen (STFC,RAL,RALSP); go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>
>
> Subject: Re: [Go-essp-tech] CMIP5 Web interface requirements matrix
>
>
>
> Hi Stephen,
>
> thanks for the matrix...
>
> I would like to suggest that we start with a full demo of the P2P system on Tuesday. The gateway 2.0
> demo took a full hour - we will try to make the P2P demo shorter, but if it ends up taking a full
> hour, we can always discuss the matrix at the next call.
>
>
>
> thanks, Luca
>
>
>
> On Jan 20, 2012, at 7:23 AM, Eric Nienhouse wrote:
>
>
>
> Hi Stephen,
>
>
>
> Thank you for this latest version of the matrix. It will be good to
>
> discuss it further on Tuesday. I would like to understand the ranking
>
> numbers and planned evaluation process and hope this is something we can
>
> address on the call.
>
>
>
> Attached is an update to the previous version I was just about to send
>
> as your recent email arrived. Please note 4 highlighted rows. (2 fall
>
> in the non-functional category.) I trust they can be incorporated and
>
> discussed.
>
>
>
> Indeed the CMIP5 system(s) are under continued development and
>
> enhancement and the matrix inherently has some relation to the
>
> architectural choices up to this point. Sticking with the current
>
> replication section for the time being seems appropriate.
>
>
>
> Thanks,
>
>
>
> -Eric
>
>
>
> stephen.pascoe at stfc.ac.uk<mailto:stephen.pascoe at stfc.ac.uk> wrote:
>
> Dear all,
>
>
>
>
>
>
>
> I have merged Luca's suggestions into the CMIP5 GUI requirements
>
> matrix and reorganised it substantially. Apart from those items that
>
> are architecture-specific I include most of them, sometimes under a
>
> different section.
>
>
>
>
>
>
>
> I'm aware there are some architectural assumptions embedded in this
>
> sheet. For instance, there has been a recent discussion about whether
>
> master/replica is the right model for depicting replicas. It's a pity
>
> we're having this discussion now! For the moment I've kept the
>
> replication section as-is.
>
>
>
>
>
>
>
> All scores are my opinion based on variable information. Sometimes
>
> I've assumed something works or is planned in P2P without verifying
>
> it, sometimes I've assumed it doesn't after a quick test. Where there
>
> is a "?" anywhere it would be really useful for the developers to
>
> contribute. To this end I'd like to go through the sheet on Tuesday,
>
> or make a start at least, so that I can clarify what some items mean.
>
>
>
>
>
>
>
> I know a lot of water has flowed under the bridge since I started this
>
> process. No matter what strategic decisions the CMIP5 cores centres
>
> take on our future deployments I hope this matrix will be useful in
>
> focusing on the core CMIP5 requirements.
>
>
>
>
>
>
>
> Thanks,
>
>
>
> Stephen.
>
>
>
>
>
>
>
> ---
>
>
>
> Stephen Pascoe +44 (0)1235 445980
>
>
>
> Centre of Environmental Data Archival
>
>
>
> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
>
>
>
>
>
>
>
> *From:* Pascoe, Stephen (STFC,RAL,RALSP)
>
> *Sent:* 10 January 2012 14:32
>
> *To:* go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>
>
> *Subject:* CMIP5 Web interface requirements matrix
>
>
>
>
>
>
>
> Dear all,
>
>
>
>
>
>
>
> To help the CMIP5 centres plan our upgrade to the next major release
>
> of the CMIP5 archive system BADC has been collating a
>
>
>
> spreadsheet of web interface requirements with input from PCMDI, DKRZ
>
> and MOHC (see attached). We hope to use this sheet as a tool to plan
>
> migration of the CMIP5 web interface at the main CMIP5 centres: PCMDI,
>
> BADC and DKRZ.
>
>
>
>
>
>
>
> The sheet is still work in progress and there is lots to discuss and
>
> clarify. I have begun to suggest scores for the three systems being
>
> considered, Gateway 1.3.4, Gateway 2.0 and P2P, but these are
>
> speculative at this stage, particularly for the P2P system with which
>
> I have the least experience.
>
>
>
>
>
>
>
> If there is time I'd like to briefly introduce the sheet at the
>
> GO-ESSP telco today and schedule a time for a more thorough discussion.
>
>
>
>
>
>
>
> Cheers,
>
>
>
> Stephen.
>
>
>
>
>
>
>
> ---
>
>
>
> Stephen Pascoe +44 (0)1235 445980
>
>
>
> Centre of Environmental Data Archival
>
>
>
> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
>
>
>
>
>
>
>
>
>
> --
>
> Scanned by iCritical.
>
>
>
>
>
> ------------------------------------------------------------------------
>
>
>
> _______________________________________________
>
> GO-ESSP-TECH mailing list
>
> GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
>
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
> <CMIP5_GUI_Requirements_v20120110_ejn.xlsx>_______________________________________________
>
> GO-ESSP-TECH mailing list
>
> GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
>
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
> --
>
> Scanned by iCritical.
>
> _______________________________________________
>
> GO-ESSP-TECH mailing list
>
> GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
>
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
>
>
>
>
>
>
> --
>
> ******************************************
>
> Sylvia Murphy
>
> NESII/CIRES/NOAA Earth System Research Laboratory
>
> 325 Broadway, Boulder CO 80305
>
> Time Zone: U.S. Mountain
>
> Web: http://www.esrl.noaa.gov/nesii/
>
> Email: sylvia.murphy at noaa.gov<mailto:sylvia.murphy at noaa.gov>
>
> Phone: 303-497-7753
>
>
>
>
>
> _______________________________________________
>
> GO-ESSP-TECH mailing list
>
> GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
>
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
> Gary Strand
>
> strandwg at ucar.edu<mailto:strandwg at ucar.edu>
>
>
>
>
>
>
>
> _______________________________________________
>
> GO-ESSP-TECH mailing list
>
> GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
>
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
> _______________________________________________
>
> GO-ESSP-TECH mailing list
>
> GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
>
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
>
>
> _______________________________________________
>
> GO-ESSP-TECH mailing list
>
> GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
>
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
>
>
> --
>
> Gavin M. Bell
>
> --
>
>
>
> "Never mistake a clear view for a short distance."
>
> -Paul Saffo
>
>
>
> --
> Scanned by iCritical.
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20120121/141ec6bd/attachment.html
>
> ------------------------------
>
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
>
> End of GO-ESSP-TECH Digest, Vol 35, Issue 59
> ********************************************
>
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
More information about the GO-ESSP-TECH
mailing list