[Go-essp-tech] GO-ESSP-TECH Digest, Vol 35, Issue 59

Sat Jan 21 04:24:34 MST 2012

Finally . . .

"Scientists and data managers at the CMIP5 modelling centres are having to struggle with the current
system and feel the ground shifting under their feet without any say in what's going on.  If PCMDI
want to deploy the P2P system within weeks then the wider CMIP5 users, not just those at PCMDI, need
to be involved.  Their work is critically dependent on what system we put in place".

+ a few thousand on that.  As a minimum all the groups submitting data to the system should be
invited to have some representation on this user panel, and there should be some representatives
from the wider community too.  How can you be responsive to the international community need without
at least seeking their views as a group.

Regards

Mark

--------------------------------------------------------------
Dr Mark Elkington
Elmswood Barn, Ogwell, Newton Abbot, Devon TQ12 6AF
tel: 01626 212545  mob: 07804 795826
email:  mark.elkington at blueyonder.co.uk

-----Original Message-----
From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of
go-essp-tech-request at ucar.edu
Sent: 21 January 2012 10:32
To: go-essp-tech at ucar.edu
Subject: GO-ESSP-TECH Digest, Vol 35, Issue 59

Send GO-ESSP-TECH mailing list submissions to
	go-essp-tech at ucar.edu

To subscribe or unsubscribe via the World Wide Web, visit
	http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
or, via email, send a message with subject or body 'help' to
	go-essp-tech-request at ucar.edu

You can reach the person managing the list at
	go-essp-tech-owner at ucar.edu

When replying, please edit your Subject line so it is more specific than "Re: Contents of
GO-ESSP-TECH digest..."

Today's Topics:

   1. Re: Plan (stephen.pascoe at stfc.ac.uk)

----------------------------------------------------------------------

Message: 1
Date: Sat, 21 Jan 2012 10:31:12 +0000
From: <stephen.pascoe at stfc.ac.uk>
Subject: Re: [Go-essp-tech] Plan
To: <gavin at llnl.gov>, <cecelia.deluca at noaa.gov>
Cc: Luca.Cinquini at jpl.nasa.gov, go-essp-tech at ucar.edu
Message-ID:
	<4C353E6E4A08AE4792B350DAA392B5211C980313 at EXCHMBX01.fed.cclrc.ac.uk>
Content-Type: text/plain; charset="us-ascii"

Briefly getting back to the Agenda for Tuesday.  I can make an extended 2 hour telco but the order
we do things in will depend on who can attend  9-10 PTD.  That is 6-7pm CET so people from DKRZ,
IPSL, SMI etc. may not be able to make it.  As well as the requirements matrix I'd like to cover
setting up a scientific feedback panel as suggested by Cecelia and supported by Balaji.

Inline comments follow, reordered for relevance.

> 2. Have the metrics discussion from 9-10.
> What kind of metrics?
I think Cecelia means the requirements matrix aka spreadsheet I circulated

> Bonus points:

> 8. Get together a management body with key stakeholders so that 
> decisions are backed up and can be

> communicated more easily.

>
> Nod, the governance structure will be formalized and posted.... among 
> other things
Gavin, who are the stakeholders going to be?  This decision should be the result of a dialog under
the auspices of GO-ESSP not just "posted".

> 3. Start putting together a list of users whose input you think would 
> be useful.  This is a good idea even

> if they only ever end up reviewing the p2p system.
> Here at PCMDI we have had a handful of key scientists looking at, 
> using and critiquing the P2P system we are participating in building.  
> The feedback they provided has been driving most of the changes that have been made and the
overall direction of the development, as they should.  They continue to review and give feedback
that we triage and address.
Well, clearly some "key" CMIP5 scientists are feeling left out of this process.  I think Cecelia's
idea is excellent and we should act on it.  The requirements matrix has had input from the UK Met
Office and I'm confident they would like to be involved in something like a feedback panel.

We should give credit to the P2P team that they are putting together plenty of documentation on the
ESGF wiki and the bug tracker is working well for developers -- I doubt many users know about it
though!  Gavin is right that keeping the federation status up to date manually is difficult and
arduous.  The P2P status system will make a big difference.

However the content on esgf.org is developer focussed.  Scientists and data managers at the CMIP5
modelling centres are having to struggle with the current system and feel the ground shifting under
their feet without any say in what's going on.  If PCMDI want to deploy the P2P system within weeks
then the wider CMIP5 users, not just those at PCMDI, need to be involved.  Their work is critically
dependent on what system we put in place.

> 4. Update the PCMDI gateway so that it is at least to 1.3.4

> and is in the best shape it can be given the circumstances.
> I think this has been done.
I thought it had too but it says "1.3.2" on pcmdi3 right now.

>  We will also be using the website, our blog, twitter page (I know... 
> I know...), and RSS feed notification as conduits for disseminating information (news, updates,
etc.) to the community.
What Gavin is hinting at here is that the ESGF blog has  been a shell since the beginning.  It's
been there some 2 years but never had more than a couple of posts on it.  I know where he's coming
from here: for developers it's a lot more fun to setup a blog than keep it current and Gavin has a
lot on his shoulders.  I think this illustrates that organising ESGF should involve a broader range
of people than the core developers.

Thanks for your attention,
Stephen.

---
Stephen Pascoe  +44 (0)1235 445980
Centre of Environmental Data Archival
STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK

From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of Gavin M.
Bell
Sent: 21 January 2012 09:00
To: Cecelia DeLuca
Cc: Cinquini, Luca (3880); go-essp-tech at ucar.edu
Subject: Re: [Go-essp-tech] Plan

Hi Cecilia et. al.

On 1/20/12 1:56 PM, Cecelia DeLuca wrote:

Thanks Luca.

Here is a plan.  I recognize that it is stupid and probably

counterproductive to continue making suggestions.  Unfortunately

that does not seem to stop me.  I think these things should

not be put off and would not take very long.
No no no... not at all... certainly it is never stupid and not counterproductive (especially coming
from you).
Your plan is clearly cogent and sound - the devil is in the details. I tried to not mince too finely
on points in the interleaved response below.  Many of the items in this plan are well underway and
others are burgeoning.  It is always good to have a plan. :-)

1. Do the ESGF demo next Tues from 8-9.
nod

2. Have the metrics discussion from 9-10.
What kind of metrics? (pardon if this is a silly question - just want clarification "metrics" means
many things to many people) Or, do you mean matrix?

3. Start putting together a list of users whose input

you think would be useful.  This is a good idea even if they

only ever end up reviewing the p2p system.
Here at PCMDI we have had a handful of key scientists looking at, using and critiquing the P2P
system we are participating in building.  The feedback they provided has been driving most of the
changes that have been made and the overall direction of the development, as they should.  They
continue to review and give feedback that we triage and address.

4. Update the PCMDI gateway so that it is at least to 1.3.4

and is in the best shape it can be given the circumstances.
I think this has been done.

5. Update the wiki on the ESGF site so that the

status of all gateways and nodes is current.
Yes, we should have a way to view the overall federation status, but I am not convinced that the
manual task of updating a wiki is quite the best use of resources and time.  In terms of the P2P
system, the mechanics of the system itself would obviate this kind of manual reporting.  Not to
mince things too finely... yes, indeed having this information would be useful.

6. Identify people and a schedule for periodically testing the

status of the federation and keeping that status on the

wiki up to date.  Create a minimum set of federation-level

tests and post them.  This can be really minimal but it

is important to start and create a routine.  Useful both

now and later.
As mentioned, there is a small group of testers, both programmers and scientists - certainly we can
and will expand the group of test (vanguard) users.  Luca has put together documentation on the
minimal set of federation tests as you mentioned.  Also, there will be a series of automated tests
that should provide a baseline for the system mechanics.  John has been emphasizing testing and is
establishing a testing working group to focus on addressing this issue.

The idea is that everyone should use the system and post bugs when they find them.  And even better
yet fix them ;-).  IMHO - First things first... it is key to have a core from which to grow.  This
core is what has been established and to be demonstrated (the ESGF P2P Node Stack).

7. Create a website where known bugs of the production system

that affect users are listed, in terms that users

can understand.  Make users aware of it.
nod
Currently we are using our bug tracking system (http://esgf.org/bugzilla/) and mailing lists
(esgf-node-dev, esgf-node-user) to get feedback from users.
We will also be using the website, our blog, twitter page (I know... I know...), and RSS feed
notification as conduits for disseminating information (news, updates, etc.) to the community.

Bonus points:

8. Get together a management body with key stakeholders so

that decisions are backed up and can be communicated more

easily.

Nod, the governance structure will be formalized and posted.... among other things.

Maybe some of this is already done.  (Going to hide now.)

nod

Cecelia

ps.  I looked up polemicize and tried not to.

:-)
I personally don't eschew polemics, such engagements can lead to quite fruitful ends - though they
do require a thick skin.

As I mentioned... first things first... and that is to meet on Tuesday and introduce the P2P system
to the go-essp community (again).... then we go from there. IMHO

On 1/20/2012 11:37 AM, Cinquini, Luca (3880) wrote:

Hi Cecilia,

      certainly, as we discussed many times, feedback from users if of the outmost important, and
being responsive to the that feedback is even more important:)

This can take the form of scientists that use the system and offer their suggestions freely, or of a
formal users review board for ESGF. We probably need both. And off course developers

need to estimate what is possible and what's not, and in which time frame. So we should devote some
of the future GO-ESSP calls to put something like this in place. Looks like a busy few months
ahead..

thanks, Luca

On Jan 20, 2012, at 11:26 AM, Cecelia DeLuca wrote:

Luca, maybe I'm wrong but this list seems overwhelmingly populated by

developers,

not by scientist users.

It would be good to get an "external" scientist user group together.

Jennifer and

Gary are a start, but it also seems important to include some scientists

who are not as

closely tied to infrastructure as the folks who would normally show up here.

I do think that, as tough as it might be, Stephen's list also needs to

be reviewed by

developers.  That discussion must be concerned with not just the current

state of the system, which

is what users see, but also the future of the system, and developers are

best positioned

to comment on that.

-- Cecelia

On 1/20/2012 10:12 AM, Cinquini, Luca (3880) wrote:

Hi Gary,

    I think everybody on this list is invited - just like it happened for the previous one. I hope
you can be there :).

thanks, Luca

On Jan 20, 2012, at 9:58 AM, Gary Strand wrote:

How many users of the overall system, outside of the developers, have been invited to these demos?

On Fri Jan 20, 2012, at 9:57 AM, Sylvia Murphy wrote:

Hi Everyone,

It seems to me that this decsion is not going to be made through demos, but by a careful analysis of
the criteria for CMIP5 and how each software stack stands up to that criteria.

We need an evaluation and testing strategy, preferably involving key stakeholders who can safely
examine both packages.

Deciding on how to do this, the timelines etc seems to me to be our highest priority.

2 cents

Sylvia

On Fri, Jan 20, 2012 at 9:49 AM, Cinquini, Luca
(3880)<Luca.Cinquini at jpl.nasa.gov><mailto:Luca.Cinquini at jpl.nasa.gov>   wrote:

Hi Stephen,

       I think it will be extremely important to give P2P and Gateway 2.0 the exact same amount of
time to showcase their capabilities, otherwise how is CMIP5 going to make an informed decision ?

Yes, P2P has been demonstrated one time before, but the gateway has been demoed many more times. So
I beg to disagree with you... off course, ultimately the decision of what the agenda should be lies
with the GO-ESSP PIs.

thanks, Luca

On Jan 20, 2012, at 9:41 AM,<stephen.pascoe at stfc.ac.uk><mailto:stephen.pascoe at stfc.ac.uk>   wrote:

Hi Luca,

I really don't think this is a good use of time.  I know P2P is improving all the time but I've seen
a number of demos now.  It would be more effective to go through the matrix and you describe how P2P
meets the requirements or how you plan to meet them.

Depending on who's on the call, we may be in a position to discuss the GW2/P2P deployment strategy
for CMIP5 in more detail.  Therefore, I wouldn't want to schedule something that is likely to take
up most of the hour.

Stephen.

---

Stephen Pascoe  +44 (0)1235 445980

Centre of Environmental Data Archival

STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK

-----Original Message-----

From: Cinquini, Luca (3880) [mailto:Luca.Cinquini at jpl.nasa.gov]

Sent: 20 January 2012 14:29

To: Eric Nienhouse

Cc: Pascoe, Stephen (STFC,RAL,RALSP); go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>

Subject: Re: [Go-essp-tech] CMIP5 Web interface requirements matrix

Hi Stephen,

      thanks for the matrix...

I would like to suggest that we start with a full demo of the P2P system on Tuesday. The gateway 2.0
demo took a full hour - we will try to make the P2P demo shorter, but if it ends up taking a full
hour, we can always discuss the matrix at the next call.

thanks, Luca

On Jan 20, 2012, at 7:23 AM, Eric Nienhouse wrote:

Hi Stephen,

Thank you for this latest version of the matrix.  It will be good to

discuss it further on Tuesday.  I would like to understand the ranking

numbers and planned evaluation process and hope this is something we can

address on the call.

Attached is an update to the previous version I was just about to send

as your recent email arrived.  Please note 4 highlighted rows.  (2 fall

in the non-functional category.)  I trust they can be incorporated and

discussed.

Indeed the CMIP5 system(s) are under continued development and

enhancement and the matrix inherently has some relation to the

architectural choices up to this point.  Sticking with the current

replication section for the time being seems appropriate.

Thanks,

-Eric

stephen.pascoe at stfc.ac.uk<mailto:stephen.pascoe at stfc.ac.uk> wrote:

Dear all,

I have merged Luca's suggestions into the CMIP5 GUI requirements

matrix and reorganised it substantially.  Apart from those items that

are architecture-specific I include most of them, sometimes under a

different section.

I'm aware there are some architectural assumptions embedded in this

sheet.  For instance, there has been a recent discussion about whether

master/replica is the right model for depicting replicas.  It's a pity

we're having this discussion now!  For the moment I've kept the

replication section as-is.

All scores are my opinion based on variable information.  Sometimes

I've assumed something works or is planned in P2P without verifying

it, sometimes I've assumed it doesn't after a quick test.  Where there

is a "?" anywhere it would be really useful for the developers to

contribute.  To this end I'd like to go through the sheet on Tuesday,

or make a start at least, so that I can clarify what some items mean.

I know a lot of water has flowed under the bridge since I started this

process.  No matter what strategic decisions the CMIP5 cores centres

take on our future deployments I hope this matrix will be useful in

focusing on the core CMIP5 requirements.

Thanks,

Stephen.

---

Stephen Pascoe  +44 (0)1235 445980

Centre of Environmental Data Archival

STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK

*From:* Pascoe, Stephen (STFC,RAL,RALSP)

*Sent:* 10 January 2012 14:32

*To:* go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>

*Subject:* CMIP5 Web interface requirements matrix

Dear all,

To help the CMIP5 centres plan our upgrade to the next major release

of the CMIP5 archive system BADC has been collating a

spreadsheet of web interface requirements with input from PCMDI, DKRZ

and MOHC (see attached).  We hope to use this sheet as a tool to plan

migration of the CMIP5 web interface at the main CMIP5 centres: PCMDI,

BADC and DKRZ.

The sheet is still work in progress and there is lots to discuss and

clarify.  I have begun to suggest scores for the three systems being

considered, Gateway 1.3.4, Gateway 2.0 and P2P, but these are

speculative at this stage, particularly for the P2P system with which

I have the least experience.

If there is time I'd like to briefly introduce the sheet at the

GO-ESSP telco today and schedule a time for a more thorough discussion.

Cheers,

Stephen.

---

Stephen Pascoe  +44 (0)1235 445980

Centre of Environmental Data Archival

STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK

--

Scanned by iCritical.

------------------------------------------------------------------------

_______________________________________________

GO-ESSP-TECH mailing list

GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>

http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

<CMIP5_GUI_Requirements_v20120110_ejn.xlsx>_______________________________________________

GO-ESSP-TECH mailing list

GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>

http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

--

Scanned by iCritical.

_______________________________________________

GO-ESSP-TECH mailing list

GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>

http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

--

******************************************

Sylvia Murphy

NESII/CIRES/NOAA Earth System Research Laboratory

325 Broadway, Boulder CO 80305

Time Zone: U.S. Mountain

Web: http://www.esrl.noaa.gov/nesii/

Email: sylvia.murphy at noaa.gov<mailto:sylvia.murphy at noaa.gov>

Phone: 303-497-7753

_______________________________________________

GO-ESSP-TECH mailing list

GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>

http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

Gary Strand

strandwg at ucar.edu<mailto:strandwg at ucar.edu>

_______________________________________________

GO-ESSP-TECH mailing list

GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>

http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

_______________________________________________

GO-ESSP-TECH mailing list

GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>

http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

_______________________________________________

GO-ESSP-TECH mailing list

GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>

http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

--

Gavin M. Bell

--

 "Never mistake a clear view for a short distance."

               -Paul Saffo

--
Scanned by iCritical.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20120121/141ec6bd/attachment.html 

------------------------------

_______________________________________________
GO-ESSP-TECH mailing list
GO-ESSP-TECH at ucar.edu
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

End of GO-ESSP-TECH Digest, Vol 35, Issue 59
********************************************