[Go-essp-tech] [esg-gateway-dev] [ESG-CET] CMIP5 Federation Testing Outline

stephen.pascoe at stfc.ac.uk stephen.pascoe at stfc.ac.uk
Mon Nov 22 04:52:58 MST 2010 

Hi everyone,

I think there have been some very good points made here, I'll try and summarise and suggest a way forward.

We have 2 separate test plans: The Security System Integration Test Plan (SSITP for short) and the Cmip5Status/Test wiki page on esgf.org.

The Security test plan is an excellent comprehensive starting point but, as Estani points out, it is not just tests for verifying the deployment of what we have now as some tests will never pass with the current software.

The wiki page has a large overlap with the workflow tests in SSITP.  It covers a few extra things like replication and has a more high-level deployment matrix.  It is a lot less detailed than SSITP.

I really like the SSITP's structure of integration workflow tests building on more low-level unit-tests.  It exposes the size of the task of integration.  I expect if we try to do high-level integration tests without the low-level ones we'll always be chasing our tail.  I also agree with Gavin that we have to automate as much of this as possible.

However, we have to be pragmatic and verify we've deployed what we have now properly before we test the requirements for the perfect system.

Suggested actions:

A. Bring the wiki closer to SSITP.

 1. Update the wiki merge in SSITP workflow tests not currently covered.
 2. Identify those requirements in SSITP could be met by the current software and gather them into the wiki.
 3. Update the wiki deployment matrix to include all/most of what is in SSITP.

B. Use SSITP as a next-release requirements document.  I'm agnostic whether we keep it in GoogleDocs or 
   transfer it to the wiki.  Provided we are clearly separated from the tests of this release.

C. Allocate responsibility for each category of test.  We need to spread out the work.

 1. Someone automates low-level tests
 2. Someone to add detail to workflow tests that can't be automated.
 3. Someone to be gatekeeper of the conformance matrix so we know the test has been consistently applied.

We can discuss this further at the telco tomorrow.

Cheers,
Stephen.

-----Original Message-----
From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of Estanislao Gonzalez
Sent: 22 November 2010 10:41
To: Rachana Ananthakrishnan
Cc: esg-cet at earthsystemgrid.org; go-essp-tech at ucar.edu; esg-gateway-dev at earthsystemgrid.org; Luca.Cinquini at jpl.nasa.gov
Subject: Re: [Go-essp-tech] [esg-gateway-dev] [ESG-CET] CMIP5 Federation Testing Outline


  Hi Rachana et al.,

I agree with you as you all have already put considerable time into 
creating that document. But I must say I don't quite follow it. To me it 
looks like a mixture of internal system tests (mostly), federation tests 
and a quite long list of requirements.

For example AFAIK practically no test in chapter 3 "Test Cross Cutting 
Concerns" will succeed, at least not with the current system versions. 
So those look more like requirements for a future release than tests. 
For instance:
list of CAs and CRLs -> Do we set up a revocation list? where?
Public key bit size - 2048, SHA2 by next year (2011) -> the installer is 
not doing this. PCMDI's cert is 1024, SHA1 and the one we got signed is 
2048, SHA1. I'm pretty sure most systems out there will not pass this.
Rejects connection from peer where no certificate is provided -> can't 
happen as we have set clientAuth="want", afaik no mechanism is in place 
to re-check SSL certificates. Or is this referring to something else?
...

I see the document as the starting point for a test-driven next 
iteration more than a compendium of tests preparing the federation for 
cmip5. But maybe I'm just missing information and we have already most 
of these features (or should have, I'm quite sure our systems here don't).

In any case there are indeed many hints on what should be tested. I 
think we should move forward from there and expand the tests regarding 
today's stand by providing the "how". And this should be done IMO at the 
esgf.org/wiki as you've mentioned.

I'm thinking we could list all tests in the main tests page 
(http://esgf.org/wiki/Cmip5Status/Tests) and create a page after each of 
these which contains information on how to perform that test, and 
possibly notes on what to expect or questions regarding it, and a list 
of already performed tests with it's result and notes. As an example 
I've done the first one so you can better see what I mean.

The procedure will be pretty simple:
1) List the test in the corresponding group and embed it with the 
[[/Perform this test]] wiki syntax for a "Perform this test" name. This 
allows the creation of sub-page.
2) Click on it and select TestsTemplate to create the page.
3) Fill the template as far as you can.

Well that should do. The sooner we start the better.

Thanks,
Estani


On 11/19/2010 11:38 PM, Rachana Ananthakrishnan wrote:
> I second this - we put in significant time to identify these tests,
> and it would be good to use the material and prioritize them for this
> purpose.
>
> Eric, what is the best way to contribute here? I can revisit the
> document produced at the prior security meeting, and augment the wiki
> page. If I understand from this thread this is the final resting
> place: http://esgf.org/wiki/Cmip5Status/Tests? If you have other
> suggestions on how best we can contribute to these tests, please let
> me know.
>
> Rachana
>
> On Nov 18, 2010, at 10:05 AM,<philip.kershaw at stfc.ac.uk>  <philip.kershaw at stfc.ac.uk
>   >  wrote:
>
>> Hi Eric,
>>
>> There's actually also the document we started at the ANL security
>> meeting in Sept. :) ...
>>
>> https://docs.google.com/document/d/17PcZqDs0zONyUa4A3Z835Rt-_ePsEBKHWWP8XkawEow/edit?hl=en
>> #
>>
>> This is not visible to all so I've attached a PDF also.  We could
>> lift what we have in there to whatever format doc we want to use to
>> share with: wiki etc.
>>
>> My only reservation about a wiki approach is that you don't lose the
>> concept of some kind of change control.  I think it's important to
>> keep track of versions with a document like this.
>>
>> I think we could also prepare some test tools for system wide
>> tests.  I've attempted to kick this off by preparing a Python test
>> script to probe the SSL endpoints in the federation and check for
>> correct configuration:
>>
>> http://proj.badc.rl.ac.uk/ndg/browser/TI12-security/trunk/esg_system_tests/esg/security/test/system
>>
>> Cheers,
>> Phil
>>
>>> -----Original Message-----
>>> From: esg-gateway-dev-bounces at mailman.earthsystemgrid.org
>>> [mailto:esg-
>>> gateway-dev-bounces at mailman.earthsystemgrid.org] On Behalf Of Eric
>>> Nienhouse
>>> Sent: 18 November 2010 15:52
>>> To: Cinquini, Luca (3880)
>>> Cc: ESG-CET Mail List; go-essp-tech at ucar.edu; esg-gateway-
>>> dev at earthsystemgrid.org
>>> Subject: Re: [esg-gateway-dev] [ESG-CET] CMIP5 Federation Testing
>>> Outline
>>>
>>> Hi Luca, All,
>>>
>>> Thanks for this input.  What I would propose are two "documents" to
>>> support this testing work:
>>>
>>> 1)  An outline of the tests with some detail about how to perform
>>> them,
>>> and what constitutes and acceptable outcome.
>>> 2)  A tracking document to capture the tests themselves with
>>> comments,
>>> dates and test related details as you note below.
>>>
>>> I think a wiki is ideal for the outline (1) above.  I agree
>>> emailing a
>>> document around is too cumbersome.
>>>
>>> I think an issue tracker is ideal for (2) above.  I propose we use
>>> the
>>> existing NCAR Jira system for this.  Once we have a good outline
>>> organized we can then create the necessary Jira tasks to track
>>> progress
>>> and issues.
>>>
>>> Does this sound reasonable?
>>>
>>> Thanks,
>>>
>>> -Eric
>>>
>>> Cinquini, Luca (3880) wrote:
>>>> Hi Eric,
>>>> 	thanks for starting this document, the list of tests to perform
>>> looks like a good start - we might want expand it in a little more
>>> detail, like for example distinguishing between:
>>>> 1) http file download via browser started at the gateway
>>>> 2) http file download via browser started at the TDS page
>>>> 3) http multi-files download via wget script (directly to the TDS)
>>>>
>>>> for both home data node and federated data node.
>>>>
>>>> We might also want to have a matrix table where we list the result
>>>> of
>>> conducting each test between institutions (A,B), with an attached
>>> date
>>> when the test was last performed.
>>>> As for where the document should live, I personally favor the esgf
>>> wiki option, although I could also go with 3) and 4). Exchanging the
>>> document by email is kind of cumbersome IMO.
>>>> thanks, Luca
>>>>
>>>> On Nov 18, 2010, at 7:25 AM, Eric Nienhouse wrote:
>>>>
>>>>
>>>>> Dear ESG Gateway Collaborators,
>>>>>
>>>>> We discussed the need for federation testing in our recent GO-ESSP
>>>>> call.  I would like to continue our test planning discussion and
>>>>> collaboratively develop a document to support this effort.
>>>>>
>>>>> I welcome your input and insights, in particular for testing
>>> scenarios
>>>>> related to security and access control of CMIP5 datasets.  I plan
>>>>> to
>>>>> serve to coordinate our testing efforts, however, I need help from
>>>>> others to develop and document them.
>>>>>
>>>>> I have an draft outline of these key federation tests at the link
>>> below
>>>>> and I would like your input:
>>>>>
>>>>>
>>> https://wiki.ucar.edu/display/esgcet/CMIP5+Federation+Testing+Task+Matr
>>> ix
>>>>> Several options exist to enable collaborative access to this
>>> document.
>>>>> I'd like suggestions from others on the best approach.  Some
>>>>> options
>>> follow:
>>>>> 1)  Exchange this document by email and merge edits (we've done
>>>>> this
>>> in
>>>>> the past)
>>>>> 2)  Move the document to the ESGF wiki (Stephen and Estani have
>>>>> suggested this already.)
>>>>> 3)  Keep it on wiki.ucar.edu and provide logins to all
>>> collaborators.
>>>>> 4)  Move it to Google Docs.
>>>>>
>>>>> There is momentum to use the ESGF wiki as at least one supporting
>>>>> document may already be there (CMIP5 Status of Gateways and
>>> Datanodes.)
>>>>> What do others think would be best?
>>>>>
>>>>> Thanks for your input on this.
>>>>>
>>>>> Kind regards,
>>>>>
>>>>> -Eric
>>>>>
>>>>> Eric Nienhouse wrote:
>>>>>
>>>>>> Dear ESG Gateway Collaborators,
>>>>>>
>>>>>> We're quickly approaching the 1.2.0 ESG Gateway software release.
>>> This
>>>>>> version will provide the basis for the upcoming CMIP5 activity
>>>>>> data
>>>>>> management effort.
>>>>>>
>>>>>> The Gateway 1.2.0 Release Candidate 1 (RC1) is now available for
>>>>>> inter-gateway federation testing.  We encourage all who can to
>>> upgrade
>>>>>> to this version as soon as is reasonably possible. (In particular,
>>> if
>>>>>> your gateway is on the list of "Key Participating Gateways"
>>>>>> below.)
>>>>>> For installation and update details, see the included email below.
>>>>>>
>>>>>> Please note that the Gateway 1.2.0 RC1 requires a corresponding
>>> Data
>>>>>> Node version 1.0.4.0 or greater.  This Gateway upgrade may require
>>> an
>>>>>> update to your Data Node components as well.
>>>>>>
>>>>>> We're in need of significant inter-gateway (eg: federation)
>>> testing.
>>>>>> In order to begin this process, it is important that all
>>> participating
>>>>>> gateways are running a consistent pre-release 1.2.0 version.  The
>>>>>> 1.2.0 RC1 is the recommended version for federation related
>>> testing.
>>>>>> Following are the Key Participating Gateways.  Several have
>>>>>> already
>>>>>> upgraded to the 1.2.0 RC1 version.  If you have not yet upgraded
>>>>>> to
>>>>>> the 1.2.0 RC1 version, please advise as to when this may be
>>> reasonably
>>>>>> possible.  (Please send any details or questions to:
>>>>>> esg-gateway-dev at earthsystemgrid.org)
>>>>>>
>>>>>> ANU
>>>>>> BADC
>>>>>> DKRZ
>>>>>> JPL
>>>>>> LBNL
>>>>>> NCAR
>>>>>> ORNL
>>>>>> PCMDI
>>>>>>
>>>>>> Please note the NCAR testing site (http://esg.prototype.ucar.edu)
>>> is
>>>>>> currently running the 1.2.0 RC1 version for federation testing
>>> purposes.
>>>>>> I would like to begin discussing the top priority federation tests
>>> and
>>>>>> would like to hear your input.  In particular, testing scenarios
>>>>>> related to security and access control of CMIP5 datasets are of
>>>>>> key
>>>>>> importance.  I will send a follow up email regarding this testing
>>> effort.
>>>>>> Thank you for your involvement in this effort!
>>>>>>
>>>>>> Kind Regards,
>>>>>>
>>>>>> -Eric
>>>>>>
>>>>>> On Sun, 14 Nov 2010 20:15:48 -0700
>>>>>> Nathan Wilhelmi<wilhelmi at ucar.edu>  wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> We have made a new release of the gateway available, 1.2.0-RC1.
>>> The
>>>>>>> release resolves a number of issues that were identified in the
>>> BETA3
>>>>>>> release.
>>>>>>>
>>>>>>> The new BETA release can be downloaded directly from here:
>>>>>>>
>>> https://vets.development.ucar.edu/nexus/content/repositories/releases/s
>>> gf/gateway/1.2.0-RC1/
>>>>>>>
>>>>>>> For this release the the new installation and upgrade from 1.1.0
>>> guides
>>>>>>> remain the same.
>>>>>>>
>>>>>>> If you already have 1.2.0-BETAx installed upgrade instructions
>>>>>>> can
>>> be
>>>>>>> found here:
>>>>>>>
>>> https://wiki.ucar.edu/display/esgcet/Gateway+Upgrade+Instructions+%28ve
>>> rsion+1.2.0-BETA-1.2.0-BETA2+to+version+1.2.0-BETA3%29
>>>>>>>
>>>>>>>
>>>>>>> Please provide feedback (good or bad!) to:
>>>>>>> esg-gateway-dev at earthsystemgrid.org
>>>>>>>
>>>>>>> -or-
>>>>>>>
>>>>>>> If you have Jira Issue Tracking access, please enter a trouble
>>> ticket
>>>>>>> against version "1.2.0-RC1" under the "Affects Version/s" drop-
>>> down.
>>>>>>> Thanks!!!
>>>>>>>
>>>>>>> -Nate
>>>>>>>
>>>>> _______________________________________________
>>>>> ESG-CET mailing list
>>>>> ESG-CET at earthsystemgrid.org
>>>>> http://mailman.ucar.edu/mailman/listinfo/esg-cet
>>>>>
>>>>
>>> _______________________________________________
>>> esg-gateway-dev mailing list
>>> esg-gateway-dev at mailman.earthsystemgrid.org
>>> http://mailman.earthsystemgrid.org/mailman/listinfo/esg-gateway-dev
>> -- 
>> Scanned by iCritical.
>>
>> <
>> ESGFSecurityAcceptanceTests
>> .pdf>_______________________________________________
>> GO-ESSP-TECH mailing list
>> GO-ESSP-TECH at ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> Rachana Ananthakrishnan
> Argonne National Lab | University of Chicago
>
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech


-- 
Estanislao Gonzalez

Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany

Phone:   +49 (40) 46 00 94-126
E-Mail:  estanislao.gonzalez at zmaw.de

_______________________________________________
GO-ESSP-TECH mailing list
GO-ESSP-TECH at ucar.edu
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
-- 
Scanned by iCritical.

More information about the GO-ESSP-TECH mailing list