[Go-essp-tech] GO-ESSP call Jan 10, 2012

[Eric Nienhouse]

Hi Karl,

Thank you for this summary.  I agree a key requirement of the Gateway is 
to make available all output released by climate modeling groups.  
You've noted certain discrepancies between the 1.3 and 2.0 Gateway 
versions which are a concern.

I believe the Gateway 2.0 is more accurately representing the ESG 
federated system state than the 1.3.4 version.  The Gateway 2.0 provides 
additional validation over 1.3.4 primarily to reduce the number of 
inconsistencies experienced by users and to accurately support access 
control.  This leads to some differences when comparing Gateway 1.3.4 
and 2.0 (we've identified about 75 such cases.)

The discrepancies you note are a reflection of inconsistencies between 
the Thredds dataset catalogs and the publication state stored at the 
1.3.4 Gateway.  Below is a brief summary.  Much of this can be (and 
should be) corrected by re-publishing the current dataset catalog to the 
corresponding gateway to fix up these errors.

* In about 40 cases, the original catalog has "moved" and the Gateway is 
referencing the old catalog path which no longer exists.  In general 
files are still down-loadable from 1.3.X.

* In 13 cases, the the original catalog no longer exists.  The files are 
still download-able from 1.3.X without access control due to a TDS 
loophole.

* In at least 2 cases, a new dataset version catalog is available, 
however it has not been published to the Gateway (and the old one removed.)

Again, much of this can be corrected by re-publishing the current 
dataset catalogs to the gateway.  This should resolve the particular 
issue Karl noted below in which "MPI-ESM-LR, rcp85, r3i1p1" datasets are 
missing from the 2.0 Gateway.  In this case, these (12) dataset catalogs 
have been moved to a new path (from /6/ to /1/) and should be republished.

Kind regards,

-Eric

Karl Taylor wrote:
> Dear all,
>
> Regarding:
>> ... Karl contacted us and asked us to remove all of the federated CMIP5 datasets from our gateway. 
>>     
> Could somebody explain why Karl asked NCAR to do this?
>
>   
> Here is part of the explanation:
>
> Before changes are made to the way the CMIP5 model output is served, 
> several requirements must be met.  PCMDI has the responsibility for 
> this, and we have not given permission for any new gateways to become 
> operational in the service of CMIP5 data.  In particular, the "Gateway 
> 2.0" has not yet completed a sufficiently testing phase.  We therefore 
> asked NCAR to remove the federated CMIP5 datasets from the operational 
> version of its gateway.
>
> One requirement of the gateway is that it make available *all*  output 
> released by the climate modeling groups.   The modeling groups have 
> worked hard to finish their simulations and produce output in a 
> user-friendly form.  Our responsibility is to make sure users can get 
> to it.   The release candidate for Gateway 2.0 fails to list some 
> datasets that are in fact being served through Gateway 1.3.4 (the 
> operational gateway at the main CMIP5 data centers at PCMDI, BADC, and 
> WDCC). [see below for an example.]  Users coming to Gateway 2.0 would 
> therefore incorrectly conclude that some models hadn't provided the 
> output they were seeking, when in fact the Gateway 2.0 was simply not 
> showing it.  This is unacceptable and is one reason that Gateway 2.0 
> cannot operationally serve CMIP5 model output. 
>
> This serious problem with Gateway 2.0 may not be the only issue that 
> could delay or rule out its use in serving CMIP5.  Rest assured that 
> more than one option is being considered for improving access to CMIP5 
> model output.  We hope to have a better system in place soon, but only 
> after a careful and thorough testing stage has been completed.  During 
> the testing phase, we will at some point (within weeks, I think) 
> invite users to try out the new system (as beta testers), but the 
> operational system will also remain available and resource limitations 
> will likely mean our CMIP5 help desk will during the testing phase 
> only be able to respond to questions concerning the operational system. 
>
> We hope, therefore, that before too long the users will see 
> substantial improvements in the system giving them access to CMIP5 
> model output. I welcome your continued input and help.
>
> Best regards,
> Karl
>
> Example of problem with Gateway 2.0:  Compare search results on 
> MPI-ESM-LR, rcp85, r3i1p1
>
> Gateway 1.3.4 (at BADC, for example): returns 15 datasets
> the prototype Gateway 2.0 (at NCAR): returns only 3 datasets.
>
> thus, Gateway 2.0 fails to show the user datasets which, in fact, are 
> available from the MPI model.
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>