[Go-essp-tech] CSIRO-Mk3-6-0 data
martin.juckes at stfc.ac.uk
martin.juckes at stfc.ac.uk
Fri Jul 15 03:33:01 MDT 2011
Hello Mark,
Concerning your comments on the QC process: have you pushed your QC results to the DKRZ database? There is a view of the operational database here: http://cera-www.dkrz.de/WDCC/CMIP5/QCStatus.jsp, but it does not show your data yet.
Regards,
Martin
From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of mark collier
Sent: 15 July 2011 07:12
To: go-essp-tech at ucar.edu
Cc: Jeffrey Stephen; Aspendale) Rotstayn Leon (CMAR
Subject: [Go-essp-tech] CSIRO-Mk3-6-0 data
Hi Martin,
just to clarify, the CMIP5 output from the CSIRO-Mk3-6-0 AOGCM is not restricted to our modelling group (CSIRO-QCCCE) but available to anyone who registers at the NCI ESG.
Quite a bit of data has been there since the middle of May, and we have published the amip and 3 historical attribution runs a few weeks ago.
In the next week or so we should be able to publish the historical and remaining attribution runs at the 3-4 RCP experiments out to 2100. The experiments out to 2300 will take a couple of more months to finish and publish.
All of our published data has been scrutinised by the QCWrapper, it has been quite beneficial to our efforts and we are using it as a tool to help diagnose experimental design setup with the ACCESS1.0 ESM (my fun is already starting with the post-processing requirements of this late model submission).
Our specialists at the NCI ESG have experienced difficulty (mostly beyond their control) getting the data accurately published and they expect that some of the issues will be resolved with a future release of the ESG software stack. We are keen to try and replicate our data at PCMDI for data security as well as ensuring that people have easy access to it. We would encourage people to assist our ESG specialists here in Australia as our remoteness often works against us (tyranny of distance even in this "connected" world), for example, no one from Australia was able to attend the recent GO-ESSP meeting in the U.S. to register these challenges.
Once again, people can obtain CSIRO-Mk3-6-0 data from the NCI ESG http://esg.nci.org.au/esgcet/home.htm after a quick registration process.
Best wishes,
Mark.
On 14/07/2011, at 7:27 PM, go-essp-tech-request at ucar.edu<mailto:go-essp-tech-request at ucar.edu> wrote:
Send GO-ESSP-TECH mailing list submissions to
go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
or, via email, send a message with subject or body 'help' to
go-essp-tech-request at ucar.edu<mailto:go-essp-tech-request at ucar.edu>
You can reach the person managing the list at
go-essp-tech-owner at ucar.edu<mailto:go-essp-tech-owner at ucar.edu>
When replying, please edit your Subject line so it is more specific
than "Re: Contents of GO-ESSP-TECH digest..."
Today's Topics:
1. Re: Searching on forcings in the ESG Gateway (Sylvia Murphy)
2. Re: Update Nodes & Gateways Web Page (martin.juckes at stfc.ac.uk<mailto:martin.juckes at stfc.ac.uk>)
3. Re: Update Nodes & Gateways Web Page (Estanislao Gonzalez)
----------------------------------------------------------------------
Message: 1
Date: Wed, 13 Jul 2011 13:35:07 -0600
From: Sylvia Murphy <Sylvia.Murphy at noaa.gov<mailto:Sylvia.Murphy at noaa.gov>>
Subject: Re: [Go-essp-tech] Searching on forcings in the ESG Gateway
To: "Schmidt, Gavin A. (GISS-6110)" <gavin.a.schmidt at nasa.gov<mailto:gavin.a.schmidt at nasa.gov>>
Cc: Metafor List <metafor at lists.enes.org<mailto:metafor at lists.enes.org>>, "go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>"
<go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>>
Message-ID: <7A1A8113-D47D-46F9-8022-D502AAB459CB at noaa.gov<mailto:7A1A8113-D47D-46F9-8022-D502AAB459CB at noaa.gov>>
Content-Type: text/plain; charset=us-ascii
Hi Gavin,
You pose some interesting questions here. Alas, some of the people who are best suited to answer them are unavailable this week. I just did not want you to think we were ignoring you. You'll hear from me again once I can get you a coordinated response.
Cheers,
Sylvia
On Jul 8, 2011, at 3:04 PM, Schmidt, Gavin A. (GISS-6110) wrote:
Hi Sylvia, a couple of thoughts.
1) are the input variable names part of the controlled vocabulary? If not, there may be some confusion - though this might be mitigated if the search was over the full text and was case-insenstive etc.
2) The reason to search on the forcings is to a) establish whether any specific forcing was used in a specific experiment across the groups. Which 'historical' simulations used ozone depletion? Which RCPs had continuations of solar cycle variability? and b) what experiments can be usefully compared to each other - ie different models' 'historicalMisc' runs with similar forcing sets, or same models' experiments that differ in some small detail in the forcings (i.e. identifying runs with and without indirect aerosol effects).
3) Forcings can differ within the same experiment even with the same model ('historicalMisc' is the obvious case, but may also occur with other simulations - i.e. we have one set of model runs that include an irrigation component but are otherwise identical). GIven the current set up the only indication of this is through the metafor name of the simulation and the 'forcings' attribute in the netcdf file - neither of which are visible via ESG. This is the biggest issue.
4) I'm not really sure how well your proposed solution would work in practice though. Perhaps you could work though the cases in points 2 + 3 and demonstrate that it would be clear?
5) Can we name the button 'Forcings' ? (and make the direct link to the 'forcings' attribute in the netcdf files.
Regards,
gavin
On Jul 8, 2011, at 2:49 PM, Sylvia Murphy wrote:
Hi Gavin,
It was a pleasure meeting you at the CCSM workshop a few weeks ago and getting your feedback on the model metadata display within the ESG Gateway. Cecelia and I have been thinking about your comment on being able to search on forcings and would like to bounce off the following with you to ensure we correctly understand your requirement. To do so, I have attached two screen shots. The first screen shot shows the current search interface and the "facets" on the left-hand side that allow the user to narrow a search. The second screen shot shows the current inputs (initial conditions, boundary conditions etc) for a sample CMIP5 simulation.
Would your requirement be met if we added a facet to the search (see figure 1) called Input or Input Condition (let us know if you have a preference) and then listed all of the variables names you see in screen shot 2 as the options to select from in the search itself?
Thanks again for your feedback. It is invaluable to us.
Cheers,
Sylvia
<search.tiff>
<inputs.tiff>
***********************************
Sylvia Murphy
NESII/CIRES/NOAA Earth System Research Laboratory
325 Broadway, Boulder CO 80305
Email: sylvia.murphy at noaa.gov<mailto:sylvia.murphy at noaa.gov>
Phone: 303-497-7753
Fax: 303-497-7649
==========
Gavin Schmidt
NASA Goddard Institute for Space Studies
2880 Broadway
New York, NY 10025
Tel: 212 678 5627
Email: Gavin.A.Schmidt at nasa.gov<mailto:Gavin.A.Schmidt at nasa.gov>
URL: http://www.giss.nasa.gov/staff/gschmidt.html
***********************************
Sylvia Murphy
NESII/CIRES/NOAA Earth System Research Laboratory
325 Broadway, Boulder CO 80305
Email: sylvia.murphy at noaa.gov<mailto:sylvia.murphy at noaa.gov>
Phone: 303-497-7753
Fax: 303-497-7649
------------------------------
Message: 2
Date: Thu, 14 Jul 2011 08:17:10 +0000
From: <martin.juckes at stfc.ac.uk<mailto:martin.juckes at stfc.ac.uk>>
Subject: Re: [Go-essp-tech] Update Nodes & Gateways Web Page
To: <gonzalez at dkrz.de<mailto:gonzalez at dkrz.de>>, <sebastien.denvil at ipsl.jussieu.fr<mailto:sebastien.denvil at ipsl.jussieu.fr>>
Cc: go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>
Message-ID:
<E21FBC3F00D7304687CB46529F9676D7141C98 at EXCHMBX01.fed.cclrc.ac.uk<mailto:E21FBC3F00D7304687CB46529F9676D7141C98 at EXCHMBX01.fed.cclrc.ac.uk>>
Content-Type: text/plain; charset="iso-8859-1"
Hi Estani,
The short answer to the first question is that is the way that they were set up and some groups don't have resources to carry out regular updates - work in progress.
Concerning not accepting PKI vs. not generating PKI scripts in the gateway: CCCMA, NCC and CNRM data nodes accept PKI certificates but are published through the PCMDI gateway, so that users are presented with tokenised wget scripts by the gateway. GISS, BCC and PCMDI nodes do not accept PKI scripts.
There is, incidentally, a data node in Australia as well, with CMIP5 data publuished - but access to data there is still restricted to the modelling group.
Cheers,
Martin
From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of Estanislao Gonzalez
Sent: 13 July 2011 15:43
To: S?bastien Denvil
Cc: go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>
Subject: Re: [Go-essp-tech] Update Nodes & Gateways Web Page
Hi Sebastien,
I still can't understand why there are data nodes not accepting PKI access. I've asked around but never got any answer.
What we can do is mark the data node we know has no PKI in place in the list as such. Non marked one are supposed to be working.
Furthermore I'm not sure if it's really the data node not accepting PKI access or the gateways not configured to provide them while generating the wget script.
By the way, I'm still not sure which version is around... The latest I've built used the checksums to assure the files were downloaded correctly, are all Gateways using that one?
Now regarding http://esgf.org/wiki/Cmip5Status/ArchiveView I've created that page just recently (glad you've found it out ;-). It's a prototype so I might have to tune it a little bit.
The main idea is to harvest known Gateways/top level collections for new datasets and building that summary you see (plus the DB we have).
Sadly the Gateway API prior to 1.3.1 together with the non conformance to DRS structures make it "unstable" so there are some data nodes that I just can't get information from.
I'll try to keep evolving it as time permits.
Thanks,
Estani
Am 13.07.2011 16:32, schrieb S?bastien Denvil:
Dear all,
from this page http://esgf.org/wiki/Cmip5Status the http://esgf.org/wiki/Cmip5Status/ArchiveView page is very useful. Thanks.
I'm sure it's in the pipeline and I mention it in case it's not:
- it would be great if we can have for each datanode a PKI access (yes/no) flag we could refer to.
All the best.
S?bastien
On 12/07/2011 21:35, Williams, Dean N. wrote:
Thanks Bob, Eric, and everyone else for updating the information on the website. We will have network connectivity listed on this site soon.
Best regards,
Dean
Bob Drach On 7/12/11 11:55 AM, "Bob Drach" <drach1 at llnl.gov<mailto:drach1 at llnl.gov>> wrote:
OK, done for the PCMDI nodes. --Bob
On 7/12/11 6:45 AM, "Williams, Dean N." <williams13 at llnl.gov<mailto:williams13 at llnl.gov>> wrote:
Dear Administrators,
Please update the Nodes & Gateways web page found at: http://esgf.org/wiki/Cmip5Status . For example, if you have updated your test gateway to 1.3.0, please indicate it.
Thanks,
Dean
_______________________________________________
GO-ESSP-TECH mailing list
GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
--
S?bastien Denvil
IPSL, P?le de mod?lisation du climat
UPMC, Case 101, 4 place Jussieu,
75252 Paris Cedex 5
Tour 45-55 2?me ?tage Bureau 209
Tel: 33 1 44 27 21 10
Fax: 33 1 44 27 39 02
_______________________________________________
GO-ESSP-TECH mailing list
GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
--
Estanislao Gonzalez
Max-Planck-Institut f?r Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: gonzalez at dkrz.de<mailto:gonzalez at dkrz.de>
--
Scanned by iCritical.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20110714/85c42c15/attachment-0001.html
------------------------------
Message: 3
Date: Thu, 14 Jul 2011 11:27:28 +0200
From: Estanislao Gonzalez <gonzalez at dkrz.de>
Subject: Re: [Go-essp-tech] Update Nodes & Gateways Web Page
To: Eric Nienhouse <ejn at ucar.edu>
Cc: go-essp-tech at ucar.edu
Message-ID: <4E1EB680.5090901 at dkrz.de>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Hi Eric,
I've sent the script again to gateway-dev. We now have a contrib repo in
esgf for "miscellaneous" (And the applause goes to Gavin :-) Thank you
very much for this) where I added that file too (under estani/gateway).
The repo is in esgf.org/esgf-contrib.git
I think we should move towards a script that provides both
functionalities. Or, better yet, let the gateway store the configured
access for every data node that published data to it and use this to set
up the script (i.e. add the token to the urls from datanodes that have
no PKI in place). The gateway can collect this data "on demand", i.e.
when the data node redirect to one of the attribute service. This could
also be in the registry, but it will take longer to get there and it
currently makes no sense, since those data nodes won't be upgraded to a
node-manager version that provides such functionality. Of course the
"fall back" solution should be token based, as it is now.
It does sound "simple" enough to me, but I might have missed something...
Thanks,
Estani
Am 13.07.2011 19:10, schrieb Eric Nienhouse:
Hi Estani, All,
Estanislao Gonzalez wrote:
Hi Sebastien,
I still can't understand why there are data nodes not accepting PKI
access. I've asked around but never got any answer.
Certain data nodes do not have the "tokenless" (ie. PKI certificate)
access configured yet. It may simply be a matter of time for these
data nodes to update. Long ago we all planned to remove token support
once we had consensus amongst the node data managers to do so.
Now is a good time to revive this discussion IMO.
What we can do is mark the data node we know has no PKI in place in
the list as such. Non marked one are supposed to be working.
Sounds like a good idea to me.
Furthermore I'm not sure if it's really the data node not accepting
PKI access or the gateways not configured to provide them while
generating the wget script.
Gateways may be configured to generate WGet scripts with tokens per
data node service via the Admin UI. This capability is in place to
allow migration of nodes off the token based auth-z.
As you know these are short lived tokens and only work for the
originating gateway's nodes. Though they are simple to use, they are
limited in capability and will eventually be phased out.
Some gateways are still configured to generate tokens for downloads
(wget and direct gateway UI download link access). NCAR is currently
configured in this way even though the NCAR data node also accepts PKI
certs. We chose to do this to support our current users who are
familiar with token based access and are primarily accessing CMIP3
datasets. We're expecting CMIP5 data publication soon, so we're
reviewing when it would be best to convert fully to PKI based access.
We'll update the CMIP5 wiki page when we do :-)
By the way, I'm still not sure which version is around... The latest
I've built used the checksums to assure the files were downloaded
correctly, are all Gateways using that one?
I don't think all gateways are using this latest script. Could you
provide it? (Especially in the FreeMarker template format it you have
it?) Thank you!
Now regarding http://esgf.org/wiki/Cmip5Status/ArchiveView I've
created that page just recently (glad you've found it out ;-). It's a
prototype so I might have to tune it a little bit.
The main idea is to harvest known Gateways/top level collections for
new datasets and building that summary you see (plus the DB we have).
Sadly the Gateway API prior to 1.3.1 together with the non
conformance to DRS structures make it "unstable" so there are some
data nodes that I just can't get information from.
I'll try to keep evolving it as time permits.
Thanks,
Estani
Am 13.07.2011 16:32, schrieb S?bastien Denvil:
Dear all,
from this page http://esgf.org/wiki/Cmip5Status the
http://esgf.org/wiki/Cmip5Status/ArchiveView page is very useful.
Thanks.
I'm sure it's in the pipeline and I mention it in case it's not:
- it would be great if we can have for each datanode a PKI access
(yes/no) flag we could refer to.
All the best.
S?bastien
On 12/07/2011 21:35, Williams, Dean N. wrote:
Thanks Bob, Eric, and everyone else for updating the information on
the website. We will have network connectivity listed on this site
soon.
Best regards,
Dean
Bob Drach On 7/12/11 11:55 AM, "Bob Drach" <drach1 at llnl.gov> wrote:
OK, done for the PCMDI nodes. --Bob
On 7/12/11 6:45 AM, "Williams, Dean N." <williams13 at llnl.gov>
wrote:
Dear Administrators,
Please update the Nodes & Gateways web page found at:
_http://esgf.org/wiki/Cmip5Status_ . For example, if you
have updated your test gateway to 1.3.0, please indicate it.
Thanks,
Dean
_______________________________________________
GO-ESSP-TECH mailing list
GO-ESSP-TECH at ucar.edu
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
--
S?bastien Denvil
IPSL, P?le de mod?lisation du climat
UPMC, Case 101, 4 place Jussieu,
75252 Paris Cedex 5
Tour 45-55 2?me ?tage Bureau 209
Tel: 33 1 44 27 21 10
Fax: 33 1 44 27 39 02
_______________________________________________
GO-ESSP-TECH mailing list
GO-ESSP-TECH at ucar.edu
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
--
Estanislao Gonzalez
Max-Planck-Institut f?r Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: gonzalez at dkrz.de
------------------------------------------------------------------------
_______________________________________________
GO-ESSP-TECH mailing list
GO-ESSP-TECH at ucar.edu
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
--
Estanislao Gonzalez
Max-Planck-Institut f?r Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: gonzalez at dkrz.de
------------------------------
_______________________________________________
GO-ESSP-TECH mailing list
GO-ESSP-TECH at ucar.edu
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
End of GO-ESSP-TECH Digest, Vol 29, Issue 30
********************************************
--
Scanned by iCritical.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20110715/df0a4cc8/attachment-0001.html
More information about the GO-ESSP-TECH
mailing list