[Go-essp-tech] CSIRO-Mk3-6-0 data
Martina Stockhause
stockhause at dkrz.de
Fri Jul 15 06:31:18 MDT 2011
Hi, Martin,
I think I should answer that.
The service http://cera-www.dkrz.de/WDCC/CMIP5/QCStatus.jsp only covers
the production QC data for the three official QC managers at PCMDI, BADC
and DKRZ. Mark is working with a testing postgres QCDB. In fact I asked
him to stay with it to not mix the qc tool results between us three and
the data nodes, who use our QC tool. So, his results does not show up in
the service.
Best wishes,
Martina
On 15.07.2011 11:33, martin.juckes at stfc.ac.uk wrote:
>
> Hello Mark,
>
> Concerning your comments on the QC process: have you pushed your QC
> results to the DKRZ database? There is a view of the operational
> database here: http://cera-www.dkrz.de/WDCC/CMIP5/QCStatus.jsp, but it
> does not show your data yet.
>
> Regards,
>
> Martin
>
> *From:* go-essp-tech-bounces at ucar.edu
> [mailto:go-essp-tech-bounces at ucar.edu] *On Behalf Of *mark collier
> *Sent:* 15 July 2011 07:12
> *To:* go-essp-tech at ucar.edu
> *Cc:* Jeffrey Stephen; Aspendale) Rotstayn Leon (CMAR
> *Subject:* [Go-essp-tech] CSIRO-Mk3-6-0 data
>
> Hi Martin,
>
> just to clarify, the CMIP5 output from the CSIRO-Mk3-6-0 AOGCM is not
> restricted to our modelling group (CSIRO-QCCCE) but available to
> anyone who registers at the NCI ESG.
>
> Quite a bit of data has been there since the middle of May, and we
> have published the amip and 3 historical attribution runs a few weeks ago.
>
> In the next week or so we should be able to publish the historical and
> remaining attribution runs at the 3-4 RCP experiments out to 2100. The
> experiments out to 2300 will take a couple of more months to finish
> and publish.
>
> All of our published data has been scrutinised by the QCWrapper, it
> has been quite beneficial to our efforts and we are using it as a tool
> to help diagnose experimental design setup with the ACCESS1.0 ESM (my
> fun is already starting with the post-processing requirements of this
> late model submission).
>
> Our specialists at the NCI ESG have experienced difficulty (mostly
> beyond their control) getting the data accurately published and they
> expect that some of the issues will be resolved with a future release
> of the ESG software stack. We are keen to try and replicate our data
> at PCMDI for data security as well as ensuring that people have easy
> access to it. We would encourage people to assist our ESG specialists
> here in Australia as our remoteness often works against us (tyranny of
> distance even in this "connected" world), for example, no one from
> Australia was able to attend the recent GO-ESSP meeting in the U.S. to
> register these challenges.
>
> Once again, people can obtain CSIRO-Mk3-6-0 data from the NCI ESG
> http://esg.nci.org.au/esgcet/home.htm after a quick registration process.
>
> Best wishes,
>
> Mark.
>
> On 14/07/2011, at 7:27 PM, go-essp-tech-request at ucar.edu
> <mailto:go-essp-tech-request at ucar.edu> wrote:
>
>
>
> Send GO-ESSP-TECH mailing list submissions to
> go-essp-tech at ucar.edu <mailto:go-essp-tech at ucar.edu>
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> or, via email, send a message with subject or body 'help' to
> go-essp-tech-request at ucar.edu <mailto:go-essp-tech-request at ucar.edu>
>
> You can reach the person managing the list at
> go-essp-tech-owner at ucar.edu <mailto:go-essp-tech-owner at ucar.edu>
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of GO-ESSP-TECH digest..."
>
>
> Today's Topics:
>
> 1. Re: Searching on forcings in the ESG Gateway (Sylvia Murphy)
> 2. Re: Update Nodes & Gateways Web Page (martin.juckes at stfc.ac.uk
> <mailto:martin.juckes at stfc.ac.uk>)
> 3. Re: Update Nodes & Gateways Web Page (Estanislao Gonzalez)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 13 Jul 2011 13:35:07 -0600
> From: Sylvia Murphy <Sylvia.Murphy at noaa.gov
> <mailto:Sylvia.Murphy at noaa.gov>>
> Subject: Re: [Go-essp-tech] Searching on forcings in the ESG Gateway
> To: "Schmidt, Gavin A. (GISS-6110)" <gavin.a.schmidt at nasa.gov
> <mailto:gavin.a.schmidt at nasa.gov>>
> Cc: Metafor List <metafor at lists.enes.org
> <mailto:metafor at lists.enes.org>>, "go-essp-tech at ucar.edu
> <mailto:go-essp-tech at ucar.edu>"
> <go-essp-tech at ucar.edu <mailto:go-essp-tech at ucar.edu>>
> Message-ID: <7A1A8113-D47D-46F9-8022-D502AAB459CB at noaa.gov
> <mailto:7A1A8113-D47D-46F9-8022-D502AAB459CB at noaa.gov>>
> Content-Type: text/plain; charset=us-ascii
>
> Hi Gavin,
> You pose some interesting questions here. Alas, some of the people
> who are best suited to answer them are unavailable this week. I just
> did not want you to think we were ignoring you. You'll hear from me
> again once I can get you a coordinated response.
> Cheers,
> Sylvia
> On Jul 8, 2011, at 3:04 PM, Schmidt, Gavin A. (GISS-6110) wrote:
>
>
> Hi Sylvia, a couple of thoughts.
>
> 1) are the input variable names part of the controlled vocabulary?
> If not, there may be some confusion - though this might be
> mitigated if the search was over the full text and was
> case-insenstive etc.
>
> 2) The reason to search on the forcings is to a) establish whether
> any specific forcing was used in a specific experiment across the
> groups. Which 'historical' simulations used ozone depletion? Which
> RCPs had continuations of solar cycle variability? and b) what
> experiments can be usefully compared to each other - ie different
> models' 'historicalMisc' runs with similar forcing sets, or same
> models' experiments that differ in some small detail in the
> forcings (i.e. identifying runs with and without indirect aerosol
> effects).
>
> 3) Forcings can differ within the same experiment even with the
> same model ('historicalMisc' is the obvious case, but may also
> occur with other simulations - i.e. we have one set of model runs
> that include an irrigation component but are otherwise identical).
> GIven the current set up the only indication of this is through
> the metafor name of the simulation and the 'forcings' attribute in
> the netcdf file - neither of which are visible via ESG. This is
> the biggest issue.
>
> 4) I'm not really sure how well your proposed solution would work
> in practice though. Perhaps you could work though the cases in
> points 2 + 3 and demonstrate that it would be clear?
>
> 5) Can we name the button 'Forcings' ? (and make the direct link
> to the 'forcings' attribute in the netcdf files.
>
> Regards,
>
> gavin
>
> On Jul 8, 2011, at 2:49 PM, Sylvia Murphy wrote:
>
> Hi Gavin,
>
> It was a pleasure meeting you at the CCSM workshop a few weeks
> ago and getting your feedback on the model metadata display
> within the ESG Gateway. Cecelia and I have been thinking
> about your comment on being able to search on forcings and
> would like to bounce off the following with you to ensure we
> correctly understand your requirement. To do so, I have
> attached two screen shots. The first screen shot shows the
> current search interface and the "facets" on the left-hand
> side that allow the user to narrow a search. The second
> screen shot shows the current inputs (initial conditions,
> boundary conditions etc) for a sample CMIP5 simulation.
>
> Would your requirement be met if we added a facet to the
> search (see figure 1) called Input or Input Condition (let us
> know if you have a preference) and then listed all of the
> variables names you see in screen shot 2 as the options to
> select from in the search itself?
>
> Thanks again for your feedback. It is invaluable to us.
>
> Cheers,
>
> Sylvia
>
> <search.tiff>
>
> <inputs.tiff>
>
> ***********************************
>
> Sylvia Murphy
>
> NESII/CIRES/NOAA Earth System Research Laboratory
>
> 325 Broadway, Boulder CO 80305
>
> Email: sylvia.murphy at noaa.gov <mailto:sylvia.murphy at noaa.gov>
>
> Phone: 303-497-7753
>
> Fax: 303-497-7649
>
> ==========
>
> Gavin Schmidt
>
> NASA Goddard Institute for Space Studies
>
> 2880 Broadway
>
> New York, NY 10025
>
> Tel: 212 678 5627
>
> Email: Gavin.A.Schmidt at nasa.gov <mailto:Gavin.A.Schmidt at nasa.gov>
>
> URL: http://www.giss.nasa.gov/staff/gschmidt.html
>
>
> ***********************************
> Sylvia Murphy
> NESII/CIRES/NOAA Earth System Research Laboratory
> 325 Broadway, Boulder CO 80305
> Email: sylvia.murphy at noaa.gov <mailto:sylvia.murphy at noaa.gov>
> Phone: 303-497-7753
> Fax: 303-497-7649
>
>
>
>
>
>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Thu, 14 Jul 2011 08:17:10 +0000
> From: <martin.juckes at stfc.ac.uk <mailto:martin.juckes at stfc.ac.uk>>
> Subject: Re: [Go-essp-tech] Update Nodes & Gateways Web Page
> To: <gonzalez at dkrz.de <mailto:gonzalez at dkrz.de>>,
> <sebastien.denvil at ipsl.jussieu.fr
> <mailto:sebastien.denvil at ipsl.jussieu.fr>>
> Cc: go-essp-tech at ucar.edu <mailto:go-essp-tech at ucar.edu>
> Message-ID:
> <E21FBC3F00D7304687CB46529F9676D7141C98 at EXCHMBX01.fed.cclrc.ac.uk
> <mailto:E21FBC3F00D7304687CB46529F9676D7141C98 at EXCHMBX01.fed.cclrc.ac.uk>>
> Content-Type: text/plain; charset="iso-8859-1"
>
> Hi Estani,
>
> The short answer to the first question is that is the way that they
> were set up and some groups don't have resources to carry out regular
> updates - work in progress.
>
> Concerning not accepting PKI vs. not generating PKI scripts in the
> gateway: CCCMA, NCC and CNRM data nodes accept PKI certificates but
> are published through the PCMDI gateway, so that users are presented
> with tokenised wget scripts by the gateway. GISS, BCC and PCMDI nodes
> do not accept PKI scripts.
>
> There is, incidentally, a data node in Australia as well, with CMIP5
> data publuished - but access to data there is still restricted to the
> modelling group.
>
> Cheers,
> Martin
>
> From: go-essp-tech-bounces at ucar.edu
> [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of Estanislao Gonzalez
> Sent: 13 July 2011 15:43
> To: S?bastien Denvil
> Cc: go-essp-tech at ucar.edu <mailto:go-essp-tech at ucar.edu>
> Subject: Re: [Go-essp-tech] Update Nodes & Gateways Web Page
>
> Hi Sebastien,
>
> I still can't understand why there are data nodes not accepting PKI
> access. I've asked around but never got any answer.
>
> What we can do is mark the data node we know has no PKI in place in
> the list as such. Non marked one are supposed to be working.
>
> Furthermore I'm not sure if it's really the data node not accepting
> PKI access or the gateways not configured to provide them while
> generating the wget script.
>
> By the way, I'm still not sure which version is around... The latest
> I've built used the checksums to assure the files were downloaded
> correctly, are all Gateways using that one?
>
> Now regarding http://esgf.org/wiki/Cmip5Status/ArchiveView I've
> created that page just recently (glad you've found it out ;-). It's a
> prototype so I might have to tune it a little bit.
> The main idea is to harvest known Gateways/top level collections for
> new datasets and building that summary you see (plus the DB we have).
> Sadly the Gateway API prior to 1.3.1 together with the non conformance
> to DRS structures make it "unstable" so there are some data nodes that
> I just can't get information from.
>
> I'll try to keep evolving it as time permits.
>
> Thanks,
> Estani
>
>
> Am 13.07.2011 16:32, schrieb S?bastien Denvil:
> Dear all,
>
> from this page http://esgf.org/wiki/Cmip5Status the
> http://esgf.org/wiki/Cmip5Status/ArchiveView page is very useful. Thanks.
>
> I'm sure it's in the pipeline and I mention it in case it's not:
> - it would be great if we can have for each datanode a PKI access
> (yes/no) flag we could refer to.
>
> All the best.
> S?bastien
>
> On 12/07/2011 21:35, Williams, Dean N. wrote:
> Thanks Bob, Eric, and everyone else for updating the information on
> the website. We will have network connectivity listed on this site soon.
>
> Best regards,
> Dean
>
>
> Bob Drach On 7/12/11 11:55 AM, "Bob Drach" <drach1 at llnl.gov
> <mailto:drach1 at llnl.gov>> wrote:
> OK, done for the PCMDI nodes. --Bob
>
>
> On 7/12/11 6:45 AM, "Williams, Dean N." <williams13 at llnl.gov
> <mailto:williams13 at llnl.gov>> wrote:
> Dear Administrators,
>
> Please update the Nodes & Gateways web page found at:
> http://esgf.org/wiki/Cmip5Status . For example, if you have updated
> your test gateway to 1.3.0, please indicate it.
>
>
> Thanks,
> Dean
>
>
>
>
> _______________________________________________
>
> GO-ESSP-TECH mailing list
>
> GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
>
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
>
>
>
> --
>
> S?bastien Denvil
>
> IPSL, P?le de mod?lisation du climat
>
> UPMC, Case 101, 4 place Jussieu,
>
> 75252 Paris Cedex 5
>
>
>
> Tour 45-55 2?me ?tage Bureau 209
>
> Tel: 33 1 44 27 21 10
>
> Fax: 33 1 44 27 39 02
>
>
>
>
>
> _______________________________________________
>
> GO-ESSP-TECH mailing list
>
> GO-ESSP-TECH at ucar.edu<mailto:GO-ESSP-TECH at ucar.edu>
>
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
>
>
>
> --
>
> Estanislao Gonzalez
>
>
>
> Max-Planck-Institut f?r Meteorologie (MPI-M)
>
> Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
>
> Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
>
>
>
> Phone: +49 (40) 46 00 94-126
>
> E-Mail: gonzalez at dkrz.de<mailto:gonzalez at dkrz.de>
>
> --
> Scanned by iCritical.
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20110714/85c42c15/attachment-0001.html
>
> ------------------------------
>
> Message: 3
> Date: Thu, 14 Jul 2011 11:27:28 +0200
> From: Estanislao Gonzalez <gonzalez at dkrz.de>
> Subject: Re: [Go-essp-tech] Update Nodes & Gateways Web Page
> To: Eric Nienhouse <ejn at ucar.edu>
> Cc: go-essp-tech at ucar.edu
> Message-ID: <4E1EB680.5090901 at dkrz.de>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Hi Eric,
>
> I've sent the script again to gateway-dev. We now have a contrib repo in
> esgf for "miscellaneous" (And the applause goes to Gavin :-) Thank you
> very much for this) where I added that file too (under estani/gateway).
> The repo is in esgf.org/esgf-contrib.git
>
> I think we should move towards a script that provides both
> functionalities. Or, better yet, let the gateway store the configured
> access for every data node that published data to it and use this to set
> up the script (i.e. add the token to the urls from datanodes that have
> no PKI in place). The gateway can collect this data "on demand", i.e.
> when the data node redirect to one of the attribute service. This could
> also be in the registry, but it will take longer to get there and it
> currently makes no sense, since those data nodes won't be upgraded to a
> node-manager version that provides such functionality. Of course the
> "fall back" solution should be token based, as it is now.
>
> It does sound "simple" enough to me, but I might have missed something...
>
> Thanks,
> Estani
>
>
> Am 13.07.2011 19:10, schrieb Eric Nienhouse:
>
> Hi Estani, All,
>
> Estanislao Gonzalez wrote:
>
> Hi Sebastien,
>
> I still can't understand why there are data nodes not
> accepting PKI
>
> access. I've asked around but never got any answer.
>
> Certain data nodes do not have the "tokenless" (ie. PKI certificate)
>
> access configured yet. It may simply be a matter of time for these
>
> data nodes to update. Long ago we all planned to remove token support
>
> once we had consensus amongst the node data managers to do so.
>
> Now is a good time to revive this discussion IMO.
>
> What we can do is mark the data node we know has no PKI in
> place in
>
> the list as such. Non marked one are supposed to be working.
>
> Sounds like a good idea to me.
>
> Furthermore I'm not sure if it's really the data node not
> accepting
>
> PKI access or the gateways not configured to provide them while
>
> generating the wget script.
>
> Gateways may be configured to generate WGet scripts with tokens per
>
> data node service via the Admin UI. This capability is in place to
>
> allow migration of nodes off the token based auth-z.
>
> As you know these are short lived tokens and only work for the
>
> originating gateway's nodes. Though they are simple to use, they are
>
> limited in capability and will eventually be phased out.
>
> Some gateways are still configured to generate tokens for downloads
>
> (wget and direct gateway UI download link access). NCAR is currently
>
> configured in this way even though the NCAR data node also accepts PKI
>
> certs. We chose to do this to support our current users who are
>
> familiar with token based access and are primarily accessing CMIP3
>
> datasets. We're expecting CMIP5 data publication soon, so we're
>
> reviewing when it would be best to convert fully to PKI based access.
>
> We'll update the CMIP5 wiki page when we do :-)
>
> By the way, I'm still not sure which version is around... The
> latest
>
> I've built used the checksums to assure the files were downloaded
>
> correctly, are all Gateways using that one?
>
> I don't think all gateways are using this latest script. Could you
>
> provide it? (Especially in the FreeMarker template format it you have
>
> it?) Thank you!
>
> Now regarding http://esgf.org/wiki/Cmip5Status/ArchiveView I've
>
> created that page just recently (glad you've found it out ;-).
> It's a
>
> prototype so I might have to tune it a little bit.
>
> The main idea is to harvest known Gateways/top level
> collections for
>
> new datasets and building that summary you see (plus the DB we
> have).
>
> Sadly the Gateway API prior to 1.3.1 together with the non
>
> conformance to DRS structures make it "unstable" so there are some
>
> data nodes that I just can't get information from.
>
> I'll try to keep evolving it as time permits.
>
> Thanks,
>
> Estani
>
> Am 13.07.2011 16:32, schrieb S?bastien Denvil:
>
> Dear all,
>
> from this page http://esgf.org/wiki/Cmip5Status the
>
> http://esgf.org/wiki/Cmip5Status/ArchiveView page is very
> useful.
>
> Thanks.
>
> I'm sure it's in the pipeline and I mention it in case
> it's not:
>
> - it would be great if we can have for each datanode a PKI
> access
>
> (yes/no) flag we could refer to.
>
> All the best.
>
> S?bastien
>
> On 12/07/2011 21:35, Williams, Dean N. wrote:
>
> Thanks Bob, Eric, and everyone else for updating the
> information on
>
> the website. We will have network connectivity listed
> on this site
>
> soon.
>
> Best regards,
>
> Dean
>
> Bob Drach On 7/12/11 11:55 AM, "Bob Drach"
> <drach1 at llnl.gov> wrote:
>
> OK, done for the PCMDI nodes. --Bob
>
> On 7/12/11 6:45 AM, "Williams, Dean N."
> <williams13 at llnl.gov>
>
> wrote:
>
> Dear Administrators,
>
> Please update the Nodes & Gateways web page
> found at:
>
> _http://esgf.org/wiki/Cmip5Status_ . For
> example, if you
>
> have updated your test gateway to 1.3.0, please
> indicate it.
>
> Thanks,
>
> Dean
>
> _______________________________________________
>
> GO-ESSP-TECH mailing list
>
> GO-ESSP-TECH at ucar.edu
>
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
> --
>
> S?bastien Denvil
>
> IPSL, P?le de mod?lisation du climat
>
> UPMC, Case 101, 4 place Jussieu,
>
> 75252 Paris Cedex 5
>
> Tour 45-55 2?me ?tage Bureau 209
>
> Tel: 33 1 44 27 21 10
>
> Fax: 33 1 44 27 39 02
>
> _______________________________________________
>
> GO-ESSP-TECH mailing list
>
> GO-ESSP-TECH at ucar.edu
>
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
> --
>
> Estanislao Gonzalez
>
> Max-Planck-Institut f?r Meteorologie (MPI-M)
>
> Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing
> Centre
>
> Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
>
> Phone: +49 (40) 46 00 94-126
>
> E-Mail: gonzalez at dkrz.de
>
> ------------------------------------------------------------------------
>
> _______________________________________________
>
> GO-ESSP-TECH mailing list
>
> GO-ESSP-TECH at ucar.edu
>
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
>
>
> --
> Estanislao Gonzalez
>
> Max-Planck-Institut f?r Meteorologie (MPI-M)
> Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
> Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
>
> Phone: +49 (40) 46 00 94-126
> E-Mail: gonzalez at dkrz.de
>
>
>
> ------------------------------
>
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
>
> End of GO-ESSP-TECH Digest, Vol 29, Issue 30
> ********************************************
>
>
> --
> Scanned by iCritical.
>
>
--
------------------ DKRZ / Data Management ------------------
Martina Stockhause
Deutsches Klimarechenzentrum phone: +49-40-460094-122
Bundesstr. 45a FAX: +49-40-460094-106
D-20146 Hamburg, Germany e-mail: stockhause at dkrz.de
------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20110715/df2f0e2a/attachment-0001.html
More information about the GO-ESSP-TECH
mailing list