[Go-essp-tech] DRS, version number & Co

Mon Jul 11 09:13:21 MDT 2011

Hi Estani,

On the last point, we should probably start using the verc.enes.org portal more at some stage, but that is still evolving. It is probably best to keep content in the portal brief, and point to a page somewhere to hold such evolving information. It could be a page on the DKRZ wiki until verc.enes.org (the url as well as the name will probably change) is ready for use.

Cheers,
Martin

> >-----Original Message-----
> >From: Estanislao Gonzalez [mailto:gonzalez at dkrz.de]
> >Sent: 09 July 2011 20:16
> >To: Juckes, Martin (STFC,RAL,RALSP)
> >Cc: go-essp-tech at ucar.edu
> >Subject: Re: [Go-essp-tech] DRS, version number & Co
> >
> >Hi Martin,
> >
> >thanks a lot for the corrections. I just realized that I wrote
> >something
> >I haven't really meant:
> >"This process implies that data can be removed without previous
> >notice.
> >Changes will be notified here"
> >
> >(Or maybe I did at that time... :-)
> >
> >In any case QCL1 doesn't force the publisher to use a new version if
> >anything gets changed nor even to notify of such changes. So I think
> >we
> >might want to change that. Regarding your correction, that might be:
> >
> >[..] Withdrawals and replacements may happen without notification.
> >[..]
> >
> >But that sounds quite rude (indeed honest... but still... :-)
> >
> >I don't see either the means or the will to publish every single
> >change
> >at this stage. We might be able to notify changes for MPI-M, but
> >that's
> >fairly easy as we are "close", it will still require some coordination
> >though. I don't think that will happen between other institutions,
> >e.g.
> >BCC and PCMDI. This will require a large effort and further
> >coordination.
> >
> >Any comments on this?
> >
> >Thanks,
> >Estani
> >
> >Am 08.07.2011 18:41, schrieb martin.juckes at stfc.ac.uk:
> >> Hello Estani,
> >>
> >> I would delete "minimal" -- the syntactical check is not much, but
> >it is better than many archives. I would suggest the following revised
> >text:
> >>
> >> CMIP5 data has been quality checked for level one (QCL1) which
> >guarantees minimal conformance. The data is currently being quality
> >checked for level two (QCL2) to assure consistency. As a consequence
> >of this process, data may be withdrawn from the archive and/or
> >replaced with corrected data. Notification of withdrawals and
> >replacements will be posted here. Please refer to this documentation
> >for more information regarding CMIP5 quality assurance procedure.
> >>
> >> I've softened the language a little, but still convey the same
> >message -- I think, trying to make it more explanatory than
> >declaratory.
> >>
> >> Cheers,
> >> Martin
> >>
> >>>> -----Original Message-----
> >>>> From: Estanislao Gonzalez [mailto:gonzalez at dkrz.de]
> >>>> Sent: 08 July 2011 11:40
> >>>> To: Juckes, Martin (STFC,RAL,RALSP)
> >>>> Cc: go-essp-tech at ucar.edu
> >>>> Subject: Re: [Go-essp-tech] DRS, version number&  Co
> >>>>
> >>>> Hi Martin,
> >>>>
> >>>>   a short comment inline:
> >>>> Am 08.07.2011 11:09, schrieb martin.juckes at stfc.ac.uk:
> >>>>> Hi Estani,
> >>>>>
> >>>>> I agree with you that this is an important issue and that we want
> >to
> >>>> have a clean implementation.
> >>>>> Unfortunately, given where we are now, I don't think there is
> >going
> >>>> to be any support for withdrawing data nodes which don't meet this
> >>>> implementation standard -- so enforcement by the gateway won't
> >work.
> >>>> So I think the only way forward is to work on simplifying the
> >>>> installation and then persuade the node managers to adopt the
> >>>> standard. Making it the default would, as you suggest, be a huge
> >help.
> >>>>> I keep telling our users in the UK that the archive is currently
> >in
> >>>> a very early stage, with a significant chance that data will be
> >>>> replaced. The same applies to the level of service. I think we
> >need to
> >>>> work on demonstrating best practise as far as data node deployment
> >>>> goes.
> >>>> Regarding this I've added a notice that I tried to push through
> >go-
> >>>> essp
> >>>> without any success. You can see it in our test Gateway, but I'll
> >be
> >>>> moving to our production one as soon as we sort other things up.
> >>>> I'd like to here your comment on this:
> >>>> http://albedo2.dkrz.de/esgcet/home.htm
> >>>>
> >>>> Thanks,
> >>>> Estani
> >>>>
> >>>>> At the moment I see the PKI security as a higher priority, since
> >>>> most of our users want scripting access rather than clicking
> >through
> >>>> the gateways, and this only works when the PKI security is
> >enabled.
> >>>>> For the versioning implementation, it would help to have a step
> >by
> >>>> step guide on esgf.org (or if it is already there, it would help
> >me to
> >>>> understand the issues if I knew where it is) -- but I guess this
> >will
> >>>> have to wait until Stephen has worked through some other
> >priorities.
> >>>>> It should be possible to get all this fixed in time, but I think
> >>>> people are working through a large number of issues in parallel at
> >>>> present.
> >>>>> Cheers,
> >>>>> Martin
> >>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-
> >>>>>>> bounces at ucar.edu] On Behalf Of Estanislao Gonzalez
> >>>>>>> Sent: 07 July 2011 16:46
> >>>>>>> To: go-essp-tech at ucar.edu
> >>>>>>> Subject: [Go-essp-tech] DRS, version number&   Co
> >>>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> What's the current stand regarding DRS and dataset version
> >number?
> >>>>>>>
> >>>>>>> I've seen too many data nodes with too many different
> >>>> configurations.
> >>>>>>>    From invalid datasets name to invalid DRS structure, names,
> >>>> missing
> >>>>>>> version numbers, etc.
> >>>>>>> The version number is a particular interesting one, since in
> >some
> >>>>>>> cases
> >>>>>>> the only way to find it is by parsing the TDS Catalogs
> >themselves,
> >>>>>>> since
> >>>>>>> the Gateway is not providing this info (AFAICT) and if the DRS
> >is
> >>>> not
> >>>>>>> followed can neither be implied from the directory structure of
> >>>> its
> >>>>>>> files.
> >>>>>>>
> >>>>>>> Obviously neither the publisher nor the Gateway is enforcing
> >those
> >>>>>>> constraints. I think this should be changed ASAP.
> >>>>>>> Both Node and Gateway publishing steps should enforce this when
> >>>>>>> publishing for cmip5. I think is the most direct way to get to
> >the
> >>>>>>> publisher at the right time.
> >>>>>>>
> >>>>>>> If we keep drifting away from what we already agreed on, we
> >won't
> >>>> be
> >>>>>>> able to do anything useful with the data at all, since we won't
> >be
> >>>>>>> able
> >>>>>>> to handle it properly.
> >>>>>>>
> >>>>>>> I'll urge the data node managers to check DRS compliance.
> >>>>>>>
> >>>>>>> I've only seen BADC publishing according to the DRS structure.
> >I
> >>>> know
> >>>>>>> PCMDI, BCC, CNRM and NCCS are not. I haven't checked others.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Estani
> >>>>>>>
> >>>>>>> --
> >>>>>>> Estanislao Gonzalez
> >>>>>>>
> >>>>>>> Max-Planck-Institut für Meteorologie (MPI-M)
> >>>>>>> Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing
> >>>> Centre
> >>>>>>> Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
> >>>>>>>
> >>>>>>> Phone:   +49 (40) 46 00 94-126
> >>>>>>> E-Mail:  gonzalez at dkrz.de
> >>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> GO-ESSP-TECH mailing list
> >>>>>>> GO-ESSP-TECH at ucar.edu
> >>>>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> >>>>
> >>>> --
> >>>> Estanislao Gonzalez
> >>>>
> >>>> Max-Planck-Institut für Meteorologie (MPI-M)
> >>>> Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing
> >Centre
> >>>> Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
> >>>>
> >>>> Phone:   +49 (40) 46 00 94-126
> >>>> E-Mail:  gonzalez at dkrz.de
> >
> >
> >--
> >Estanislao Gonzalez
> >
> >Max-Planck-Institut für Meteorologie (MPI-M)
> >Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
> >Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
> >
> >Phone:   +49 (40) 46 00 94-126
> >E-Mail:  gonzalez at dkrz.de

-- 
Scanned by iCritical.