[Go-essp-tech] DRS, version number & Co

Sat Jul 9 13:15:59 MDT 2011

Hi Martin,

thanks a lot for the corrections. I just realized that I wrote something 
I haven't really meant:
"This process implies that data can be removed without previous notice. 
Changes will be notified here"

(Or maybe I did at that time... :-)

In any case QCL1 doesn't force the publisher to use a new version if 
anything gets changed nor even to notify of such changes. So I think we 
might want to change that. Regarding your correction, that might be:

[..] Withdrawals and replacements may happen without notification. [..]

But that sounds quite rude (indeed honest... but still... :-)

I don't see either the means or the will to publish every single change 
at this stage. We might be able to notify changes for MPI-M, but that's 
fairly easy as we are "close", it will still require some coordination 
though. I don't think that will happen between other institutions, e.g. 
BCC and PCMDI. This will require a large effort and further coordination.

Any comments on this?

Thanks,
Estani

Am 08.07.2011 18:41, schrieb martin.juckes at stfc.ac.uk:
> Hello Estani,
>
> I would delete "minimal" -- the syntactical check is not much, but it is better than many archives. I would suggest the following revised text:
>
> CMIP5 data has been quality checked for level one (QCL1) which guarantees minimal conformance. The data is currently being quality checked for level two (QCL2) to assure consistency. As a consequence of this process, data may be withdrawn from the archive and/or replaced with corrected data. Notification of withdrawals and replacements will be posted here. Please refer to this documentation for more information regarding CMIP5 quality assurance procedure.
>
> I've softened the language a little, but still convey the same message -- I think, trying to make it more explanatory than declaratory.
>
> Cheers,
> Martin
>
>>> -----Original Message-----
>>> From: Estanislao Gonzalez [mailto:gonzalez at dkrz.de]
>>> Sent: 08 July 2011 11:40
>>> To: Juckes, Martin (STFC,RAL,RALSP)
>>> Cc: go-essp-tech at ucar.edu
>>> Subject: Re: [Go-essp-tech] DRS, version number&  Co
>>>
>>> Hi Martin,
>>>
>>>   a short comment inline:
>>> Am 08.07.2011 11:09, schrieb martin.juckes at stfc.ac.uk:
>>>> Hi Estani,
>>>>
>>>> I agree with you that this is an important issue and that we want to
>>> have a clean implementation.
>>>> Unfortunately, given where we are now, I don't think there is going
>>> to be any support for withdrawing data nodes which don't meet this
>>> implementation standard -- so enforcement by the gateway won't work.
>>> So I think the only way forward is to work on simplifying the
>>> installation and then persuade the node managers to adopt the
>>> standard. Making it the default would, as you suggest, be a huge help.
>>>> I keep telling our users in the UK that the archive is currently in
>>> a very early stage, with a significant chance that data will be
>>> replaced. The same applies to the level of service. I think we need to
>>> work on demonstrating best practise as far as data node deployment
>>> goes.
>>> Regarding this I've added a notice that I tried to push through go-
>>> essp
>>> without any success. You can see it in our test Gateway, but I'll be
>>> moving to our production one as soon as we sort other things up.
>>> I'd like to here your comment on this:
>>> http://albedo2.dkrz.de/esgcet/home.htm
>>>
>>> Thanks,
>>> Estani
>>>
>>>> At the moment I see the PKI security as a higher priority, since
>>> most of our users want scripting access rather than clicking through
>>> the gateways, and this only works when the PKI security is enabled.
>>>> For the versioning implementation, it would help to have a step by
>>> step guide on esgf.org (or if it is already there, it would help me to
>>> understand the issues if I knew where it is) -- but I guess this will
>>> have to wait until Stephen has worked through some other priorities.
>>>> It should be possible to get all this fixed in time, but I think
>>> people are working through a large number of issues in parallel at
>>> present.
>>>> Cheers,
>>>> Martin
>>>>
>>>>>> -----Original Message-----
>>>>>> From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-
>>>>>> bounces at ucar.edu] On Behalf Of Estanislao Gonzalez
>>>>>> Sent: 07 July 2011 16:46
>>>>>> To: go-essp-tech at ucar.edu
>>>>>> Subject: [Go-essp-tech] DRS, version number&   Co
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> What's the current stand regarding DRS and dataset version number?
>>>>>>
>>>>>> I've seen too many data nodes with too many different
>>> configurations.
>>>>>>    From invalid datasets name to invalid DRS structure, names,
>>> missing
>>>>>> version numbers, etc.
>>>>>> The version number is a particular interesting one, since in some
>>>>>> cases
>>>>>> the only way to find it is by parsing the TDS Catalogs themselves,
>>>>>> since
>>>>>> the Gateway is not providing this info (AFAICT) and if the DRS is
>>> not
>>>>>> followed can neither be implied from the directory structure of
>>> its
>>>>>> files.
>>>>>>
>>>>>> Obviously neither the publisher nor the Gateway is enforcing those
>>>>>> constraints. I think this should be changed ASAP.
>>>>>> Both Node and Gateway publishing steps should enforce this when
>>>>>> publishing for cmip5. I think is the most direct way to get to the
>>>>>> publisher at the right time.
>>>>>>
>>>>>> If we keep drifting away from what we already agreed on, we won't
>>> be
>>>>>> able to do anything useful with the data at all, since we won't be
>>>>>> able
>>>>>> to handle it properly.
>>>>>>
>>>>>> I'll urge the data node managers to check DRS compliance.
>>>>>>
>>>>>> I've only seen BADC publishing according to the DRS structure. I
>>> know
>>>>>> PCMDI, BCC, CNRM and NCCS are not. I haven't checked others.
>>>>>>
>>>>>> Thanks,
>>>>>> Estani
>>>>>>
>>>>>> --
>>>>>> Estanislao Gonzalez
>>>>>>
>>>>>> Max-Planck-Institut für Meteorologie (MPI-M)
>>>>>> Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing
>>> Centre
>>>>>> Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
>>>>>>
>>>>>> Phone:   +49 (40) 46 00 94-126
>>>>>> E-Mail:  gonzalez at dkrz.de
>>>>>>
>>>>>> _______________________________________________
>>>>>> GO-ESSP-TECH mailing list
>>>>>> GO-ESSP-TECH at ucar.edu
>>>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>
>>> --
>>> Estanislao Gonzalez
>>>
>>> Max-Planck-Institut für Meteorologie (MPI-M)
>>> Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
>>> Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
>>>
>>> Phone:   +49 (40) 46 00 94-126
>>> E-Mail:  gonzalez at dkrz.de

-- 
Estanislao Gonzalez

Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany

Phone:   +49 (40) 46 00 94-126
E-Mail:  gonzalez at dkrz.de