[Go-essp-tech] [esgf-devel] Changing checksums without changing version number ?

Fri Dec 21 04:03:07 MST 2012

Hello all,

indeed Stéphane the best practice as things stand now is to republish a 
new version.

I converge with the Estani's idea that we might want to have checksums 
over the data part only (or versioning over the data part and versioning 
over the metadata part).

In Europe IS-ENES phase 1 will come to an end early next year. In this 
context we plan to write a data management white paper. I will draft a 
first version of this paper by mid-January. This white paper will 
emphasize best practices and recommendations in particular with respect 
to versioning (and how to keep track of changes and versions history). I 
already have a few names I plan to ask advices and/or contributions. If 
you want to contribute/share your ideas on this do not hesitate to ping me.

regards and merry Christmas!
Sébastien

Le 21/12/2012 11:27, Stéphane Senesi a écrit :
> Estanislao,
>
> Estanislao Gonzalez wrote, On 21/12/2012 10:25:
>> Hi All,
>>
>> IMHO:
>> new checksum == new version
>>
>> As the system is designed it means we will have more versions than
>> required if (and only if) the same data is regenerated with some
>> unimportant meta-data change (In all other cases, a new version is
>> compelling). I don't see a reason while this particular scenario should
>> happen often, but perhaps I'm wrong.
>> So if I were you Stéphane, I would make sure the data is generated
>> "exactly" as the old one (same checksum). If that's not possible, then
>> it's not the same data, ergo new version.
>>
>> In the future, we might want to have checksums over the data part only
>> and skip the meta-data (or at most include just some meta-data entries,
>> which I wouldn't by the way)
> In my case, the only metadata that I cannot re-construct is a small part
> of the 'history' attribute, namely the date at which the (other, useful)
> meta-data attributes were fixed ...  But I understand the argument.
>
> The need for a coordinated way to describe and publish (clear text)
> version information remains strong. Not sure about the usefulness of a
> CIM on version information ;-)
>
> S
>
>> . But as the system is designed right now
>> that's not possible.
>>
>> My 2c,
>> Estani
>>
>> On 21.12.2012 10:04, Kettleborough, Jamie wrote:
>>> Hello Karl,
>>>
>>> Is there any documentation anywhere on what changes should or
>>> shouldn't trigger new versions? Is it worth adding this advice to
>>> that
>>> documentation?
>>>
>>> Thanks,
>>>
>>> Jamie
>>>
>>>> -------------------------
>>>> FROM: go-essp-tech-bounces at ucar.edu
>>>> [mailto:go-essp-tech-bounces at ucar.edu] ON BEHALF OF Karl Taylor
>>>> SENT: 20 December 2012 17:51
>>>> TO: stephen.pascoe at stfc.ac.uk
>>>> CC: esgf-devel at lists.llnl.gov; go-essp-tech at ucar.edu
>>>> SUBJECT: Re: [Go-essp-tech] [esgf-devel] Changing checksums without
>>>> changing version number ?
>>>>
>>>> Hi Stephane and Stephen,
>>>>
>>>> I agree with Stephen that what you should do is not obvious, but I
>>>> think least confusing is to indeed publish as a new version. We'll
>>>> need to work to make easily accessible information about how versions
>>>> differ (or don't) so as to avoid folks reanalyzing output when it is
>>>> unnecessary.
>>>>
>>>> Karl
>>>>
>>>> On 12/20/12 2:42 AM, stephen.pascoe at stfc.ac.uk wrote:
>>>>
>>>>> Hi Stephane,
>>>>>
>>>>> This is a difficult use case to resolve. I have broadened the
>>>>> thread to go-essp-tech because it affects the whole plan of how we
>>>>> keep track of changing data.
>>>>>
>>>>> My opinion is that you should publish this data as a new version.
>>>>> We have been assuming that each dataset version has a stable set of
>>>>> checksums. We'd like to build tools around this assumption that
>>>>> checks the integrity of the archive (admittedly we haven't got there
>>>>> yet).
>>>>>
>>>>> If you republish files as the same versions but with different
>>>>> checksums we cannot tell that only the metadata has changed. Thinking
>>>>> about the archive as a whole, we have to assume that any
>>>>> file-versions that change checksum could be different data and flag
>>>>> it as such. It would be better to create a new version and document
>>>>> that this version is a trivial change.
>>>>>
>>>>> Unfortunately we don't have a good system for documenting these
>>>>> version transitions. BADC did produce a web-app for this some months
>>>>> ago but it didn't catch on [1]. Also there is a wiki page [2] where
>>>>> you can note down data provider issues but I doubt any users know it
>>>>> exists. If you record what you've done in one of those places the
>>>>> knowledge will not be lost.
>>>>>
>>>>> [1] http://cmip-ingest1.badc.rl.ac.uk/drscomm/search/ [1] (Contact
>>>>> Ag Stephens for details: CC'd above)
>>>>> [2] http://esgf.org/wiki/CMIP5ProviderFAQ [2]
>>>>>
>>>>> Cheers,
>>>>> Stephen.
>>>>>
>>>>> ---
>>>>> Stephen Pascoe +44 (0)1235 445980
>>>>> Centre of Environmental Data Archival
>>>>> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11
>>>>> 0QX, UK
>>>>>
>>>>> -----Original Message-----
>>>>> From: owner-esgf-devel at lists.llnl.gov
>>>>> [mailto:owner-esgf-devel at lists.llnl.gov] On Behalf Of Stéphane Senesi
>>>>> Sent: 19 December 2012 16:58
>>>>> To: esgf-devel at lists.llnl.gov
>>>>> Subject: [esgf-devel] Changing checksums without changing version
>>>>> number ?
>>>>>
>>>>> Dear all,
>>>>>
>>>>> We experienced a failure of a disk array for our CMIP5 data's ESG
>>>>> datanode. We are able to produce again the same data and metadata,
>>>>> except that, in each NetCDF file, a small part of the "history"
>>>>> metadata is changed (namely the date of the last setting for some
>>>>> metadata).
>>>>> Hence, the cheksum does change, and we have no way to avoid it.
>>>>>
>>>>> We can either re-publish the affected datasets with a new version
>>>>> number or with the same version number.
>>>>>
>>>>> In the first case, all users may think that the data is new, and
>>>>> will have to consider if they want to download it again, and, if they
>>>>> do, may eventually complain that we generate additional non-sensical
>>>>> work
>>>>>
>>>>> In the second one, meticulous users will complain that the
>>>>> checksums in our thredds catalog are not the same as the checksums
>>>>> for the files they have already downloaded
>>>>>
>>>>> What is the best way forward ? I suspect it is the second one,
>>>>> because checksums are not supposed to be data identifiers but only
>>>>> used for check of data corruption immediately after the transfer. But
>>>>> does everybody agree with that ?
>>>>>
>>>>> Regards
>>>>>
>>>>> --
>>>>> Stéphane Sénési
>>>>> Ingénieur - équipe Assemblage du Système Terre Centre National de
>>>>> Recherches Météorologiques Groupe de Météorologie à Grande Echelle et
>>>>> Climat
>>>>>
>>>>> CNRM/GMGEC/ASTER
>>>>> 42 Av Coriolis
>>>>> F-31057 Toulouse Cedex 1
>>>>>
>>>>> +33.5.61.07.99.31 (Fax :....9610)
>>> Links:
>>> ------
>>> [1] http://cmip-ingest1.badc.rl.ac.uk/drscomm/search/
>>> [2] http://esgf.org/wiki/CMIP5ProviderFAQ
>

-- 
Sébastien Denvil
IPSL, Pôle de modélisation du climat
UPMC, Case 101, 4 place Jussieu,
75252 Paris Cedex 5

Tour 45-55 2ème étage Bureau 209
Tel: 33 1 44 27 21 10
Fax: 33 1 44 27 39 02

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2958 bytes
Desc: Signature cryptographique S/MIME
Url : http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20121221/33d757f3/attachment.bin 