[Go-essp-tech] Publishing dataset with option --update

[Ashish Bhardwaj]

Hello Serguei,

You can say these to be the best CMIP5 data publishing 
practices/guidelines. I have not seen any official document enforcing 
data publisher(s) to do so. You can publish data without changing 
dataset version.

Same case apply with using checksums for the dataset published to the 
gateway. We know the importance of using checksums for the files. Don't we ?

There could also be a debate on using multiple versions. Some will argue 
it is not useful as only the latest one is accessible through gateway. I 
have not checked with ESGF P2P node.

 From CMIP5 users end point of view, using different versions 
will/should help them to know that something has changed in the dataset 
and it is a potential candidate to perform data download next time.

Ashish


Serguei Nikonov wrote:
> Hello Ashish,
>
> is it ESG official policy how to add files to existing datasets or 
> it's your suggestion? If it's the only way how to deal with updating 
> datasets why this option was implemented to esgpublisher?
>
> Sergey
>
>
> On 12/21/2011 05:44 AM, Ashish Bhardwaj wrote:
>> Hello,
>>
>> Any change (add/delete/update of files) in the dataset should be 
>> published with
>> a new dataset version. It will help CMIP5 users to know that 
>> something has
>> changed since the last time they downloaded a particular dataset.
>>
>> Ashish
>>
>>
>>
>>
>> Kettleborough, Jamie wrote:
>>> Hello,
>>>
>>> Isn't one way of dealing with this is simply to publish the whole
>>> dataset (old files and additional files) at a new version. In fact I
>>> thought that was the 'policy' - but I could have misunderstood.
>>>
>>> Jamie
>>>> -----Original Message-----
>>>> From: go-essp-tech-bounces at ucar.edu 
>>>> [mailto:go-essp-tech-bounces at ucar.edu] On
>>>> Behalf Of Serguei Nikonov
>>>> Sent: 20 December 2011 17:12
>>>> To: Drach, Bob
>>>> Cc: go-essp-tech at ucar.edu
>>>> Subject: [Go-essp-tech] Publishing dataset with option --update
>>>>
>>>> Hi Bob,
>>>>
>>>> I needed to add some missed variables to existing dataset and I 
>>>> found in
>>>> esgpublish command an option --update. When I tried it I've got normal
>>>> message like
>>>> INFO 2011-12-20 11:21:00,893 Publishing:
>>>> cmip5.output1.NOAA-GFDL.GFDL-CM3.historical.mon.atmos.Amon.r1i
>>> 1p1, parent = pcmdi.GFDL
>>>> INFO 2011-12-20 11:21:07,564 Result: PROCESSING
>>>> INFO 2011-12-20 11:21:11,209 Result: PROCESSING
>>>> ....
>>>>
>>>> but nothing happened on gateway - new variables are not there. The 
>>>> files
>>>> corresponding to these variables are in database and in THREDDS 
>>>> catalog but
>>>> apparently were not published on gateway.
>>>>
>>>> I used command line
>>>> esgpublish --update --keep-version --map <map_file> --project cmip5 
>>>> --noscan
>>>> --publish.
>>>>
>>>> Should map file be of some specific format to make it works in mode 
>>>> I need?
>>>>
>>>> Thanks,
>>>> Sergey Nikonov
>>>> GFDL
>>>>
>>>>
>>>> _______________________________________________
>>>> GO-ESSP-TECH mailing list
>>>> GO-ESSP-TECH at ucar.edu
>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>
>>
>


-- 
Ashish Bhardwaj

Software Engineer
Institut Pierre Simon Laplace
UPMC, Case 101, 4 place Jussieu,
75252 Paris Cedex 5

Tour 45-55 2ème étage Bureau 207
Tel: 33 1 44 27 49 10