[Go-essp-tech] Publishing dataset with option --update
Ashish Bhardwaj
abhipsl at ipsl.jussieu.fr
Thu Dec 22 03:47:55 MST 2011
Hello Serguei,
You can say these to be the best CMIP5 data publishing
practices/guidelines. I have not seen any official document enforcing
data publisher(s) to do so. You can publish data without changing
dataset version.
Same case apply with using checksums for the dataset published to the
gateway. We know the importance of using checksums for the files. Don't we ?
There could also be a debate on using multiple versions. Some will argue
it is not useful as only the latest one is accessible through gateway. I
have not checked with ESGF P2P node.
From CMIP5 users end point of view, using different versions
will/should help them to know that something has changed in the dataset
and it is a potential candidate to perform data download next time.
Ashish
Serguei Nikonov wrote:
> Hello Ashish,
>
> is it ESG official policy how to add files to existing datasets or
> it's your suggestion? If it's the only way how to deal with updating
> datasets why this option was implemented to esgpublisher?
>
> Sergey
>
>
> On 12/21/2011 05:44 AM, Ashish Bhardwaj wrote:
>> Hello,
>>
>> Any change (add/delete/update of files) in the dataset should be
>> published with
>> a new dataset version. It will help CMIP5 users to know that
>> something has
>> changed since the last time they downloaded a particular dataset.
>>
>> Ashish
>>
>>
>>
>>
>> Kettleborough, Jamie wrote:
>>> Hello,
>>>
>>> Isn't one way of dealing with this is simply to publish the whole
>>> dataset (old files and additional files) at a new version. In fact I
>>> thought that was the 'policy' - but I could have misunderstood.
>>>
>>> Jamie
>>>> -----Original Message-----
>>>> From: go-essp-tech-bounces at ucar.edu
>>>> [mailto:go-essp-tech-bounces at ucar.edu] On
>>>> Behalf Of Serguei Nikonov
>>>> Sent: 20 December 2011 17:12
>>>> To: Drach, Bob
>>>> Cc: go-essp-tech at ucar.edu
>>>> Subject: [Go-essp-tech] Publishing dataset with option --update
>>>>
>>>> Hi Bob,
>>>>
>>>> I needed to add some missed variables to existing dataset and I
>>>> found in
>>>> esgpublish command an option --update. When I tried it I've got normal
>>>> message like
>>>> INFO 2011-12-20 11:21:00,893 Publishing:
>>>> cmip5.output1.NOAA-GFDL.GFDL-CM3.historical.mon.atmos.Amon.r1i
>>> 1p1, parent = pcmdi.GFDL
>>>> INFO 2011-12-20 11:21:07,564 Result: PROCESSING
>>>> INFO 2011-12-20 11:21:11,209 Result: PROCESSING
>>>> ....
>>>>
>>>> but nothing happened on gateway - new variables are not there. The
>>>> files
>>>> corresponding to these variables are in database and in THREDDS
>>>> catalog but
>>>> apparently were not published on gateway.
>>>>
>>>> I used command line
>>>> esgpublish --update --keep-version --map <map_file> --project cmip5
>>>> --noscan
>>>> --publish.
>>>>
>>>> Should map file be of some specific format to make it works in mode
>>>> I need?
>>>>
>>>> Thanks,
>>>> Sergey Nikonov
>>>> GFDL
>>>>
>>>>
>>>> _______________________________________________
>>>> GO-ESSP-TECH mailing list
>>>> GO-ESSP-TECH at ucar.edu
>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>
>>
>
--
Ashish Bhardwaj
Software Engineer
Institut Pierre Simon Laplace
UPMC, Case 101, 4 place Jussieu,
75252 Paris Cedex 5
Tour 45-55 2ème étage Bureau 207
Tel: 33 1 44 27 49 10
More information about the GO-ESSP-TECH
mailing list