[Go-essp-tech] Publishing dataset with option --update

[Karl Taylor]

Bob should be in before noon.  Let's wait for him to weigh in on this.

thanks,
Karl

On 12/21/11 8:40 AM, Serguei Nikonov wrote:
> Hi Nate,
>
> unfortunately this is not the only dataset I have a problem - there are at least
> 5 more. Should I unpublish them locally (db, thredds) and than create new
> version containing full set of files? What is the official way to update dataset?
>
> Thanks,
> Sergey
>
>
> On 12/20/2011 07:06 PM, Nathan Wilhelmi wrote:
>> Hi Bob/Mike,
>>
>> I believe the problem is that when files were added the timestamp on the dataset
>> wasn't updated.
>>
>> The triple store will only harvest datasets that have files and an updated
>> timestamp after the last harvest.
>>
>> So what likely happened is the dataset was created without files, so it wasn't
>> initially harvested. Files were subsequently added, but the timestamp wasn't
>> updated, so it was still not a candidate for harvesting.
>>
>> Can you update the date_updated timestamp for the dataset in question and then
>> trigger the RDF harvesting, I believe the dataset will show up then.
>>
>> Thanks!
>> -Nate
>>
>> On 12/20/2011 11:49 AM, Serguei Nikonov wrote:
>>> Hi Mike,
>>>
>>> I am a member of data publishers group. I have been publishing considerable
>>> amount of data without such kind of troubles but this one occurred only when I
>>> tried to add some files to existing dataset. Publishing from scratch works fine
>>> for me.
>>>
>>> Thanks,
>>> Sergey
>>>
>>> On 12/20/2011 01:29 PM, Ganzberger, Michael wrote:
>>>> Hi Serguei,
>>>>
>>>> That task is on a scheduler and will re-run every 10 minutes. If your data
>>>> does not appear after that time then perhaps there is another issue. One
>>>> issue could be that publishing to the gateway requires that you have the role
>>>> of "Data Publisher";
>>>>
>>>> "check that the account is member of the proper group and has the special
>>>> role of Data Publisher"
>>>>
>>>> http://esgf.org/wiki/ESGFNode/FAQ
>>>>
>>>> Mike
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Serguei Nikonov [mailto:serguei.nikonov at noaa.gov]
>>>> Sent: Tuesday, December 20, 2011 10:12 AM
>>>> To: Ganzberger, Michael
>>>> Cc: Stéphane Senesi; Drach, Bob; go-essp-tech at ucar.edu
>>>> Subject: Re: [Go-essp-tech] Publishing dataset with option --update
>>>>
>>>> Hi Mike,
>>>>
>>>> thansk for suggestion but I don't have any privileges to do anything on gateway.
>>>> I am just publishing data on GFDL data node.
>>>>
>>>> Regards,
>>>> Sergey
>>>>
>>>> On 12/20/2011 01:05 PM, Ganzberger, Michael wrote:
>>>>>
>>>>> Hi Serguei,
>>>>>
>>>>> I'd like to suggest this that may help you from
>>>>> http://esgf.org/wiki/Cmip5Gateway/FAQ
>>>>>
>>>>>
>>>>>
>>>>> "The search does not reflect the latest DB changes I've made
>>>>>
>>>>> You have to manually trigger the 3store harvesting. Logging as root and go
>>>>> to Admin->"Gateway Scheduled Tasks"->"Run tasks" and restart the job named
>>>>> RDFSynchronizationJobDetail"
>>>>>
>>>>> Mike Ganzberger
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-bounces at ucar.edu]
>>>>> On Behalf Of Stéphane Senesi
>>>>> Sent: Tuesday, December 20, 2011 9:42 AM
>>>>> To: Serguei Nikonov
>>>>> Cc: Drach, Bob; go-essp-tech at ucar.edu
>>>>> Subject: Re: [Go-essp-tech] Publishing dataset with option --update
>>>>>
>>>>> Serguei
>>>>>
>>>>> We have for some time now experienced similar problems when publishing
>>>>> to the PCMDI gateway, i.e. not getting a "SUCCESS" message when
>>>>> publishing . Sometimes, files are actually published (or at least
>>>>> accessible through the gateway, their status being actually
>>>>> "START_PUBLISHING", after esg_list_datasets report) , sometimes not. An
>>>>> hypothesis is that the PCMDI Gateway load do generate the problem. We
>>>>> havn't yet got a confirmation by Bob.
>>>>>
>>>>> In contrast to your case, this happens when publishing a dataset from
>>>>> scratch (I mean, not an update)
>>>>>
>>>>> Best regards (do not expect any feeback from me since early january, yet)
>>>>>
>>>>> S
>>>>>
>>>>>
>>>>> Serguei Nikonov wrote, On 20/12/2011 18:11:
>>>>>> Hi Bob,
>>>>>>
>>>>>> I needed to add some missed variables to existing dataset and I found in
>>>>>> esgpublish command an option --update. When I tried it I've got normal
>>>>>> message like
>>>>>> INFO 2011-12-20 11:21:00,893 Publishing:
>>>>>> cmip5.output1.NOAA-GFDL.GFDL-CM3.historical.mon.atmos.Amon.r1i1p1, parent =
>>>>>> pcmdi.GFDL
>>>>>> INFO 2011-12-20 11:21:07,564 Result: PROCESSING
>>>>>> INFO 2011-12-20 11:21:11,209 Result: PROCESSING
>>>>>> ....
>>>>>>
>>>>>> but nothing happened on gateway - new variables are not there. The files
>>>>>> corresponding to these variables are in database and in THREDDS catalog but
>>>>>> apparently were not published on gateway.
>>>>>>
>>>>>> I used command line
>>>>>> esgpublish --update --keep-version --map<map_file>  --project cmip5 --noscan
>>>>>> --publish.
>>>>>>
>>>>>> Should map file be of some specific format to make it works in mode I need?
>>>>>>
>>>>>> Thanks,
>>>>>> Sergey Nikonov
>>>>>> GFDL
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> GO-ESSP-TECH mailing list
>>>>>> GO-ESSP-TECH at ucar.edu
>>>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>>>>
>>>>>>
>>>>>
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20111221/6a7d0901/attachment.html