[Go-essp-tech] Latest discrepancies among datasets available from different gateways

[Estanislao Gonzalez]

Hi Karl,

I think some institutions allow different publishers (probably modelers) 
to publish data. I assume this happens in small batches so I guess you 
might see a small number of datasets missing from time to time.
Sadly this will hide true problems, so I guess it's just better to 
report them.

What about synchronizing daily updates to UTC midnight? Would that 
help/make sense? It's not possible with the current Gateway though...

Thanks,
Estani
Am 01.08.2011 18:24, schrieb Karl Taylor:
> Hi Estani,
>
>  I understand that discrepancies can arise because things are only 
> synced once a day.  I was a little surprised that you were missing 
> only 8 datasets, as I assumed that folks would usually publish in 
> larger chunks.
>
> Anyway, thanks for looking into it, and I agree that once a day is fine.
>
> Best regards,
> Karl
>
> On 7/29/11 7:49 AM, Estanislao Gonzalez wrote:
>> Hi Karl,
>>
>> I'm checking this, but there will always be discrepancies as the
>> harvesting gets triggered once every while and not immediately after
>> changes are performed.
>>
>> Harvesting is not a cheap operation as AFAIK everything is sent and
>> then compared to what was already present. This will grow considerably
>> (harvesting PCMDI already takes more than a minute).
>>
>> After re-triggering the harvesting operations we have exactly the same
>> values as PCMDI and BADC (which are at this time 8193).
>>
>> The harvesting operations are triggered once per day in our case. I
>> don't think there's a real benefit for increasing this frequency, but we
>> could do it if it's required.
>>
>> Thanks,
>> Estani
>>
>> On Wed, 27 Jul 2011 15:53:53 -0700, Karl Taylor wrote:
>>> Dear all,
>>>
>>>   Currently (as of a few minutes ago) the ESG sites are showing the
>>> following number of CMIP5 datasets:
>>>
>>>   PCMDI 8040
>>>   BADC 8040
>>>   NCAR 8040
>>>   NCI 8041
>>>   DKRZ 8032
>>>   ORNL 8031
>>>   NERSC 7525
>>>   JPL 7524
>>>
>>>   I think we know about the NCI single extra dataset, which they're
>>> presumably working on removing. Also we know NERSC and JPL are not
>>> seeing data at the NCI gateway (CSIRO model). Can anyone explain
>>>
>>>   1) the missing 8 datasets at DKRZ and the 9 missing datasets at
>>> ORNL?
>>>   2) the difference of 1 between NERSC and JPL?
>>>
>>>   Best regards,
>>>   Karl
>
>
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech


-- 
Estanislao Gonzalez

Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany

Phone:   +49 (40) 46 00 94-126
E-Mail:  gonzalez at dkrz.de

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20110802/d77abef0/attachment.html