[Go-essp-tech] Latest discrepancies among datasets available from different gateways
Estanislao Gonzalez
gonzalez at dkrz.de
Tue Aug 2 01:59:37 MDT 2011
Hi Karl,
I think some institutions allow different publishers (probably modelers)
to publish data. I assume this happens in small batches so I guess you
might see a small number of datasets missing from time to time.
Sadly this will hide true problems, so I guess it's just better to
report them.
What about synchronizing daily updates to UTC midnight? Would that
help/make sense? It's not possible with the current Gateway though...
Thanks,
Estani
Am 01.08.2011 18:24, schrieb Karl Taylor:
> Hi Estani,
>
> I understand that discrepancies can arise because things are only
> synced once a day. I was a little surprised that you were missing
> only 8 datasets, as I assumed that folks would usually publish in
> larger chunks.
>
> Anyway, thanks for looking into it, and I agree that once a day is fine.
>
> Best regards,
> Karl
>
> On 7/29/11 7:49 AM, Estanislao Gonzalez wrote:
>> Hi Karl,
>>
>> I'm checking this, but there will always be discrepancies as the
>> harvesting gets triggered once every while and not immediately after
>> changes are performed.
>>
>> Harvesting is not a cheap operation as AFAIK everything is sent and
>> then compared to what was already present. This will grow considerably
>> (harvesting PCMDI already takes more than a minute).
>>
>> After re-triggering the harvesting operations we have exactly the same
>> values as PCMDI and BADC (which are at this time 8193).
>>
>> The harvesting operations are triggered once per day in our case. I
>> don't think there's a real benefit for increasing this frequency, but we
>> could do it if it's required.
>>
>> Thanks,
>> Estani
>>
>> On Wed, 27 Jul 2011 15:53:53 -0700, Karl Taylor wrote:
>>> Dear all,
>>>
>>> Currently (as of a few minutes ago) the ESG sites are showing the
>>> following number of CMIP5 datasets:
>>>
>>> PCMDI 8040
>>> BADC 8040
>>> NCAR 8040
>>> NCI 8041
>>> DKRZ 8032
>>> ORNL 8031
>>> NERSC 7525
>>> JPL 7524
>>>
>>> I think we know about the NCI single extra dataset, which they're
>>> presumably working on removing. Also we know NERSC and JPL are not
>>> seeing data at the NCI gateway (CSIRO model). Can anyone explain
>>>
>>> 1) the missing 8 datasets at DKRZ and the 9 missing datasets at
>>> ORNL?
>>> 2) the difference of 1 between NERSC and JPL?
>>>
>>> Best regards,
>>> Karl
>
>
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
--
Estanislao Gonzalez
Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: gonzalez at dkrz.de
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20110802/d77abef0/attachment.html
More information about the GO-ESSP-TECH
mailing list