[Go-essp-tech] Fwd: Expected number of variables for which quality control will be needed

Thu Jan 6 09:35:17 MST 2011

Hello everyone,

I hope we can get some conclusion on what constitutes passed QC L2 before the Asheville meeting -- we have some UKMO data which appears to pass all the tests -- but it is hard to be sure because of the complexity of the output from the quality control code. We would like to declare this data as passed, so that we can get onto the next problem (replicating it to other centres).

The quality control document (attached) lists 4 objective tests under QC level 2:
(1) Number of records in each file consistent with metadata
(2) Regular time steps
(3) Metadata consistent with data request
(4) Minimum and maximum checked against specified ranges (or a default based on the mean).

An email from Martina this morning implied that subjective tests would only come in at the QC L3 stage or if there is some doubt about the objective tests. 

I'm not clear why the process is so complex when the document only specifies a small number of objective tests -- though there clearly is complexity of the workflow (ensuring that results for tests of all files are recorded and retrievable). 

Regards,
Martin

-----Original Message-----
From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of Michael Lautenschlager
Sent: 06 January 2011 16:16
To: go-essp-tech at ucar.edu
Subject: Re: [Go-essp-tech] Fwd: Expected number of variables for which quality control will be needed

Hi Martina, Bryan, Martin, Karl, and .....

this discussion stream fits a discussion Karl raised shortly before 
Christmas when Martina and Frank are captured in San Francisco due to 
snow air traffic shut down in Europe. Karl had been asking for 
interpretation of QC-L2 output interpretation. This is connected to two 
more questions:
the complexity of QC-L2 checks (What can be achieved in an acceptable 
period of time?) and
the definition of criteria for the assignment of the data flag "QC-L2 
passed".

The complexity discussion of QC-L2 checks has just been started on this 
thread and should be continued. But I would like to see also a 
discussion about the criteria for assignment of "QC-L2 passed" to CMIP5 
data. The problem I see is that QC-L2 does not give a clear yes/no or 
white/black decision (as for the CMOR-2 compliance). QC-L2 disseminate 
also grey colour results and we have to deal with these grey colours. 
How much grey colour we can accept without loosing too much quality?

I think we have to discuss precisely what tests can be achieved in QC-L2 
with respect to work load and benefit and develop guidelines to weight 
the results.

Since I am not continuously available for the next weeks I  suggest that 
Martina and Frank own this discussion for DKRZ as Stephan and Estani do 
for the technical part.

With respect to our upcoming GO-ESSP meeting May 2011 in Asheville I 
think we could present our CMIP5 quality control management and discuss 
it in a slightly wider community.

So far for the moment my ideas to this thread on QC in CMIP5.

Best wishes, Michael

---------------
Dr. Michael Lautenschlager

German Climate Computing Centre (DKRZ)
World Data Center Climate (WDCC)
ADDRESS: Bundesstrasse 45a, D-20146 Hamburg, Germany
PHONE:   +4940-460094-118
E-Mail:  lautenschlager at dkrz.de

URL:    http://www.dkrz.de/
         http://www.wdc-climate.de/

Geschäftsführer: Prof. Dr. Thomas Ludwig
Sitz der Gesellschaft: Hamburg
Amtsgericht Hamburg HRB 39784

Am 06.01.2011 12:13, schrieb Martina Stockhause:
>    Hi, Bryan,
>
> I expect that
> - existing errors are found and documented by the QC tool
> - errors are analysed and catagorised by the select-Script (part of the
> wrapper package) : We are still working on this catagorisation and
> evaluation of errors.
>
> The plots are an additional help if the person running the QC is in
> doubt and for documentation of QC results. If there were errors in the
> data  not found by the QC tool, I would expect that they were visible
> within the first few plots.
>
> During QC L3 checks we will double-check the QC L2 results using
> logfiles and plots for spot checking.
>
> Best wishes,
> Martina
>
>
> On 01/05/2011 06:11 PM, Bryan Lawrence wrote:
>> Hi Karl, Martina
>>
>> So are we really expecting PCMDI to look at more than 10,000 plots per
>> day as part of QC level 2?
>>
>> (Martin's numbers are on an internal wiki, but a back of the envelope
>> calculation goes something like:
>>    - one plot per atomic data set,
>>    - o(10^6) atomic datasets
>>    - 100 days
>>    - o(10^4) per day at PCMDI given BADC and DKRZ doing a negligible
>> amount cf PCMDI under current plans)
>>
>> Cheers
>> Bryan
>>
>> ---- Original Message ----
>>> From: "Juckes, Martin (STFC,RAL,SSTD)"<martin.juckes at stfc.ac.uk>
>>> To: "Lawrence, Bryan (STFC,RAL,SSTD)"<bryan.lawrence at stfc.ac.uk>,
>> badc<badc-internal at zonda.badc.rl.ac.uk>
>>> CC: "Pepler, Sam (STFC,RAL,SSTD)"<sam.pepler at stfc.ac.uk>
>>> Subject:Expected number of variables for which quality control will be
>> needed
>>> Following the discussion in the CMIP5 meeting this morning, I've put
>>> some estimates of numbers of variables in
>>> http://proj.badc.rl.ac.uk/badc/wiki/Ar5Cmip5/MOHCCmip5/VolumePredict
>>> ions
>>>
>>> It looks as though we will have up to 40,000 from UKMO (including
>>> HiGEM). If we want to do this in around 100 working days (leaving
>>> some room for repeats), we have 400 plots to verify per day. This
>>> much is, I think, manageable. If we expand it to, say, a third of
>>> the CMIP5 experiment, we will have 4,000 per day, which looks
>>> problematic -- but if it has to be done we could probably keep the
>>> plot inspection time quite small.
>>>
>>> Cheers,
>>> Martin
>>>
>>>
>>> -----Original Message-----
>>> From: Bryan Lawrence [mailto:bryan.lawrence at stfc.ac.uk]
>>> Sent: 05 January 2011 15:51
>>> To: badc
>>> Cc: Pepler, Sam (STFC,RAL,SSTD); Juckes, Martin (STFC,RAL,SSTD)
>>> Subject: cmip5 meeting summary
>>>
>>> Hi Folks
>>>
>>> I've posted a short (2 page) doc suitable for public viewing of where
>>> we are at with cmip5 support on the MIRP website at
>>> http://proj.badc.rl.ac.uk/mirp/wiki/CMIP5status
>>>
>>> If I can't make future meetings, could Sam or Martin please ensure
>>> that a version of this document is updated for future meetings.
>>>
>>> Cheers
>>> Bryan
>>>
>>>
>>> --
>>> Bryan Lawrence
>>> Director of Environmental Archival and Associated Research
>>> (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
>>> STFC, Rutherford Appleton Laboratory
>>> Phone +44 1235 445012; Fax ... 5848;
>>> Web: home.badc.rl.ac.uk/lawrence
>>> --
>>> Scanned by iCritical.
>> --
>> Bryan Lawrence
>> Director of Environmental Archival and Associated Research
>> (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
>> STFC, Rutherford Appleton Laboratory
>> Phone +44 1235 445012; Fax ... 5848;
>> Web: home.badc.rl.ac.uk/lawrence
>

_______________________________________________
GO-ESSP-TECH mailing list
GO-ESSP-TECH at ucar.edu
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech

-- 
Scanned by iCritical.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: CMIP5-AR5-QualityControl-100820.pdf
Type: application/pdf
Size: 711792 bytes
Desc: CMIP5-AR5-QualityControl-100820.pdf
Url : http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20110106/dbb0e2a5/attachment-0001.pdf 