[Go-essp-tech] Handling missingdata in the CMIP5 archive

[Bentley, Philip]

Hi George,

Your chosen solution to use a metadata attribute ('nodata' in your case)
to flag that a given file is an empty/null file is exactly the solution
that I had in mind for CMIP5 files comprised of all missing data.

Unfortunately - and the reason I didn't pursue it on mailing lists -
such files would not, I think, be CF-compliant and as such would likely
trip up current netCDF client software (and certainly the tools likely
to be used for analysing CMIP5 datasets).

Although it's probably too late to use this device on the CMIP5 project,
nonetheless I wonder if it isn't worth making a proposal along these
lines to the CF mailing list?

Regards,
Phil

> -----Original Message-----
> From: go-essp-tech-bounces at ucar.edu 
> [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of George J. Huffman
> Sent: 24 June 2011 14:23
> To: Kettleborough, Jamie
> Cc: go-essp-tech at ucar.edu
> Subject: Re: [Go-essp-tech] Handling missingdata in the CMIP5 archive
> 
> Hi all - to quote a different context ... in the 
> Precipitation Processing System, which is the processing 
> center for TRMM and GPM satellite project data, the choice is 
> to provide a file whether or not the data actually exist.  If 
> parts of the file are missing, they are filled with the 
> missing value, as you'd expect.  If the entire contents of 
> the file are unavailable, the metadata in the header includes 
> a "nodata=true" flag and no space is wasted.  To follow up on 
> the "early failure" comments, if you process the header for 
> the nodata flag, you'd immediately hit it, and if you don't, 
> you'd immediately hit read failures.  As a visual check, the 
> all-missing file's size is tiny compared to the usual file 
> that has data.
> 
> George
>