[Go-essp-tech] Handling missingdata in the CMIP5 archive

[Kettleborough, Jamie]

Hello Karl,

Is this extra flexibility really needed?  I think that giving
alternative 'representations' of the fact that some data sets have
unavailable time slices may make it harder for the data user.  Don't
they have to write code to deal with all the alternatives? 

Sorry this is so terse and there are other things in your e-mail I
haven't commented on - because I haven't had chance to think through
what you are saying.  I'm also aware you need to get a decision made on
this so you can inform people what to do.

Jamie



> So, I'm inclined to allow some flexibility summarized here 
> since unless folks are careful, they'll make mistakes no 
> matter what we decide:
> 
> When isolated time-slices in a dataset are lost and it is 
> impossible to recover them, it is recommended that those 
> isolated missing time-slices be:
> 1) filled entirely with the "missing data" value, or
> 2) be entirely omitted from the file (making sure the 
> time-coordinate reflects their absence)
> 
> When significant portions of a time-series are omitted 
> (either my design or otherwise), one should simply not create 
> files for those portions of the time-series.  This might 
> require the user to divide data normally found in a single 
> file into two files.  For example, if 100-years of monthly 
> mean data are normally packaged into a single file, but a 
> decade  (i.e., 120 consecutive samples) is unavailable (say 
> years 40-49), the user should write instead two files, the 
> first with 40 years of day and the second with the last 50 
> years of data.
> 
> Further discussion invited.
>