[Go-essp-tech] comparison of GDS2.0 with climate modellers format CMIP5

[Craig Donlon]

Hi all
Just as background for those who have not mer Bryan, he is the director if the British atmospheric data centre  (BADC) Centre for environmental data archival (ceda) and has worked extensively with eo and model data for cmip5.

He's also a good guy and can help us understand how beat to get ghrsst data to the climate modelling community (as opposed to nrt and science users).

All the best
Craig

Sent from my iPhone
--
Dr Craig Donlon
Principal Scientist for Oceans and Ice
ESA/ESTEC (EOP-SME)
Keplerlaan 1, 2201 AZ
Noordwijk The Netherlands

t: +31 (0)715 653687
f: +31 (0)715 655675
e: craig.donlon at esa.int
m:+31 (0)627 013244
Skype ID:crazit

On 18 Mar 2011, at 20:31, Bryan Lawrence <bryan.lawrence at stfc.ac.uk> wrote:

> Hi Folks
> 
> I've seen some of your correspondence on the above subject.
> 
> Suffice to I think it'd be helpful if this discussion was  conducted on a 
> slightly wider stage. To that end, I've copied in the go- essp-tech 
> list, where you'll get the folk who have devised the CMIP5 
> data standards - and data distiribution system. 
> 
> There are some general points that might help in the discussoin to 
> follow:
> 
> CMOR is absolutely a tool devised for climate model data output, and 
> some of the things you find strange (demanding double precisoin etc) are 
> absolutely necessary in that context ...
> 
> The decision to use NetCDF3 rather than NetCDF4 was taken some time ago 
> after much discussion and heartache (and we had just about settled on 
> NetCDF4 before we did an "about turn"). In practice, many of the reasons 
> we used are probably not now relevant, but we are where we are ... wrt 
> the CMIP5 model data! (Which is to say I think there might be room to do 
> things differently with the EO data.)
> (Incidentally however, the 2 GB limit is helpful in chunking data over 
> low bandwidth links ... and we need to deal with between file aggregation 
> for many other reasons, so it's not a big deal if we break things up).
> 
> Which brings me to this:  The CMIP5 community is working to 
> accommodating  EO data, but indeed there are signifcant diffences, and 
> many of those are  obviously evident in the relative importance folks 
> apply to the various  CF headers. My personal opinion is that the way 
> forward is to itemise  the specific difference, and then have a disucssion 
> as to why one might do  things in a specific way.
> 
> For example, if you want level 3 EO data to be easily useful by the CMIP 
> community (and I believe you do), then I suggest you conform as closely 
> as you can to the CMIP5 paradigm.  Remember that the CMIP5 protocol is 
> already the joint agreement of hundreds of climate modellers ...
> 
> (In general my rule of thumb is organise the data for the consumers, not 
> the providers!)
> 
> Clearly however, most level 2 data is going to be consumed by folks who 
> are far more "satellite-aware" ... there you could be proposing 
> suggested accommodations within the CMIP5 frame (i.e. getting the CMIP5 
> community to extend their protocols,  not change them, there is no 
> chance of that now given the amount of effort being expended worldwide to 
> try and conform with what we have ... the last thing anyone in the 
> modelling community wants is a  moving target for *their* output formats 
> etc).
> 
> Why do I think you should still do this in the CMIP5 frame, rather than 
> just do your own thing and expose it somehow to the climate community?
> Because it's not just about the applications at the user end, it's also 
> about the metadata and data distribution systems. If we get it right, 
> we can use ESGF to replicate your data globally, making it easier to 
> consume (even as we provide adequate logging etc so data downloads are 
> attributed to the data provider, no matter where the data is downloaded 
> from). We can also exploit the tools that are being built in the ESG 
> community to manipulate the data ....
> 
> Ok, so taking some more specific points from the emails I have seen:
> 
> scale factors etc. For model data, precision matters because of the 
> necessity to do post processing budget studies. No such argument applies 
> to EO data (especially after being munged to level 3 in some 
> unphysically  interesting way ... it might be important if it were done 
> using a physical reanalysis). But In truth, the volume of EO data in 
> level 3 is going to  be trivial compared to the amount of model data, 
> and most (climate)  folks wont have the code all set up to the scaling 
> offset stuff. Yes it's  trivial to do, but using my rubric about consumers 
> above, I'd  suggest you just  put the data in using the correct physical 
> value with  respect to the CF units. Likewise native floats etc. Don't 
> make it harder  for the consumer .... (including the tools mentioned 
> above).
> 
> CMIP has a number of levels of metadata requirements, including both CF 
> header requirements, and directory layout. Some thoughts on dealing with 
> this for EO (and other observatioinal data)  can be found at 
> https://oodt.jpl.nasa.gov/wiki/display/CLIMATE/Data+and+Metadata+Requirements+for+CMIP5+Observational+Datasets 
> as you have found. It'd be good if you engaged directly with the authors 
> of that page, to make constructive suggesitons about the way forward ... 
> but some of the decisions you don't like (one variable per file etc) are 
> pretty much non-negotiable ... there are good reasons spanning back over 
> years as to why this is done.
> 
> ... but most of the EO side of things are far from cast in stone, so get 
> involved now ... but quickly.
> 
> Hope this is helpful. 
> 
> Regards,
> Bryan
> 
> --
> Bryan Lawrence
> Director of Environmental Archival and Associated Research
> (NCAS/British Atmospheric Data Centre and NCEO/NERC NEODC)
> STFC, Rutherford Appleton Laboratory
> Phone +44 1235 445012; Fax ... 5848; 
> Web: home.badc.rl.ac.uk/lawrence