[Go-essp-tech] NetCDF4 compression. write efficiency andCMIP5policy

[martin.juckes at stfc.ac.uk]

Hi Gary,

We are sure that there is no alternative to using compression at this
stage, which doesn't quite answer you question, but is relevant. We
would not be able to cope with the volume of data and the logistics of
moving it around if we could not use compression. The netcdf-4 library
is available and very easy for the technically aware to switch to now,
and I think we can be sure that it will percolate into user software
before the general release at the end of next year.

The issue now is how to configure the compression so that we get maximum
down the line benefits for minimum up front pain. This data is going to
be in use for a long time, so it is worth spending a few CPU cycles to
store it efficiently,

Cheers,
Martin

  
> -----Original Message-----
> From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-
> bounces at ucar.edu] On Behalf Of Gary Strand
> Sent: 12 November 2009 15:16
> To: go-essp-tech at ucar.edu
> Subject: Re: [Go-essp-tech] NetCDF4 compression. write efficiency
> andCMIP5policy
> 
> 
> Are we sure that there's sufficient usage of compressed netCDF-4, by
> both scientists/analysts and by clients (pick your favorite software)
> that we (modeling groups) must use compressed netCDF-4 for our CMIP5
> data submission?
> 
> With CMIP3, we were told to keep file sizes under 2 GB to allow for
> folks who had only 32-bit netCDF available. Is it a good idea to
> mandate compressed netCDF-4 this time around?
> 
> Gary Strand
> strandwg at ucar.edu
> 
> 
> 
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
-- 
Scanned by iCritical.