[Go-essp-tech] NetCDF4 compression. write efficiency and CMIP5 policy
Dean N. Williams
williams13 at llnl.gov
Tue Nov 10 05:50:34 MST 2009
This is not what we experienced. In fact, I recall just the opposite.
Charles Doutriaux did the work, so I'll let him respond directly. Also
it would be good if Ed Hartnett and Russ Rew respond as well to the
slowness that you are seeing. They may be able to help you on this.
Charles, if I recall correctly, were the zlib compressed netCDF files
read faster in CDAT?
We are using CMOR2, in which CMOR2 does the DRS, netCDF-4 classic
compressed output.
Best regards,
Dean
On Nov 10, 2009, at 4:20 AM, <stephen.pascoe at stfc.ac.uk> wrote:
> Hi Dean
>
> In tests we've done at BADC we have experienced 10-20x slowdown in
> write speed with NetCDF4 compression. Is this typical and are
> modelling centres aware that they can expect a significant I/O
> bottleneck?
>
> This makes me think, have we said CMIP5 data *must* be compressed?
> Is there a danger we will get a higher volume of data than we expect
> because it will be uncompressed to speed up the process at the
> modelling centres? How would we enforce NetCDF compression --
> presumably it would be discovered during replication.
>
> Cheers,
> Stephen.
>
> ---
> Stephen Pascoe +44 (0)1235 445980
> British Atmospheric Data Centre
> Rutherford Appleton Laboratory
>
>
> --
> Scanned by iCritical.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20091110/acde2acf/attachment.html
More information about the GO-ESSP-TECH
mailing list