[Go-essp-tech] NetCDF4 compression. write efficiency and CMIP5 policy
stephen.pascoe at stfc.ac.uk
stephen.pascoe at stfc.ac.uk
Tue Nov 10 05:20:08 MST 2009
Hi Dean
In tests we've done at BADC we have experienced 10-20x slowdown in write
speed with NetCDF4 compression. Is this typical and are modelling
centres aware that they can expect a significant I/O bottleneck?
This makes me think, have we said CMIP5 data *must* be compressed? Is
there a danger we will get a higher volume of data than we expect
because it will be uncompressed to speed up the process at the modelling
centres? How would we enforce NetCDF compression -- presumably it would
be discovered during replication.
Cheers,
Stephen.
---
Stephen Pascoe +44 (0)1235 445980
British Atmospheric Data Centre
Rutherford Appleton Laboratory
--
Scanned by iCritical.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20091110/40657a4b/attachment.html
More information about the GO-ESSP-TECH
mailing list