[Go-essp-tech] NetCDF4 compression. write efficiency and CMIP5 policy

[stephen.pascoe at stfc.ac.uk]

Hi Dean
 
In tests we've done at BADC we have experienced 10-20x slowdown in write
speed with NetCDF4 compression.  Is this typical and are modelling
centres aware that they can expect a significant I/O bottleneck?
 
This makes me think, have we said CMIP5 data *must* be compressed?  Is
there a danger we will get a higher volume of data than we expect
because it will be uncompressed to speed up the process at the modelling
centres?  How would we enforce NetCDF compression -- presumably it would
be discovered during replication.
 
Cheers,
Stephen.
 
---
Stephen Pascoe  +44 (0)1235 445980
British Atmospheric Data Centre
Rutherford Appleton Laboratory
 

-- 
Scanned by iCritical.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20091110/40657a4b/attachment.html