[Go-essp-tech] NetCDF4 compression. write efficiency and CMIP5 policy

[stephen.pascoe at stfc.ac.uk]

Thanks Charles,
 
Martin did the tests our end so it would be useful to compare with him how you are testing.  Martin is at conference until Thursday so he may not respond immediately.
 
Cheers,
Stephen.
 
---
Stephen Pascoe  +44 (0)1235 445980
British Atmospheric Data Centre
Rutherford Appleton Laboratory
 

________________________________

From: Charles سمير Doutriaux [mailto:doutriaux1 at llnl.gov] 
Sent: 10 November 2009 15:01
To: Dean N. Williams
Cc: Pascoe, Stephen (STFC,RAL,SSTD); Juckes, Martin (STFC,RAL,SSTD); Stephens, Ag (STFC,RAL,SSTD); go-essp-tech at ucar.edu
Subject: Re: NetCDF4 compression. write efficiency and CMIP5 policy


Yes I did some testing on the speed and it was much faster to write compressed, which did make sense to me since there was so much less data to write, and cpu time nothing next to i/o time. 

I'll rerun the test again today.

C

On Nov 10, 2009, at 4:50 AM, Dean N. Williams wrote:


	This is not what we experienced. In fact, I recall just the opposite. Charles Doutriaux did the work, so I'll let him respond directly. Also it would be good if Ed Hartnett and Russ Rew respond as well to the slowness that you are seeing. They may be able to help you on this. 

	Charles, if I recall correctly, were the zlib compressed netCDF files read faster in CDAT?
	

	We are using CMOR2, in which CMOR2 does the DRS, netCDF-4 classic compressed output.

	Best regards,
	Dean

	On Nov 10, 2009, at 4:20 AM, <stephen.pascoe at stfc.ac.uk> wrote:


		Hi Dean
		 
		In tests we've done at BADC we have experienced 10-20x slowdown in write speed with NetCDF4 compression.  Is this typical and are modelling centres aware that they can expect a significant I/O bottleneck?
		 
		This makes me think, have we said CMIP5 data *must* be compressed?  Is there a danger we will get a higher volume of data than we expect because it will be uncompressed to speed up the process at the modelling centres?  How would we enforce NetCDF compression -- presumably it would be discovered during replication.
		 
		Cheers,
		Stephen.
		 
		---
		Stephen Pascoe  +44 (0)1235 445980
		British Atmospheric Data Centre
		Rutherford Appleton Laboratory
		 


		-- 
		Scanned by iCritical. 





-- 
Scanned by iCritical.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20091110/4e59e41d/attachment-0001.html