[Go-essp-tech] NetCDF4 compression. write efficiency and CMIP5policy

[Bentley, Philip]

Hi Martin,

Purely out of curiosity :-) it would be interesting to know what the
comparable figure is for writing vanilla netcdf-3 classic output
(uncompressed only, of course). Would it be easy for you to produce that
figure for the same test file?

Regards,
Phil

> -----Original Message-----
> From: go-essp-tech-bounces at ucar.edu 
> [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of 
> martin.juckes at stfc.ac.uk
> Sent: 12 November 2009 11:47
> To: doutriaux1 at llnl.gov; williams13 at llnl.gov
> Cc: go-essp-tech at ucar.edu
> Subject: Re: [Go-essp-tech] NetCDF4 compression. write 
> efficiency and CMIP5policy
> 
> Hi Charles,
> 
> I've been looking at write speed from fortran, using dummy 
> data, looking at a single array of size 360x180x120.
> 
> The speed of writing compressed data depends on the choice of 
> chunk sizes, and I'm not sure I've got the optimal values. 
> The tables below list a range of choices (the first line for 
> uncompressed data), which give speeds from 6 to 24 MB/s. 
> These choices also affect the degree of compression and the 
> speed which different subsets of a file can be read, so 
> determining the right values is not a trivial exercise. I 
> have just started exploring the possibilities, and would 
> appreciate any guidance you have,
> 
> Cheers,
> Martin 
> 
> File size (MB)				Write speed (MB/s)
> 31.1	360	180	120	222.5
> 15.0	8	8	8	7.7
> 16.8	18	8	4	9.1
> 20.3	90	4	1	6.0
> 15.8	90	45	2	17.0
> 16.6	90	45	1	12.1
> 16.2	32	32	1	10.5
> 15.1	32	32	2	14.0
> 
> And for a larger field, 360x180x2400:
> 622.09	360	180	2400	512.39
> 292.16	8	8	8	11.3
> 309.28	18	8	4	14.78
> 340.76	90	4	1	8.69
> 279.29	90	45	2	24
> 273.24	90	45	1	12.94
> 269.25	32	32	1	8.1
> 265.75	32	32	2	11.78
>