[Go-essp-tech] NetCDF4 compression. write efficiency and CMIP5policy
Bentley, Philip
philip.bentley at metoffice.gov.uk
Thu Nov 12 06:56:32 MST 2009
Hi Martin,
Purely out of curiosity :-) it would be interesting to know what the
comparable figure is for writing vanilla netcdf-3 classic output
(uncompressed only, of course). Would it be easy for you to produce that
figure for the same test file?
Regards,
Phil
> -----Original Message-----
> From: go-essp-tech-bounces at ucar.edu
> [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of
> martin.juckes at stfc.ac.uk
> Sent: 12 November 2009 11:47
> To: doutriaux1 at llnl.gov; williams13 at llnl.gov
> Cc: go-essp-tech at ucar.edu
> Subject: Re: [Go-essp-tech] NetCDF4 compression. write
> efficiency and CMIP5policy
>
> Hi Charles,
>
> I've been looking at write speed from fortran, using dummy
> data, looking at a single array of size 360x180x120.
>
> The speed of writing compressed data depends on the choice of
> chunk sizes, and I'm not sure I've got the optimal values.
> The tables below list a range of choices (the first line for
> uncompressed data), which give speeds from 6 to 24 MB/s.
> These choices also affect the degree of compression and the
> speed which different subsets of a file can be read, so
> determining the right values is not a trivial exercise. I
> have just started exploring the possibilities, and would
> appreciate any guidance you have,
>
> Cheers,
> Martin
>
> File size (MB) Write speed (MB/s)
> 31.1 360 180 120 222.5
> 15.0 8 8 8 7.7
> 16.8 18 8 4 9.1
> 20.3 90 4 1 6.0
> 15.8 90 45 2 17.0
> 16.6 90 45 1 12.1
> 16.2 32 32 1 10.5
> 15.1 32 32 2 14.0
>
> And for a larger field, 360x180x2400:
> 622.09 360 180 2400 512.39
> 292.16 8 8 8 11.3
> 309.28 18 8 4 14.78
> 340.76 90 4 1 8.69
> 279.29 90 45 2 24
> 273.24 90 45 1 12.94
> 269.25 32 32 1 8.1
> 265.75 32 32 2 11.78
>
More information about the GO-ESSP-TECH
mailing list