[Go-essp-tech] NetCDF4 compression. write efficiency and CMIP5policy

[martin.juckes at stfc.ac.uk]

Hello Phil,

Not too hard -- just compiling the code against the earlier library and
deleting a few calls.

For the large file (380x180x2400), I get a write of 3.5 seconds with
netcdf 3 compared with 1.2 seconds for netcdf 4 uncompressed and 27 to
60 seconds for netcdf 4 compressed. 

So there is an impressive speed-up between netcdf 3 and netcdf 4. 

Cheers,
Martin

> -----Original Message-----
> From: Bentley, Philip [mailto:philip.bentley at metoffice.gov.uk]
> Sent: 12 November 2009 13:57
> To: Juckes, Martin (STFC,RAL,SSTD)
> Cc: go-essp-tech at ucar.edu
> Subject: RE: [Go-essp-tech] NetCDF4 compression. write efficiency and
> CMIP5policy
> 
> Hi Martin,
> 
> Purely out of curiosity :-) it would be interesting to know what the
> comparable figure is for writing vanilla netcdf-3 classic output
> (uncompressed only, of course). Would it be easy for you to produce
> that
> figure for the same test file?
> 
> Regards,
> Phil
> 
> > -----Original Message-----
> > From: go-essp-tech-bounces at ucar.edu
> > [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of
> > martin.juckes at stfc.ac.uk
> > Sent: 12 November 2009 11:47
> > To: doutriaux1 at llnl.gov; williams13 at llnl.gov
> > Cc: go-essp-tech at ucar.edu
> > Subject: Re: [Go-essp-tech] NetCDF4 compression. write
> > efficiency and CMIP5policy
> >
> > Hi Charles,
> >
> > I've been looking at write speed from fortran, using dummy
> > data, looking at a single array of size 360x180x120.
> >
> > The speed of writing compressed data depends on the choice of
> > chunk sizes, and I'm not sure I've got the optimal values.
> > The tables below list a range of choices (the first line for
> > uncompressed data), which give speeds from 6 to 24 MB/s.
> > These choices also affect the degree of compression and the
> > speed which different subsets of a file can be read, so
> > determining the right values is not a trivial exercise. I
> > have just started exploring the possibilities, and would
> > appreciate any guidance you have,
> >
> > Cheers,
> > Martin
> >
> > File size (MB)				Write speed (MB/s)
> > 31.1	360	180	120	222.5
> > 15.0	8	8	8	7.7
> > 16.8	18	8	4	9.1
> > 20.3	90	4	1	6.0
> > 15.8	90	45	2	17.0
> > 16.6	90	45	1	12.1
> > 16.2	32	32	1	10.5
> > 15.1	32	32	2	14.0
> >
> > And for a larger field, 360x180x2400:
> > 622.09	360	180	2400	512.39
> > 292.16	8	8	8	11.3
> > 309.28	18	8	4	14.78
> > 340.76	90	4	1	8.69
> > 279.29	90	45	2	24
> > 273.24	90	45	1	12.94
> > 269.25	32	32	1	8.1
> > 265.75	32	32	2	11.78
> >
> 
-- 
Scanned by iCritical.