[Go-essp-tech] NetCDF4 compression. write efficiency and CMIP5policy
martin.juckes at stfc.ac.uk
martin.juckes at stfc.ac.uk
Thu Nov 12 08:26:42 MST 2009
Hello Phil,
Not too hard -- just compiling the code against the earlier library and
deleting a few calls.
For the large file (380x180x2400), I get a write of 3.5 seconds with
netcdf 3 compared with 1.2 seconds for netcdf 4 uncompressed and 27 to
60 seconds for netcdf 4 compressed.
So there is an impressive speed-up between netcdf 3 and netcdf 4.
Cheers,
Martin
> -----Original Message-----
> From: Bentley, Philip [mailto:philip.bentley at metoffice.gov.uk]
> Sent: 12 November 2009 13:57
> To: Juckes, Martin (STFC,RAL,SSTD)
> Cc: go-essp-tech at ucar.edu
> Subject: RE: [Go-essp-tech] NetCDF4 compression. write efficiency and
> CMIP5policy
>
> Hi Martin,
>
> Purely out of curiosity :-) it would be interesting to know what the
> comparable figure is for writing vanilla netcdf-3 classic output
> (uncompressed only, of course). Would it be easy for you to produce
> that
> figure for the same test file?
>
> Regards,
> Phil
>
> > -----Original Message-----
> > From: go-essp-tech-bounces at ucar.edu
> > [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of
> > martin.juckes at stfc.ac.uk
> > Sent: 12 November 2009 11:47
> > To: doutriaux1 at llnl.gov; williams13 at llnl.gov
> > Cc: go-essp-tech at ucar.edu
> > Subject: Re: [Go-essp-tech] NetCDF4 compression. write
> > efficiency and CMIP5policy
> >
> > Hi Charles,
> >
> > I've been looking at write speed from fortran, using dummy
> > data, looking at a single array of size 360x180x120.
> >
> > The speed of writing compressed data depends on the choice of
> > chunk sizes, and I'm not sure I've got the optimal values.
> > The tables below list a range of choices (the first line for
> > uncompressed data), which give speeds from 6 to 24 MB/s.
> > These choices also affect the degree of compression and the
> > speed which different subsets of a file can be read, so
> > determining the right values is not a trivial exercise. I
> > have just started exploring the possibilities, and would
> > appreciate any guidance you have,
> >
> > Cheers,
> > Martin
> >
> > File size (MB) Write speed (MB/s)
> > 31.1 360 180 120 222.5
> > 15.0 8 8 8 7.7
> > 16.8 18 8 4 9.1
> > 20.3 90 4 1 6.0
> > 15.8 90 45 2 17.0
> > 16.6 90 45 1 12.1
> > 16.2 32 32 1 10.5
> > 15.1 32 32 2 14.0
> >
> > And for a larger field, 360x180x2400:
> > 622.09 360 180 2400 512.39
> > 292.16 8 8 8 11.3
> > 309.28 18 8 4 14.78
> > 340.76 90 4 1 8.69
> > 279.29 90 45 2 24
> > 273.24 90 45 1 12.94
> > 269.25 32 32 1 8.1
> > 265.75 32 32 2 11.78
> >
>
--
Scanned by iCritical.
More information about the GO-ESSP-TECH
mailing list