[Go-essp-tech] CMOR2 compress?
martin.juckes at stfc.ac.uk
martin.juckes at stfc.ac.uk
Mon Jan 11 03:50:27 MST 2010
Hello Karl, Joerg,
I did some tests and found that reading compressed netcdf 4 is slower than reading uncompressed netcdf 4, bur faster than reading netcdf 3. For reading a single latitude longitude field out of a 1Gb file, compression only caused a 10% slow down (though this depends on how the "chunking" parameters are set). When reading the whole file, compression made it more than 4 times slower, but still faster than netcdf 3.
My feeling is that we should stick with compression: if users expect to be reading files repeatedly they can decompress it themselves.
Cheers,
Martin
> -----Original Message-----
> From: Jörg Wegner [mailto:joerg.wegner at zmaw.de]
> Sent: 11 January 2010 10:35
> To: Karl Taylor
> Cc: Juckes, Martin (STFC,RAL,SSTD)
> Subject: CMOR2 compress?
>
> Dear Karl and all,
>
> my experiences with compressed netcdf4 files are not that bad for
> atmosphere variables. I've to admit that for some variables (like
> precipitation) the compression is only 20% but in general ( and for
> all 3d variables) the compression is better then 35% and up to 60%.
> To select 3 years with cdos from an uncompressed 600MByte 40years 3d
> variable file our IBM needs 2.027s, the compressed file is 333MByte
> big and the selection takes 2.654s which is about 30% slower.
> So for the time being we would prefer a compression on all our files.
>
> regards,
> Joerg
--
Scanned by iCritical.
More information about the GO-ESSP-TECH
mailing list