[Go-essp-tech] CMOR2 compress?

[martin.juckes at stfc.ac.uk]

Hello Karl, Joerg,

I did some tests and found that reading compressed netcdf 4 is slower than reading uncompressed netcdf 4, bur faster than reading netcdf 3. For reading a single latitude longitude field out of a 1Gb file, compression only caused a 10% slow down (though this depends on how the "chunking" parameters are set). When reading the whole file, compression made it more than 4 times slower, but still faster than netcdf 3. 

My feeling is that we should stick with compression: if users expect to be reading files repeatedly they can decompress it themselves. 

Cheers,
Martin

> -----Original Message-----
> From: Jörg Wegner [mailto:joerg.wegner at zmaw.de]
> Sent: 11 January 2010 10:35
> To: Karl Taylor
> Cc: Juckes, Martin (STFC,RAL,SSTD)
> Subject: CMOR2 compress?
> 
> Dear Karl and all,
> 
> my experiences with compressed netcdf4 files are not that bad for
> atmosphere variables. I've to admit that for some variables (like
> precipitation) the compression is only  20% but in general ( and for
> all 3d variables) the compression is better then 35% and up to 60%.
> To select 3 years with cdos from an uncompressed 600MByte 40years 3d
> variable file our IBM needs 2.027s, the compressed file is 333MByte
> big and the selection takes 2.654s which is about 30% slower.
> So for the time being we would prefer a compression on all our files.
> 
> regards,
> Joerg

-- 
Scanned by iCritical.