[Go-essp-tech] NetCDF4 compression. write efficiency and CMIP5 policy

[Dean N. Williams]

Yes I know... I just added more (i.e., read speed and file size  
reduction) for others to comment on.

Best regards,
	Dean

On Nov 10, 2009, at 4:55 AM, <stephen.pascoe at stfc.ac.uk> <stephen.pascoe at stfc.ac.uk 
 > wrote:

> We are talking about *write* speeds.
>
> S.
>
> ---
> Stephen Pascoe  +44 (0)1235 445980
> British Atmospheric Data Centre
> Rutherford Appleton Laboratory
>
>
> From: Dean N. Williams [mailto:williams13 at llnl.gov]
> Sent: 10 November 2009 12:51
> To: Pascoe, Stephen (STFC,RAL,SSTD)
> Cc: Juckes, Martin (STFC,RAL,SSTD); Stephens, Ag (STFC,RAL,SSTD); go-essp-tech at ucar.edu 
> ; Charles سمير Doutriaux
> Subject: Re: NetCDF4 compression. write efficiency and CMIP5 policy
>
> This is not what we experienced. In fact, I recall just the  
> opposite. Charles Doutriaux did the work, so I'll let him respond  
> directly. Also it would be good if Ed Hartnett and Russ Rew respond  
> as well to the slowness that you are seeing. They may be able to  
> help you on this.
>
> Charles, if I recall correctly, were the zlib compressed netCDF  
> files read faster in CDAT?
>
> We are using CMOR2, in which CMOR2 does the DRS, netCDF-4 classic  
> compressed output.
>
> Best regards,
> Dean
>
> On Nov 10, 2009, at 4:20 AM, <stephen.pascoe at stfc.ac.uk> wrote:
>
>> Hi Dean
>>
>> In tests we've done at BADC we have experienced 10-20x slowdown in  
>> write speed with NetCDF4 compression.  Is this typical and are  
>> modelling centres aware that they can expect a significant I/O  
>> bottleneck?
>>
>> This makes me think, have we said CMIP5 data *must* be compressed?   
>> Is there a danger we will get a higher volume of data than we  
>> expect because it will be uncompressed to speed up the process at  
>> the modelling centres?  How would we enforce NetCDF compression --  
>> presumably it would be discovered during replication.
>>
>> Cheers,
>> Stephen.
>>
>> ---
>> Stephen Pascoe  +44 (0)1235 445980
>> British Atmospheric Data Centre
>> Rutherford Appleton Laboratory
>>
>>
>> -- 
>> Scanned by iCritical.
>>
>>
>
>
> -- 
> Scanned by iCritical.
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20091110/afe288da/attachment-0001.html