[Go-essp-tech] NetCDF4 compression. write efficiency and CMIP5 policy
Dean N. Williams
williams13 at llnl.gov
Tue Nov 10 07:56:37 MST 2009
Hi Stephen,
I was looking at the CMIP website and we should be a little more
specific about what the required output (http://cmip-pcmdi.llnl.gov/cmip5/output_req.html?submenuheader=2%23cmor
).
Best regards,
Dean
On Nov 10, 2009, at 4:55 AM, <stephen.pascoe at stfc.ac.uk> <stephen.pascoe at stfc.ac.uk
> wrote:
> We are talking about *write* speeds.
>
> S.
>
> ---
> Stephen Pascoe +44 (0)1235 445980
> British Atmospheric Data Centre
> Rutherford Appleton Laboratory
>
>
> From: Dean N. Williams [mailto:williams13 at llnl.gov]
> Sent: 10 November 2009 12:51
> To: Pascoe, Stephen (STFC,RAL,SSTD)
> Cc: Juckes, Martin (STFC,RAL,SSTD); Stephens, Ag (STFC,RAL,SSTD); go-essp-tech at ucar.edu
> ; Charles سمير Doutriaux
> Subject: Re: NetCDF4 compression. write efficiency and CMIP5 policy
>
> This is not what we experienced. In fact, I recall just the
> opposite. Charles Doutriaux did the work, so I'll let him respond
> directly. Also it would be good if Ed Hartnett and Russ Rew respond
> as well to the slowness that you are seeing. They may be able to
> help you on this.
>
> Charles, if I recall correctly, were the zlib compressed netCDF
> files read faster in CDAT?
>
> We are using CMOR2, in which CMOR2 does the DRS, netCDF-4 classic
> compressed output.
>
> Best regards,
> Dean
>
> On Nov 10, 2009, at 4:20 AM, <stephen.pascoe at stfc.ac.uk> wrote:
>
>> Hi Dean
>>
>> In tests we've done at BADC we have experienced 10-20x slowdown in
>> write speed with NetCDF4 compression. Is this typical and are
>> modelling centres aware that they can expect a significant I/O
>> bottleneck?
>>
>> This makes me think, have we said CMIP5 data *must* be compressed?
>> Is there a danger we will get a higher volume of data than we
>> expect because it will be uncompressed to speed up the process at
>> the modelling centres? How would we enforce NetCDF compression --
>> presumably it would be discovered during replication.
>>
>> Cheers,
>> Stephen.
>>
>> ---
>> Stephen Pascoe +44 (0)1235 445980
>> British Atmospheric Data Centre
>> Rutherford Appleton Laboratory
>>
>>
>> --
>> Scanned by iCritical.
>>
>>
>
>
> --
> Scanned by iCritical.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20091110/e50a884d/attachment.html
More information about the GO-ESSP-TECH
mailing list