<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=windows-1256">
<META content="MSHTML 6.00.6000.16890" name=GENERATOR></HEAD>
<BODY
style="WORD-WRAP: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space">
<DIV dir=ltr align=left><SPAN class=277005512-10112009><FONT face=Arial
color=#0000ff size=2>We are talking about *write* speeds.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=277005512-10112009><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=277005512-10112009><FONT face=Arial
color=#0000ff size=2>S.</FONT></SPAN></DIV>
<DIV> </DIV>
<DIV align=left><FONT face=Arial size=2>---</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>Stephen Pascoe +44 (0)1235
445980</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>British Atmospheric Data
Centre</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>Rutherford Appleton
Laboratory</FONT></DIV>
<DIV> </DIV><BR>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> Dean N. Williams
[mailto:williams13@llnl.gov] <BR><B>Sent:</B> 10 November 2009
12:51<BR><B>To:</B> Pascoe, Stephen (STFC,RAL,SSTD)<BR><B>Cc:</B> Juckes, Martin
(STFC,RAL,SSTD); Stephens, Ag (STFC,RAL,SSTD); go-essp-tech@ucar.edu; Charles
سمير Doutriaux<BR><B>Subject:</B> Re: NetCDF4 compression. write efficiency and
CMIP5 policy<BR></FONT><BR></DIV>
<DIV></DIV>This is not what we experienced. In fact, I recall just the opposite.
Charles Doutriaux did the work, so I'll let him respond directly. Also it would
be good if Ed Hartnett and Russ Rew respond as well to the slowness that you are
seeing. They may be able to help you on this.
<DIV><BR></DIV>
<DIV>Charles, if I recall correctly, were the zlib compressed netCDF files read
faster in CDAT?<BR>
<DIV><BR></DIV>
<DIV>We are using CMOR2, in which CMOR2 does the DRS, netCDF-4 classic
compressed output.</DIV>
<DIV><BR></DIV>
<DIV>Best regards,</DIV>
<DIV><SPAN class=Apple-tab-span style="WHITE-SPACE: pre"></SPAN>Dean</DIV>
<DIV><BR>
<DIV>
<DIV>On Nov 10, 2009, at 4:20 AM, <<A
href="mailto:stephen.pascoe@stfc.ac.uk">stephen.pascoe@stfc.ac.uk</A>>
wrote:</DIV><BR class=Apple-interchange-newline>
<BLOCKQUOTE type="cite">
<DIV>
<DIV><FONT face=Arial size=2><SPAN class=884391312-10112009>Hi
Dean</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=884391312-10112009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=884391312-10112009>In tests we've
done at BADC we have experienced 10-20x slowdown in write speed with NetCDF4
compression. Is this typical and are modelling centres aware that they
can expect a significant I/O bottleneck?</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=884391312-10112009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=884391312-10112009>This makes me
think, have we said CMIP5 data *must* be compressed? Is there a danger
we will get a higher volume of data than we expect because it will be
uncompressed to speed up the process at the modelling centres? How would
we enforce NetCDF compression -- presumably it would be discovered during
replication.</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=884391312-10112009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN
class=884391312-10112009>Cheers,</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=884391312-10112009>Stephen.</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV align=left><FONT face=Arial size=2>---</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>Stephen Pascoe +44 (0)1235
445980</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>British Atmospheric Data
Centre</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>Rutherford Appleton
Laboratory</FONT></DIV>
<DIV> </DIV><BR>
<P>-- <BR>Scanned by iCritical.
</P><BR></DIV></BLOCKQUOTE></DIV><BR></DIV></DIV>
<br><p>--
<BR>Scanned by iCritical.
</p>
<br></BODY></HTML>