<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=windows-1256">
<META content="MSHTML 6.00.6000.16890" name=GENERATOR></HEAD>
<BODY
style="WORD-WRAP: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space">
<DIV dir=ltr align=left><SPAN class=572313315-10112009><FONT face=Arial
color=#0000ff size=2>Thanks Charles,</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=572313315-10112009><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=572313315-10112009><FONT face=Arial
color=#0000ff size=2>Martin did the tests our end so it would be useful to
compare with him how you are testing. Martin is at conference until
Thursday so he may not respond immediately.</FONT></SPAN></DIV>
<DIV><FONT face=Arial color=#0000ff size=2></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=572313315-10112009>Cheers,</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=572313315-10112009>Stephen.</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2></FONT> </DIV>
<DIV align=left><FONT face=Arial size=2>---</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>Stephen Pascoe +44 (0)1235
445980</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>British Atmospheric Data
Centre</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>Rutherford Appleton
Laboratory</FONT></DIV>
<DIV> </DIV><BR>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> Charles سمير Doutriaux
[mailto:doutriaux1@llnl.gov] <BR><B>Sent:</B> 10 November 2009
15:01<BR><B>To:</B> Dean N. Williams<BR><B>Cc:</B> Pascoe, Stephen
(STFC,RAL,SSTD); Juckes, Martin (STFC,RAL,SSTD); Stephens, Ag (STFC,RAL,SSTD);
go-essp-tech@ucar.edu<BR><B>Subject:</B> Re: NetCDF4 compression. write
efficiency and CMIP5 policy<BR></FONT><BR></DIV>
<DIV></DIV>Yes I did some testing on the speed and it was much faster to write
compressed, which did make sense to me since there was so much less data to
write, and cpu time nothing next to i/o time.
<DIV><BR></DIV>
<DIV>I'll rerun the test again today.</DIV>
<DIV><BR></DIV>
<DIV>C</DIV>
<DIV><BR>
<DIV>
<DIV>On Nov 10, 2009, at 4:50 AM, Dean N. Williams wrote:</DIV><BR
class=Apple-interchange-newline>
<BLOCKQUOTE type="cite">
<DIV
style="WORD-WRAP: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space">This
is not what we experienced. In fact, I recall just the opposite. Charles
Doutriaux did the work, so I'll let him respond directly. Also it would be
good if Ed Hartnett and Russ Rew respond as well to the slowness that you are
seeing. They may be able to help you on this.
<DIV><BR></DIV>
<DIV>Charles, if I recall correctly, were the zlib compressed netCDF files
read faster in CDAT?<BR>
<DIV><BR></DIV>
<DIV>We are using CMOR2, in which CMOR2 does the DRS, netCDF-4 classic
compressed output.</DIV>
<DIV><BR></DIV>
<DIV>Best regards,</DIV>
<DIV><SPAN class=Apple-tab-span style="WHITE-SPACE: pre"></SPAN>Dean</DIV>
<DIV><BR>
<DIV>
<DIV>On Nov 10, 2009, at 4:20 AM, <<A
href="mailto:stephen.pascoe@stfc.ac.uk">stephen.pascoe@stfc.ac.uk</A>>
wrote:</DIV><BR class=Apple-interchange-newline>
<BLOCKQUOTE type="cite">
<DIV>
<DIV><FONT face=Arial size=2><SPAN class=884391312-10112009>Hi
Dean</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=884391312-10112009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=884391312-10112009>In tests we've
done at BADC we have experienced 10-20x slowdown in write speed with NetCDF4
compression. Is this typical and are modelling centres aware that they
can expect a significant I/O bottleneck?</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=884391312-10112009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=884391312-10112009>This makes me
think, have we said CMIP5 data *must* be compressed? Is there a danger
we will get a higher volume of data than we expect because it will be
uncompressed to speed up the process at the modelling centres? How
would we enforce NetCDF compression -- presumably it would be discovered
during replication.</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=884391312-10112009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN
class=884391312-10112009>Cheers,</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=884391312-10112009>Stephen.</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV align=left><FONT face=Arial size=2>---</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>Stephen Pascoe +44 (0)1235
445980</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>British Atmospheric Data
Centre</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>Rutherford Appleton
Laboratory</FONT></DIV>
<DIV> </DIV><BR>
<P>-- <BR>Scanned by iCritical.
</P><BR></DIV></BLOCKQUOTE></DIV><BR></DIV></DIV></DIV></BLOCKQUOTE></DIV><BR></DIV>
<br><p>--
<BR>Scanned by iCritical.
</p>
<br></BODY></HTML>