<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi Stephen,<div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>I was looking at the CMIP website and we should be a little more specific about what the required output (<a href="http://cmip-pcmdi.llnl.gov/cmip5/output_req.html?submenuheader=2%23cmor">http://cmip-pcmdi.llnl.gov/cmip5/output_req.html?submenuheader=2%23cmor</a>).</div><div><br></div><div>Best regards,</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Dean</div><div><br><div><div>On Nov 10, 2009, at 4:55 AM, <<a href="mailto:stephen.pascoe@stfc.ac.uk">stephen.pascoe@stfc.ac.uk</a>> <<a href="mailto:stephen.pascoe@stfc.ac.uk">stephen.pascoe@stfc.ac.uk</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"> <div style="WORD-WRAP: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space"> <div dir="ltr" align="left"><span class="277005512-10112009"><font face="Arial" color="#0000ff" size="2">We are talking about *write* speeds.</font></span></div> <div dir="ltr" align="left"><span class="277005512-10112009"><font face="Arial" color="#0000ff" size="2"></font></span> </div> <div dir="ltr" align="left"><span class="277005512-10112009"><font face="Arial" color="#0000ff" size="2">S.</font></span></div> <div> </div> <div align="left"><font face="Arial" size="2">---</font></div> <div align="left"><font face="Arial" size="2">Stephen Pascoe +44 (0)1235 445980</font></div> <div align="left"><font face="Arial" size="2">British Atmospheric Data Centre</font></div> <div align="left"><font face="Arial" size="2">Rutherford Appleton Laboratory</font></div> <div> </div><br> <div class="OutlookMessageHeader" lang="en-us" dir="ltr" align="left"> <hr tabindex="-1"> <font face="Tahoma" size="2"><b>From:</b> Dean N. Williams [<a href="mailto:williams13@llnl.gov">mailto:williams13@llnl.gov</a>] <br><b>Sent:</b> 10 November 2009 12:51<br><b>To:</b> Pascoe, Stephen (STFC,RAL,SSTD)<br><b>Cc:</b> Juckes, Martin (STFC,RAL,SSTD); Stephens, Ag (STFC,RAL,SSTD); <a href="mailto:go-essp-tech@ucar.edu">go-essp-tech@ucar.edu</a>; Charles سمير Doutriaux<br><b>Subject:</b> Re: NetCDF4 compression. write efficiency and CMIP5 policy<br></font><br></div> <div></div>This is not what we experienced. In fact, I recall just the opposite. Charles Doutriaux did the work, so I'll let him respond directly. Also it would be good if Ed Hartnett and Russ Rew respond as well to the slowness that you are seeing. They may be able to help you on this. <div><br></div> <div>Charles, if I recall correctly, were the zlib compressed netCDF files read faster in CDAT?<br> <div><br></div> <div>We are using CMOR2, in which CMOR2 does the DRS, netCDF-4 classic compressed output.</div> <div><br></div> <div>Best regards,</div> <div><span class="Apple-tab-span" style="WHITE-SPACE: pre"></span>Dean</div> <div><br> <div> <div>On Nov 10, 2009, at 4:20 AM, <<a href="mailto:stephen.pascoe@stfc.ac.uk">stephen.pascoe@stfc.ac.uk</a>> wrote:</div><br class="Apple-interchange-newline"> <blockquote type="cite"> <div> <div><font face="Arial" size="2"><span class="884391312-10112009">Hi Dean</span></font></div> <div><font face="Arial" size="2"><span class="884391312-10112009"></span></font> </div> <div><font face="Arial" size="2"><span class="884391312-10112009">In tests we've done at BADC we have experienced 10-20x slowdown in write speed with NetCDF4 compression. Is this typical and are modelling centres aware that they can expect a significant I/O bottleneck?</span></font></div> <div><font face="Arial" size="2"><span class="884391312-10112009"></span></font> </div> <div><font face="Arial" size="2"><span class="884391312-10112009">This makes me think, have we said CMIP5 data *must* be compressed? Is there a danger we will get a higher volume of data than we expect because it will be uncompressed to speed up the process at the modelling centres? How would we enforce NetCDF compression -- presumably it would be discovered during replication.</span></font></div> <div><font face="Arial" size="2"><span class="884391312-10112009"></span></font> </div> <div><font face="Arial" size="2"><span class="884391312-10112009">Cheers,</span></font></div> <div><font face="Arial" size="2"><span class="884391312-10112009">Stephen.</span></font></div> <div><font face="Arial" size="2"></font> </div> <div align="left"><font face="Arial" size="2">---</font></div> <div align="left"><font face="Arial" size="2">Stephen Pascoe +44 (0)1235 445980</font></div> <div align="left"><font face="Arial" size="2">British Atmospheric Data Centre</font></div> <div align="left"><font face="Arial" size="2">Rutherford Appleton Laboratory</font></div> <div> </div><br><p>-- <br>Scanned by iCritical. </p><br></div></blockquote></div><br></div></div> <br><p>-- <br>Scanned by iCritical. </p> <br></div></blockquote></div><br></div></body></html>