<div dir="ltr"><span style="font-size:14px;font-family:arial,sans-serif">Ruping,</span><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">I have experienced similar slow downs </font><span style="font-size:14px;font-family:arial,sans-serif">when working with large data sets. I</span><font face="arial, sans-serif">t is an important clue that </font><span style="font-size:14px;font-family:arial,sans-serif">the s</span><span style="font-size:14px;font-family:arial,sans-serif">econd run is fast up to the previous point of termination, then slow again. This suggests system file caches, somewhere, that are probably not part of NCL. This is similar to what Kyle suggested, but also consider network behavior, if that applies to you.</span></div><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">We have slow file servers on my work group network that behave like this, especially when loaded by multiple processes. The behavior is not limited to NCL. </font><font face="arial, sans-serif">You might check</font><span style="font-size:14px;font-family:arial,sans-serif"> with your system administrators about file servers and cache behavior, and whether there are adjustments that might improve performance.</span><br></div><div><span style="font-size:14px;font-family:arial,sans-serif"><br></span></div><div><span style="font-size:14px;font-family:arial,sans-serif">You might also be able to use performance monitoring tools on your own computer and on network files servers (with admin assistance, I suppose), to try to locate where the slow downs are coming from.</span></div><div><span style="font-size:14px;font-family:arial,sans-serif"><br></span></div><div><span style="font-size:14px;font-family:arial,sans-serif">In the meantime, if system caches really are the problem, you might just have to live with the slow behavior for data sets like this one. </span><span style="font-size:14px;font-family:arial,sans-serif">Sorry I </span><span style="font-size:14px;font-family:arial,sans-serif">do not have any more specific advice.</span></div><div style="font-size:14px;font-family:arial,sans-serif"><br></div><div style="font-size:14px;font-family:arial,sans-serif">--Dave</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 12, 2014 at 4:19 PM, <span dir="ltr"><<a href="mailto:rupingmo@gmail.com" target="_blank">rupingmo@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u><div>Hi Jon,<br><br>I do have the latest version of NCL (v6.2.1). The code is running fast to the point where the previous run was terminated, then begins to slow down. The delete function (in my case, delete(dat)) at the end of each iteration doesn't seem to help.<br><br>Ruping<div>Sent from my BlackBerry device on the Rogers Wireless Network</div><hr><div><b>From: </b> Jon Meyer <<a href="mailto:jonathan.meyer@aggiemail.usu.edu" target="_blank">jonathan.meyer@aggiemail.usu.edu</a>>
</div><div><b>Date: </b>Fri, 12 Sep 2014 16:03:19 -0600</div><div><b>To: </b>Ruping Mo<<a href="mailto:rupingmo@gmail.com" target="_blank">rupingmo@gmail.com</a>>; <a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a><<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a>></div><div><b>Subject: </b>Re: [ncl-talk] The slow down when addfile is used in do loop</div><div><br></div><div dir="ltr">What version of NCL are you using? Version 6.2 fixed a minor memory leak associated when .grb files are opened. When numerous .grb files are opened sequentially, the small memory leak can cause your memory footprint to grow with time...even if you are using the 'delete' function after each loop iteration.<div><br></div><div>If your code is running fast at first, using the delete function at the end of each iteration will likely solve your issue.<br><div><br></div><div>Jon</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 12, 2014 at 2:31 PM, Ruping Mo <span dir="ltr"><<a href="mailto:rupingmo@gmail.com" target="_blank">rupingmo@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi NCLers,</div><div><br></div><div>I have an archive of large grib files. Each grib file contains a month of multiple daily global variables. I use the following script to extract a small part of one variable. The script, which uses addfile in a large loop, runs very fast for the first few months, then begins to slow down substantially. I let it run for a few years, then terminate the job (Ctrl-Z) and re-run the same script. Surprisingly, the new job runs very fast to the month when the previous job was terminated. I wonder why it behaves this way, and what I can do to avoid the slow down problem.</div><div><br></div><div>Have a nice day!</div><div><br></div><div>Ruping</div><div><br></div><div>The script:</div><div>;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;</div><div>begin</div><div> nday = 12784 ; number of days from Jan 1979 to Dec 2013</div><div> temp = new((/nday, 9, 9/), float)</div><div>;;;;</div><div> t0 = 0</div><div> do yr = 1979, 2013</div><div> syr = tostring(yr)</div><div> do mo = 1, 12</div><div> smo = tostring(mo)</div><div> print(syr + "-" + smo)</div><div> fname = syr + "/" + syr + "-" + smo + ".grib"</div><div> dat = addfile(fname, "r")</div><div> mday = getfilevardimsizes(dat, "initial_time0") - 1</div><div> i1 = i0 + mday</div><div> temp(i0:i1, :, :) = (/ dat->2T_GDS0_SFC(:, 80:88, 146:154) /)</div><div>;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; 2T_GDS0_SFC(mday, 241, 480)</div><div> i0 = i1 + 1</div><div> end do</div><div> end do</div><div>end</div><div>;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;</div></div>
<br>_______________________________________________<br>
ncl-talk mailing list<br>
List instructions, subscriber options, unsubscribe:<br>
<a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" target="_blank">http://mailman.ucar.edu/mailman/listinfo/ncl-talk</a><br>
<br></blockquote></div><br></div>
</div><br>_______________________________________________<br>
ncl-talk mailing list<br>
List instructions, subscriber options, unsubscribe:<br>
<a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" target="_blank">http://mailman.ucar.edu/mailman/listinfo/ncl-talk</a><br>
<br></blockquote></div><br></div>