<div dir="ltr"><div class="gmail_default" style="font-size:small">Dear Paul,</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">In order to see if there's a memory leak in NCL with regard to your code, we would need access to your data files, which I'm guessing might be a problem if you have a lot of them. How many files do you have and how big is each one? Also, you are running an older version of NCL. You might want to consider upgrading to NCL V6.5.0. You could use conda to install this version in a separate environment, while still keeping your V6.3.0 version for test purposes. The conda instructions can be found <a href="http://www.ncl.ucar.edu/Download/conda.shtml">here</a>.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">There might be some things you can do to make your code run more efficiently. </div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">[1] Creating strings in NCL can be expensive. Every time you do something like this:</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"> indat = readAsciiTable(infile(ifile),</span><wbr style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">7,"float",1)</span></font></div><div class="gmail_default" style="font-size:small"><div class="gmail_default" style="text-decoration-style:initial;text-decoration-color:initial"><br class="gmail-Apple-interchange-newline">I believe it causes a little bit of new memory to be allocated and it doesn't get freed. I would suggest creating a variable outside both do loops to hold the type:</div><div class="gmail_default" style="text-decoration-style:initial;text-decoration-color:initial"><br></div><div class="gmail_default" style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">indat_type = "float" </font></div><div class="gmail_default" style="text-decoration-style:initial;text-decoration-color:initial"><br></div><div class="gmail_default" style="text-decoration-style:initial;text-decoration-color:initial">and then use it in the read call:</div><div class="gmail_default" style="text-decoration-style:initial;text-decoration-color:initial"><br></div><div class="gmail_default" style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace"><span style="text-decoration-style:initial;text-decoration-color:initial;font-size:12.8px;background-color:rgb(255,255,255);float:none;display:inline"> indat = readAsciiTable(infile(ifile),</span><wbr style="text-decoration-style:initial;text-decoration-color:initial;font-size:12.8px;background-color:rgb(255,255,255)"><span style="text-decoration-style:initial;text-decoration-color:initial;font-size:12.8px;background-color:rgb(255,255,255);float:none;display:inline">7,indat_type,1)</span></font></div><br></div><div class="gmail_default" style="font-size:small">[2] The inner do loop may not be needed, *if* the size of the arrays don't change with every iteration of reading the data files. The code could be trimmed to something like this (UNTESTED):</div><div class="gmail_default"><br></div><div class="gmail_default"><div class="gmail_default"><font face="monospace, monospace">indat_type = "float"</font></div><div class="gmail_default"><font face="monospace, monospace">dims = (/naz,nr/)</font></div><div class="gmail_default"><font face="monospace, monospace">stats =<span style="white-space:pre"> </span>new((/nfiles,naz,nr/),float)</font></div><div class="gmail_default"><font face="monospace, monospace">do ifile = 0,nfiles-1</font></div><div class="gmail_default"><font face="monospace, monospace"> indat = readAsciiTable(infile(ifile),7,indat_type,1)</font></div><div class="gmail_default"><font face="monospace, monospace"> field = indat(:,5)</font></div><div class="gmail_default"><font face="monospace, monospace"> numrows = dimsizes(field)</font></div><div class="gmail_default"><font face="monospace, monospace"> iaz = toint(floor(indat(:,2)/0.5))</font></div><div class="gmail_default"><font face="monospace, monospace"> ir = toint(floor(indat(:,3)/gate))</font></div><div class="gmail_default"><font face="monospace, monospace"> indexes = (iaz*(nr-1)) + ir</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace"> maxlat = reshape(indat(indexes,0),dims)</font></div><div class="gmail_default"><font face="monospace, monospace"> maxlon = reshape(indat(indexes,1),dims)</font></div><div class="gmail_default"><font face="monospace, monospace"> stats(ifile,:,:) = reshape(abs(field(indexes)),dims)</font></div><div class="gmail_default"><font face="monospace, monospace"> . . .</font></div><div class="gmail_default"><font face="monospace, monospace">end do</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div></div><div class="gmail_default" style="font-size:small">What I'm doing is calculating the 1D indexes necessary to populate maxlat, maxlon, and stats, and then reshaping this 1D array to a 2D array. </div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">However, If the number of rows in the data files is changing with each iteration, then you will need to rethink this a little. It would have to look something like this (UNTESTED):</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default"><div class="gmail_default"><br></div><div class="gmail_default"><font face="monospace, monospace">dims = (/naz,nr/)</font></div><div class="gmail_default"><font face="monospace, monospace">dims_1d = product(dims)</font></div><div class="gmail_default"><font face="monospace, monospace">stats_2d = new((/nfiles,dims_1d/),float)</font></div><div class="gmail_default"><font face="monospace, monospace">maxlat_1d = new(dims_1d,float)</font></div><div class="gmail_default"><font face="monospace, monospace">maxlon_1d = new(dims_1d,float)</font></div><div class="gmail_default"><font face="monospace, monospace">do ifile = 0,nfiles-1</font></div><div class="gmail_default"><font face="monospace, monospace"> indat = readAsciiTable(infile(ifile),7,indat_type,1)</font></div><div class="gmail_default"><font face="monospace, monospace"> field = indat(:,5)</font></div><div class="gmail_default"><font face="monospace, monospace"> iaz = toint(floor(indat(:,2)/0.5))</font></div><div class="gmail_default"><font face="monospace, monospace"> ir = toint(floor(indat(:,3)/gate))</font></div><div class="gmail_default"><font face="monospace, monospace"> indexes = (iaz*(nr-1)) + ir</font></div><div class="gmail_default"><font face="monospace, monospace"><br></font></div><div class="gmail_default"><font face="monospace, monospace"> maxlat_1d(indexes) = indat(:,0)</font></div><div class="gmail_default"><font face="monospace, monospace"> maxlon_1d<span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">(indexes)</span> = indat(:,1)</font></div><div class="gmail_default"><font face="monospace, monospace"> stats_2d(ifile,indexes) = abs(field)</font></div><div class="gmail_default"><font face="monospace, monospace"> . . .</font></div><div class="gmail_default"><font face="monospace, monospace">end do</font></div><div class="gmail_default"><font face="monospace, monospace">maxlat = reshape(maxlat_1d,dims)</font></div><div class="gmail_default"><font face="monospace, monospace">maxlon = reshape(maxlon_1d,dims)</font></div><div class="gmail_default"><font face="monospace, monospace">stats = reshape(stats_2d,(/nfiles,naz,nr/))</font></div><div class="gmail_default"><font face="monospace, monospace">delete([/maxlat_1d,maxlon_1d,stats_2d/])</font></div></div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">[3] This is just a small tip that won't help much with memory, but in a "new" statement, you don't need to enclose the type in quotes:</div><div class="gmail_default"><br></div><div class="gmail_default"><font face="monospace, monospace"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">stats = new((/nfiles,naz,nr/),float)</span><br style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px">. . .</span><br style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">maxfield = new((/naz,nr/),float)</span><br style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px">. . .</span><br style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">maxlat = new((/naz,nr/),float)</span><br style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">. . .</span></font><br></div><div class="gmail_default"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div class="gmail_default"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">--Mary</span></div><div class="gmail_default"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div class="gmail_default" style="font-size:small"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 28, 2018 at 7:53 AM, robinsonp <a href="http://cswr.org">cswr.org</a> <span dir="ltr"><<a href="mailto:robinsonp@cswr.org" target="_blank">robinsonp@cswr.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>
<div><p>Hi all,<br></p><p><br></p><p>I am trying to read a large number of files, using the data to fill an array which has been declared with a new statement. After each file is read, all arrays no longer needed are deleted prior to reading the next file. Yet after 80 files are read, I get an NhlMalloc Failed error. "fatal:NhlMalloc Failed:[errno=12]". If I read all the files in smaller groups with individual runs of the program, I get no errors, which suggests the files them selves are not to blame. It SEEMS there is a memory leak. I have tried many things and read this forum on the topic. </p><p><br></p><p>My ncl version for RedHat is ncl_ncarg-6.3.0.Linux_RHEL5.<wbr>11_i686_nodap_gcc412.tar.gz.<br></p><p><br></p><p>Code snippet of loop within which error occurs:<br></p><p>______________________________<wbr>_____________________________</p><p> naz = 720<br> nr = 976</p><p class="m_3303442998752577528default-style">infile = systemfunc("ls rawdats/rawdat0*")</p><p class="m_3303442998752577528default-style">nfiles = dimsizes(infile)</p><p class="m_3303442998752577528default-style">gate = 50.0 ; from swp metadata</p><p class="m_3303442998752577528default-style">stats = new((/nfiles,naz,nr/),"float")<br> stats@_FillValue = bad<br> stats = stats@_FillValue<br> maxfield = new((/naz,nr/),"float")<br> maxfield@_FillValue = bad<br> maxfield = 0.0<br> maxlat = new((/naz,nr/),"float")<br> maxlon = new((/naz,nr/),"float")</p><p class="m_3303442998752577528default-style">do ifile = 0,nfiles-1<br> indat = readAsciiTable(infile(ifile),<wbr>7,"float",1)<br> field = indat(:,5)<br> lat1d = indat(:,0)<br> lon1d = indat(:,1)</p><p class="m_3303442998752577528default-style"> numrows = dimsizes(field)<br> az = indat(:,2)<br> rn = indat(:,3)</p><p class="m_3303442998752577528default-style"> do i = 0,numrows-1<br> ; Determine indeces for az,range and load into stats array<br> iaz = toint(floor(az(i)/0.5))<br> ir = toint(floor(rn(i)/gate))</p><p class="m_3303442998752577528default-style"> ; Add this data to stats array<br> stats(ifile,iaz,ir) = abs(field(i))</p><p class="m_3303442998752577528default-style"> maxlat(iaz,ir) = lat1d(i)<br> maxlon(iaz,ir) = lon1d(i)</p><p class="m_3303442998752577528default-style"> end do</p><p class="m_3303442998752577528default-style"> delete(indat)<br> delete(numrows)<br> delete(az)<br> delete(rn)<br> delete(field)<br> delete(lat1d)<br> delete(lon1d)<br> end do<br></p><p class="m_3303442998752577528default-style">______________________________<wbr>______________________________<wbr>___<br></p><p class="m_3303442998752577528default-style">Any ideas?<br></p><p class="m_3303442998752577528default-style"><br></p><p class="m_3303442998752577528default-style">Paul Robinson<br></p><p class="m_3303442998752577528default-style">Center for Severe Weather Research in Boulder, CO<br></p></div>
<br>______________________________<wbr>_________________<br>
ncl-talk mailing list<br>
<a href="mailto:ncl-talk@ucar.edu">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br>
<a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" rel="noreferrer" target="_blank">http://mailman.ucar.edu/<wbr>mailman/listinfo/ncl-talk</a><br>
<br></blockquote></div><br></div>