[ncl-talk] NhlMalloc error despite using delete

[Mary Haley]

Dear Paul,

In order to see if there's a memory leak in NCL with regard to your code,
we would need access to your data files, which I'm guessing might be a
problem if you have a lot of them. How many files do you have and how big
is each one? Also, you are running an older version of NCL. You might want
to consider upgrading to NCL V6.5.0. You could use conda to install this
version in a separate environment, while still keeping your V6.3.0 version
for test purposes. The conda instructions can be found here
<http://www.ncl.ucar.edu/Download/conda.shtml>.

There might be some things you can do to make your code run more
efficiently.

[1] Creating strings in NCL can be expensive. Every time you do something
like this:

   indat = readAsciiTable(infile(ifile),7,"float",1)

I believe it causes a little bit of new memory to be allocated and it
doesn't get freed. I would suggest creating a variable outside both do
loops to hold the type:

indat_type = "float"

and then use it in the read call:

   indat = readAsciiTable(infile(ifile),7,indat_type,1)

[2] The inner do loop may not be needed, *if* the size of the arrays don't
change with every iteration of reading the data files.  The code could be
trimmed to something like this (UNTESTED):

indat_type = "float"
dims = (/naz,nr/)
stats = new((/nfiles,naz,nr/),float)
do ifile = 0,nfiles-1
   indat = readAsciiTable(infile(ifile),7,indat_type,1)
   field = indat(:,5)
   numrows = dimsizes(field)
   iaz     = toint(floor(indat(:,2)/0.5))
   ir      = toint(floor(indat(:,3)/gate))
   indexes = (iaz*(nr-1)) + ir

   maxlat           = reshape(indat(indexes,0),dims)
   maxlon           = reshape(indat(indexes,1),dims)
   stats(ifile,:,:) = reshape(abs(field(indexes)),dims)
   . . .
end do

What I'm doing is calculating the 1D indexes necessary to populate maxlat,
maxlon, and stats, and then reshaping this 1D array to a 2D array.

However, If the number of rows in the data files is changing with each
iteration, then you will need to rethink this a little. It would have to
look something like this (UNTESTED):


dims      = (/naz,nr/)
dims_1d   = product(dims)
stats_2d  = new((/nfiles,dims_1d/),float)
maxlat_1d = new(dims_1d,float)
maxlon_1d = new(dims_1d,float)
do ifile = 0,nfiles-1
   indat = readAsciiTable(infile(ifile),7,indat_type,1)
   field = indat(:,5)
   iaz     = toint(floor(indat(:,2)/0.5))
   ir      = toint(floor(indat(:,3)/gate))
   indexes = (iaz*(nr-1)) + ir

   maxlat_1d(indexes)      = indat(:,0)
   maxlon_1d(indexes)      = indat(:,1)
   stats_2d(ifile,indexes) = abs(field)
   . . .
end do
maxlat = reshape(maxlat_1d,dims)
maxlon = reshape(maxlon_1d,dims)
stats  = reshape(stats_2d,(/nfiles,naz,nr/))
delete([/maxlat_1d,maxlon_1d,stats_2d/])

[3] This is just a small tip that won't help much with memory, but in a
"new" statement, you don't need to enclose the type in quotes:

stats = new((/nfiles,naz,nr/),float)
. . .
maxfield = new((/naz,nr/),float)
. . .
maxlat = new((/naz,nr/),float)
. . .

--Mary



On Tue, Aug 28, 2018 at 7:53 AM, robinsonp cswr.org <robinsonp at cswr.org>
wrote:

> Hi all,
>
>
> I am trying to read a large number of files, using the data to fill an
> array which has been declared with a new statement.  After each file is
> read, all arrays no longer needed are deleted prior to reading the next
> file.  Yet after 80 files are read, I get an NhlMalloc Failed error.
> "fatal:NhlMalloc Failed:[errno=12]".  If I read all the files in smaller
> groups with individual runs of the program, I get no errors, which suggests
> the files them selves are not to blame.  It SEEMS there is a memory leak.
> I have tried many things and read this forum on the topic.
>
>
> My ncl version for RedHat is ncl_ncarg-6.3.0.Linux_RHEL5.
> 11_i686_nodap_gcc412.tar.gz.
>
>
> Code snippet of loop within which error occurs:
>
> ___________________________________________________________
>
> naz = 720
> nr = 976
>
> infile = systemfunc("ls rawdats/rawdat0*")
>
> nfiles = dimsizes(infile)
>
> gate = 50.0 ; from swp metadata
>
> stats = new((/nfiles,naz,nr/),"float")
> stats at _FillValue = bad
> stats = stats at _FillValue
> maxfield = new((/naz,nr/),"float")
> maxfield at _FillValue = bad
> maxfield = 0.0
> maxlat = new((/naz,nr/),"float")
> maxlon = new((/naz,nr/),"float")
>
> do ifile = 0,nfiles-1
>    indat = readAsciiTable(infile(ifile),7,"float",1)
>    field = indat(:,5)
>    lat1d = indat(:,0)
>    lon1d = indat(:,1)
>
>    numrows = dimsizes(field)
>    az = indat(:,2)
>    rn = indat(:,3)
>
>   do i = 0,numrows-1
>    ; Determine indeces for az,range and load into stats array
>      iaz = toint(floor(az(i)/0.5))
>      ir = toint(floor(rn(i)/gate))
>
>    ; Add this data to stats array
>      stats(ifile,iaz,ir) = abs(field(i))
>
>      maxlat(iaz,ir) = lat1d(i)
>      maxlon(iaz,ir) = lon1d(i)
>
>    end do
>
>    delete(indat)
>    delete(numrows)
>    delete(az)
>    delete(rn)
>    delete(field)
>    delete(lat1d)
>    delete(lon1d)
> end do
>
> _______________________________________________________________
>
> Any ideas?
>
>
> Paul Robinson
>
> Center for Severe Weather Research in Boulder, CO
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180828/bf08a504/attachment.html>