[ncl-talk] NCL memory leak issue

Thu Oct 8 11:00:46 MDT 2020

I copied your script and ran it with a few print statements added:

/glade/work/shea/test.mem_leak.ncl

%> ncl test.mem_leak.ncl

==================
/glade/work/shea>ncl test.mem_leak.ncl
 Copyright (C) 1995-2019 - All Rights Reserved
 University Corporation for Atmospheric Research
 NCAR Command Language Version 6.6.2
 The use of this software is governed by a License Agreement.
 See http://www.ncl.ucar.edu/ for more details.
(0) /glade/scratch/srahimi/interp/meta2/wrfinput_d01

Variable: var_wrf
Type: float
Total Size: 16550560 bytes
            4137640 values
Number of Dimensions: 3
Dimensions and sizes: [365] x [104] x [109]
Coordinates:
Number Of Attributes: 1
  _FillValue : 9.96921e+36
(0) ------------------
(0)
/glade/scratch/srahimi/ucla/downscale/test/WRF-4.1.3/test/MPI_r8_1980/d01/
(0) iyear=1980: iday=0
(0) iyear=1980: iday=1
(0) iyear=1980: iday=2
[SNIP]
0) iyear=1980: iday=362
(0) iyear=1980: iday=363
(0) iyear=1980: iday=364
(0) ====================
(0) ===EXIT IDAY LOOP===
(0) ====================

Variable: var_wrf
Type: float
Total Size: 16550560 bytes
            4137640 values
Number of Dimensions: 3
Dimensions and sizes: [day | 365] x [lat2d | 104] x [lon2d | 109]
Coordinates:
            day: [19800901..19810831]
Number Of Attributes: 2
  units : mm/d
  _FillValue : 9.96921e+36
(0)
(0) min=0   max=4.89127
(0) ====================

/bin/rm: cannot remove 'files/
etrans_sfc.daily.mpi-esm1-2-lr_ssp370_r8i1p1f1_d01_1980.nc': No such file
or directory
fatal:Could not create (files/
etrans_sfc.daily.mpi-esm1-2-lr_ssp370_r8i1p1f1_d01_1980.nc)
fatal:["Execute.c":8637]:Execute: Error occurred at or near line 113 in
file test.mem_leak.ncl

+++++++++++++++++++++++++++++
The 'iday'  loop has

 63  do iday = 0, ndays - 1
 64
 65   files = filei(4*(day_count-1):4*day_count-1)
 66 ;;print (files) ;each is a string

The following is after the loop. 'files' is a 1D array of strings. If so,
the following looks incorrect.

/bin/rm: cannot remove 'files/
etrans_sfc.daily.mpi-esm1-2-lr_ssp370_r8i1p1f1_d01_1980.nc': No such file
or directory

On Thu, Oct 8, 2020 at 9:24 AM STEFAN RAHIMI-ESFARJANI via ncl-talk <
ncl-talk at mailman.ucar.edu> wrote:

> Greetings,
>
>
> I spent a lot of time developing post-processing software for my research,
> which are written in NCL. The code here creates single-year files of
> daily evapotranspiration values on a lat/lon grid f(time,lat,lon). I am
> experiencing a memory leak that is affecting my ability to run these jobs
> using Cheyenne compute nodes.
>
> In short, I have traced the memory leak to the following line (70) in
> /glade/scratch/srahimi/interp/mpi_r8/test.ncl on Cheyenne or see below:
>
> ff = addfiles(dir_x+files,"r")
>
> The files are wrfout .nc files. Variable "ff" should be deleted 2 lines
> later, and that space should be re-used throughout each iteration of the
> loop. What seems to be happening however is that new memory is being
> allocated for ff in each iteration of the loop, eventually overpowering the
> core and the node.
>
> I know ways around this in python, but I really do not want to rewrite
> these scripts at present. Any suggestions?
>
> Thanks,
> -Stefan Rahimi, UCLA
>
> ;Created by S. Rahimi on 15 Jan. 2020
>
> ;to interpolate WRF fields toa common
>
> ;0.03 rectilinear grid
>
>
> ;Daily averages from 6-h data
>
>
> ;For a 2-D variable
>
>
> ;f = f(year,day,ny,nx)
>
>
> begin
>
>
> var = "etrans_sfc"
>
> model  = "mpi-esm1-2-lr"
>
> variant  = "r8i1p1f1"
>
> ssp  = "ssp370"
>
> freq = "daily"
>
> domain  = "d01"
>
>
> startyear  = 1980
>
> endyear  = 1989
>
> year0 = 1980
>
>
> dir  = "/glade/scratch/srahimi/ucla/downscale/test/WRF-4.1.3/test/"
>
> dir_inp  = "/glade/scratch/srahimi/interp/meta2/"
>
> file_wrfinput  = "/glade/scratch/srahimi/interp/meta2/wrfinput_d01"
>
>
> s2d = 86400.
>
>
> ndays = 365
>
> nyears = endyear - startyear + 1
>
>
> nx  = 109
>
> ny  = 104
>
>
> ;lat/lon from wrfinput
>
> print (file_wrfinput+"")
>
> f = addfile(file_wrfinput,"r")
>
> lat_wrf = f->XLAT(0,:,:)
>
> lon_wrf = f->XLONG(0,:,:)
>
> delete(f)
>
>
> years = ispan(startyear,endyear,1)
>
> days = new((/ndays/),"integer")
>
> var_wrf = new((/ndays,ny,nx/),"float")
>
>
> do iyear = startyear, endyear
>
>
> if (iyear .eq. 2014 ) then
>
>         continue
>
> end if
>
>
>  dir_x =  dir + "MPI_r8_" + iyear + "/" + domain + "/"
>
>
>  print (dir_x+"")
>
>  file_str = "wrfout*"
>
>  filei = systemfunc("cd " + dir_x + " ; ls " + file_str)
>
>
>  day_count = 1
>
>
>  do iday = 0, ndays - 1
>
>
>   files = filei(4*(day_count-1):4*day_count-1)
>
>   print (files) ;each is a string
>
>
>   date_str = stringtochar(files(0))
>
>   date = tostring(date_str(11:14)) + tostring(date_str(16:17)) +
> tostring(date_str(19:20))
>
>   days(iday) = stringtointeger(date)
>
>
>   ff = addfiles(dir_x+files,"r")
>
>   var_wrf(iday,:,:) =  dim_avg_n_Wrap(ff[0:dimsizes(files)-1]->ETRAN,0) *
> s2d
>
>   delete(ff)
>
>   var_wrf(iday,:,:) = where(abs(var_wrf(iday,:,:)).gt.1e20,0.,
> var_wrf(iday,:,:))
>
>
>   day_count = day_count + 1
>
>   delete(files)
>
>
>  end do         ;end day loop
>
>
>  var_wrf!0 = "day"
>
>  var_wrf!1 = "lat2d"
>
>  var_wrf!2 = "lon2d"
>
>  var_wrf&day = days
>
>  var_wrf at units = "mm/d"
>
>
>  ;Write to .nc files
>
>  dirout = "files/"
>
>  fil_out = var + "." + freq + "." + \
>
>   model + "_" + ssp + "_" + variant + "_" + \
>
>   domain + "_" + iyear + ".nc"
>
>  outfile = dirout+fil_out
>
>  system("/bin/rm " +outfile)
>
>  ncdf = addfile(outfile ,"c")  ; open output netCDF file
>
>  ;setfileoption(outfile,"DefineMode",True)
>
>  fAtt               = True            ; assign file attributes
>
>  fAtt at title         = var+ " for  "+iyear
>
>  fAtt at Conventions   = "None"
>
>  fAtt at creation_date = systemfunc("date")
>
>  ;fileattdef(ncdf,f[0])            ; copy file attributes
>
>  filedimdef(ncdf,"region",-1,True)
>
>
>  ncdf->$var$ = var_wrf
>
>
>  delete(ncdf)
>
>
> end do          ;end year loop
>
>
> end
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at mailman.ucar.edu
> List instructions, subscriber options, unsubscribe:
> https://mailman.ucar.edu/mailman/listinfo/ncl-talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20201008/fa350277/attachment.html>