[ncl-talk] NCL memory leak issue
STEFAN RAHIMI-ESFARJANI
s.rahimi at ucla.edu
Thu Oct 8 12:16:45 MDT 2020
Sorry, I didn't realize that I didn't reply to NCL talk....
See below a much simplified version of the code. By using top, VIRT and RES
increase substantially through each iteration of the day loop. If I comment
out the addfiles line and subsequent delete calls, there is no problem
memory-wise. Why is there iteratively more memory consumption upon the
addfiles call?
Thanks,
-Stefan
begin
domain = "d01"
dir = "/glade/scratch/srahimi/ucla/downscale/test/WRF-4.1.3/test/"
ndays = 365
iyear = 2019
dir_x = dir + "MPI_r8_" + iyear + "/" + domain + "/"
print (dir_x+"")
file_str = "wrfout*"
filei = systemfunc("cd " + dir_x + " ; ls " + file_str)
day_count = 1
do iday = 0, ndays - 1
files = filei(4*(day_count-1):4*day_count-1)
ff = addfiles(dir_x+files,"r")
delete(ff)
end do ;end day loop
end
On Thu, Oct 8, 2020 at 11:56 AM Dennis Shea <shea at ucar.edu> wrote:
> Hello,
>
> It is always best to respond the ncl-talk. I am sure there are people who
> know more than me about these types of issues.
> ---
> FYI: to my knowledge,
>
> ff = addfiles(dir_x+files,"r")
>
> The 'ff' is a variable of type 'list' Each element of 'ff' is a pointer
> [one 'location'] to each variable. So, it is not consuming memory.
> I am not aware of any memory leaks associated with netCDF files.
>
> Again, I will look this evening.
>
> Cheers
> D
>
>
>
>
>
>
>
> On Thu, Oct 8, 2020 at 11:27 AM STEFAN RAHIMI-ESFARJANI <s.rahimi at ucla.edu>
> wrote:
>
>> Hi Dennis,
>>
>> As you can see in this much simplified version of the code, the problem
>> just described persists....
>>
>> /glade/scratch/srahimi/test.ncl
>>
>> Thanks so much,
>> -Stefan
>>
>> On Thu, Oct 8, 2020 at 11:23 AM STEFAN RAHIMI-ESFARJANI <
>> s.rahimi at ucla.edu> wrote:
>>
>>> Thanks for the reply,
>>>
>>> The program is trying to write the created etrans_sfc
>>> <http://etrans_sfc.daily.mpi-esm1-2-lr_ssp370_r8i1p1f1_d01_1980.nc/>...nc
>>> to the /files directory. Since this directory isn't in your work directory,
>>> that is where the explicit issue is happening that you see.
>>>
>>> My issue is that, if you run the code for 365 days for a given year, top
>>> -u shea, you will see that there is a steady increase in VIRT and RES
>>> memory associated with the line ff = addfiles(stuff). Even when ff
>>> is deleted, the increase persists. This problem is independent of the .nc
>>> write follwoing the completion of the day loop, and the problem persists
>>> even if all var_wrf variables are removed from the code...
>>>
>>> I have seen references to this type of leak before with GRB2 files (
>>> https://mailman.ucar.edu/pipermail/ncl-talk/2017-June/009365.html), but
>>> it seems that iterations of NCL versions solved this issue with addfile....
>>>
>>> Is there a different way to release memory for ff?
>>>
>>> Thanks,
>>> -Stefan
>>>
>>> On Thu, Oct 8, 2020 at 11:02 AM Dennis Shea <shea at ucar.edu> wrote:
>>>
>>>> I copied your script and ran it with a few print statements added:
>>>>
>>>> /glade/work/shea/test.mem_leak.ncl
>>>>
>>>> %> ncl test.mem_leak.ncl
>>>>
>>>> ==================
>>>> /glade/work/shea>ncl test.mem_leak.ncl
>>>> Copyright (C) 1995-2019 - All Rights Reserved
>>>> University Corporation for Atmospheric Research
>>>> NCAR Command Language Version 6.6.2
>>>> The use of this software is governed by a License Agreement.
>>>> See http://www.ncl.ucar.edu/ for more details.
>>>> (0) /glade/scratch/srahimi/interp/meta2/wrfinput_d01
>>>>
>>>> Variable: var_wrf
>>>> Type: float
>>>> Total Size: 16550560 bytes
>>>> 4137640 values
>>>> Number of Dimensions: 3
>>>> Dimensions and sizes: [365] x [104] x [109]
>>>> Coordinates:
>>>> Number Of Attributes: 1
>>>> _FillValue : 9.96921e+36
>>>> (0) ------------------
>>>> (0)
>>>> /glade/scratch/srahimi/ucla/downscale/test/WRF-4.1.3/test/MPI_r8_1980/d01/
>>>> (0) iyear=1980: iday=0
>>>> (0) iyear=1980: iday=1
>>>> (0) iyear=1980: iday=2
>>>> [SNIP]
>>>> 0) iyear=1980: iday=362
>>>> (0) iyear=1980: iday=363
>>>> (0) iyear=1980: iday=364
>>>> (0) ====================
>>>> (0) ===EXIT IDAY LOOP===
>>>> (0) ====================
>>>>
>>>> Variable: var_wrf
>>>> Type: float
>>>> Total Size: 16550560 bytes
>>>> 4137640 values
>>>> Number of Dimensions: 3
>>>> Dimensions and sizes: [day | 365] x [lat2d | 104] x [lon2d | 109]
>>>> Coordinates:
>>>> day: [19800901..19810831]
>>>> Number Of Attributes: 2
>>>> units : mm/d
>>>> _FillValue : 9.96921e+36
>>>> (0)
>>>> (0) min=0 max=4.89127
>>>> (0) ====================
>>>>
>>>> /bin/rm: cannot remove 'files/
>>>> etrans_sfc.daily.mpi-esm1-2-lr_ssp370_r8i1p1f1_d01_1980.nc': No such
>>>> file or directory
>>>> fatal:Could not create (files/
>>>> etrans_sfc.daily.mpi-esm1-2-lr_ssp370_r8i1p1f1_d01_1980.nc)
>>>> fatal:["Execute.c":8637]:Execute: Error occurred at or near line 113 in
>>>> file test.mem_leak.ncl
>>>>
>>>> +++++++++++++++++++++++++++++
>>>> The 'iday' loop has
>>>>
>>>> 63 do iday = 0, ndays - 1
>>>> 64
>>>> 65 files = filei(4*(day_count-1):4*day_count-1)
>>>> 66 ;;print (files) ;each is a string
>>>>
>>>> The following is after the loop. 'files' is a 1D array of strings. If
>>>> so, the following looks incorrect.
>>>>
>>>> /bin/rm: cannot remove 'files/
>>>> etrans_sfc.daily.mpi-esm1-2-lr_ssp370_r8i1p1f1_d01_1980.nc': No such
>>>> file or directory
>>>>
>>>> On Thu, Oct 8, 2020 at 9:24 AM STEFAN RAHIMI-ESFARJANI via ncl-talk <
>>>> ncl-talk at mailman.ucar.edu> wrote:
>>>>
>>>>> Greetings,
>>>>>
>>>>>
>>>>> I spent a lot of time developing post-processing software for my
>>>>> research, which are written in NCL. The code here creates single-year files
>>>>> of daily evapotranspiration values on a lat/lon grid f(time,lat,lon). I am
>>>>> experiencing a memory leak that is affecting my ability to run these jobs
>>>>> using Cheyenne compute nodes.
>>>>>
>>>>> In short, I have traced the memory leak to the following line (70) in
>>>>> /glade/scratch/srahimi/interp/mpi_r8/test.ncl on Cheyenne or see
>>>>> below:
>>>>>
>>>>> ff = addfiles(dir_x+files,"r")
>>>>>
>>>>> The files are wrfout .nc files. Variable "ff" should be deleted 2
>>>>> lines later, and that space should be re-used throughout each iteration of
>>>>> the loop. What seems to be happening however is that new memory is being
>>>>> allocated for ff in each iteration of the loop, eventually overpowering the
>>>>> core and the node.
>>>>>
>>>>> I know ways around this in python, but I really do not want to rewrite
>>>>> these scripts at present. Any suggestions?
>>>>>
>>>>> Thanks,
>>>>> -Stefan Rahimi, UCLA
>>>>>
>>>>> ;Created by S. Rahimi on 15 Jan. 2020
>>>>>
>>>>> ;to interpolate WRF fields toa common
>>>>>
>>>>> ;0.03 rectilinear grid
>>>>>
>>>>>
>>>>> ;Daily averages from 6-h data
>>>>>
>>>>>
>>>>> ;For a 2-D variable
>>>>>
>>>>>
>>>>> ;f = f(year,day,ny,nx)
>>>>>
>>>>>
>>>>> begin
>>>>>
>>>>>
>>>>> var = "etrans_sfc"
>>>>>
>>>>> model = "mpi-esm1-2-lr"
>>>>>
>>>>> variant = "r8i1p1f1"
>>>>>
>>>>> ssp = "ssp370"
>>>>>
>>>>> freq = "daily"
>>>>>
>>>>> domain = "d01"
>>>>>
>>>>>
>>>>> startyear = 1980
>>>>>
>>>>> endyear = 1989
>>>>>
>>>>> year0 = 1980
>>>>>
>>>>>
>>>>> dir = "/glade/scratch/srahimi/ucla/downscale/test/WRF-4.1.3/test/"
>>>>>
>>>>> dir_inp = "/glade/scratch/srahimi/interp/meta2/"
>>>>>
>>>>> file_wrfinput = "/glade/scratch/srahimi/interp/meta2/wrfinput_d01"
>>>>>
>>>>>
>>>>> s2d = 86400.
>>>>>
>>>>>
>>>>> ndays = 365
>>>>>
>>>>> nyears = endyear - startyear + 1
>>>>>
>>>>>
>>>>> nx = 109
>>>>>
>>>>> ny = 104
>>>>>
>>>>>
>>>>> ;lat/lon from wrfinput
>>>>>
>>>>> print (file_wrfinput+"")
>>>>>
>>>>> f = addfile(file_wrfinput,"r")
>>>>>
>>>>> lat_wrf = f->XLAT(0,:,:)
>>>>>
>>>>> lon_wrf = f->XLONG(0,:,:)
>>>>>
>>>>> delete(f)
>>>>>
>>>>>
>>>>> years = ispan(startyear,endyear,1)
>>>>>
>>>>> days = new((/ndays/),"integer")
>>>>>
>>>>> var_wrf = new((/ndays,ny,nx/),"float")
>>>>>
>>>>>
>>>>> do iyear = startyear, endyear
>>>>>
>>>>>
>>>>> if (iyear .eq. 2014 ) then
>>>>>
>>>>> continue
>>>>>
>>>>> end if
>>>>>
>>>>>
>>>>> dir_x = dir + "MPI_r8_" + iyear + "/" + domain + "/"
>>>>>
>>>>>
>>>>> print (dir_x+"")
>>>>>
>>>>> file_str = "wrfout*"
>>>>>
>>>>> filei = systemfunc("cd " + dir_x + " ; ls " + file_str)
>>>>>
>>>>>
>>>>> day_count = 1
>>>>>
>>>>>
>>>>> do iday = 0, ndays - 1
>>>>>
>>>>>
>>>>> files = filei(4*(day_count-1):4*day_count-1)
>>>>>
>>>>> print (files) ;each is a string
>>>>>
>>>>>
>>>>> date_str = stringtochar(files(0))
>>>>>
>>>>> date = tostring(date_str(11:14)) + tostring(date_str(16:17)) +
>>>>> tostring(date_str(19:20))
>>>>>
>>>>> days(iday) = stringtointeger(date)
>>>>>
>>>>>
>>>>> ff = addfiles(dir_x+files,"r")
>>>>>
>>>>> var_wrf(iday,:,:)
>>>>> = dim_avg_n_Wrap(ff[0:dimsizes(files)-1]->ETRAN,0) * s2d
>>>>>
>>>>> delete(ff)
>>>>>
>>>>> var_wrf(iday,:,:) = where(abs(var_wrf(iday,:,:)).gt.1e20,0.,
>>>>> var_wrf(iday,:,:))
>>>>>
>>>>>
>>>>> day_count = day_count + 1
>>>>>
>>>>> delete(files)
>>>>>
>>>>>
>>>>> end do ;end day loop
>>>>>
>>>>>
>>>>> var_wrf!0 = "day"
>>>>>
>>>>> var_wrf!1 = "lat2d"
>>>>>
>>>>> var_wrf!2 = "lon2d"
>>>>>
>>>>> var_wrf&day = days
>>>>>
>>>>> var_wrf at units = "mm/d"
>>>>>
>>>>>
>>>>> ;Write to .nc files
>>>>>
>>>>> dirout = "files/"
>>>>>
>>>>> fil_out = var + "." + freq + "." + \
>>>>>
>>>>> model + "_" + ssp + "_" + variant + "_" + \
>>>>>
>>>>> domain + "_" + iyear + ".nc"
>>>>>
>>>>> outfile = dirout+fil_out
>>>>>
>>>>> system("/bin/rm " +outfile)
>>>>>
>>>>> ncdf = addfile(outfile ,"c") ; open output netCDF file
>>>>>
>>>>> ;setfileoption(outfile,"DefineMode",True)
>>>>>
>>>>> fAtt = True ; assign file attributes
>>>>>
>>>>> fAtt at title = var+ " for "+iyear
>>>>>
>>>>> fAtt at Conventions = "None"
>>>>>
>>>>> fAtt at creation_date = systemfunc("date")
>>>>>
>>>>> ;fileattdef(ncdf,f[0]) ; copy file attributes
>>>>>
>>>>> filedimdef(ncdf,"region",-1,True)
>>>>>
>>>>>
>>>>> ncdf->$var$ = var_wrf
>>>>>
>>>>>
>>>>> delete(ncdf)
>>>>>
>>>>>
>>>>> end do ;end year loop
>>>>>
>>>>>
>>>>> end
>>>>>
>>>>> _______________________________________________
>>>>> ncl-talk mailing list
>>>>> ncl-talk at mailman.ucar.edu
>>>>> List instructions, subscriber options, unsubscribe:
>>>>> https://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20201008/cb34ed40/attachment.html>
More information about the ncl-talk
mailing list