[ncl-talk] NCL addfile memory leak in functions
Ghassan Alaka - NOAA Federal
ghassan.alaka at noaa.gov
Wed Dec 11 14:59:45 MST 2019
Hi All,
Similar but different to a previous submission (see link below), I am
experiencing a memory leak when reading data from a file that was read in
with the function "addfile". There are some key differences in my situation
from the one in the link:
1) I am creating variable "f" using "f = addfile()" in my parent script to
read a GRIB2 data file.
2) I wrote a custom function that resides in another script and takes "f"
as an input variable. This was done because an undetermined number of
variables need to be read from the GRIB2 data file described by "f". Plus,
a lot of ugly logic related to the actual reading of the data is stored in
this function as well.
3) In the function, I read the data into a variable "V" and return that
variable to the parent script.
4) In the parent script, a loop iterates over multiple GRIB2 data files.
The variables "f" and "V" are eventually overwritten by new data.
5) I am very careful to delete old variables, including "f" and "V" in the
parent script.
6) I have monitored the memory increasing until it sometimes reaches the
maximum value and the system kills my job. This out-of-memory kill occurs
after several iterations through the list of GRIB2 files.
http://mailman.ucar.edu/pipermail/ncl-talk/2017-July/009512.html
Given this setup, I am experiencing a memory leak when reading the data
from file "f" into variable "V" in the function called from the parent
script. I can't delete "f" within the function because it is an input
argument and its deletion is not allowed. I directly tested the following
and monitored memory usage with "systemfunc("ps u")" calls from within the
script.
a) If I read the data from file "f" into variable "V" in the primary
script, then "delete([/V,f/])", there is no apparent memory leak.
b) If I create a variable "V" using the "new()" function with the same
dimensions as the data from the GRIB2 file, then delete it, there is no
apparent memory leak.
c) If I create file "f" from the primary script, then pass "f" as a
function argument and read the data into variable "V", then return "V" to
the primary script, then "delete([/V,f/])", *there is an apparent memory
leak*.
I have not tested what happens if file "f" is defined as a local variable
in the function and then explicitly deleted before "V" is returned to the
parent script. Although that may work, it is not a feasible solution given
my the script would slow down significantly by repeatedly executing
"f=addfile()". Another possible solution is to execute "f->VAR" from within
the primary script. However, such a change would negatively impact these
scripts since I have a lot of logic related to the creation of "V" that I
am able to hide from the primary script.
Below I pasted a sample script using NCL V6.6.2 and its output that I used
to test these issues. The issue was reproducible using NCL V6.5.0. Can
variable type "file" be safely passed as a function argument? Do you think
the reference to "f" is being copied by the function and the extra memory
associated with the "f->" statement is never released? Any ideas on the
best way forward while maintaining the efficiency of my function usage?
Best,
Gus
--
Ghassan J. Alaka, Jr., Ph.D.
NOAA/AOML/Hurricane Research Division
4301 Rickenbacker Causeway
Miami. FL 33149
Tel: (305)361-4409
Fax: (305)361-4402
**Opinions expressed in this email are my own and do not represent AOML or
NOAA.**
*NCL SCRIPT*
; A simple NCL script that shows a potential memory leak when using
"addfile"
load "$GPLOT_DIR/ncl/GPLOT_util.ncl"
load "$GPLOT_DIR/ncl/GPLOT_func.ncl"
begin
IFILE = "/PATH/TO/FILE/natl00l.2019082700.hafsprs.synoptic.0p03.f000.grb2"
VAR = "UGRD_P0_L100_GLL0"
ILON = "lon_0"
print("Checking original memory usage")
psrep = systemfunc("ps u")
print("MEMORY CHECK 01: "+psrep)
print("")
print("Using addfile to read the data file")
f = addfile(IFILE,"r")
psrep = systemfunc("ps u")
print("MEMORY CHECK 02: "+psrep)
print("")
print("Creating new variable V (from data file)")
V = f->$VAR$
psrep = systemfunc("ps u")
print("MEMORY CHECK 03: "+psrep)
print("")
print("Deleting variable V (from data file)")
;delete([/V,f/])
delete(V)
psrep = systemfunc("ps u")
print("MEMORY CHECK 04: "+psrep)
print("")
print("Deleting variable f (data file)")
;delete([/V,f/])
delete(f)
psrep = systemfunc("ps u")
print("MEMORY CHECK 05: "+psrep)
print("")
print("Creating new variable V (not from data file)")
V := new((/46,2001,3501/),"float")
psrep = systemfunc("ps u")
print("MEMORY CHECK 06: "+psrep)
print("")
print("Deleting variable V (not from data file)")
delete(V)
psrep = systemfunc("ps u")
print("MEMORY CHECK 07: "+psrep)
print("")
print("Creating new variable V2 (not from data file)")
V2 = new((/46,2001,3501/),"float")
psrep = systemfunc("ps u")
print("MEMORY CHECK 08: "+psrep)
print("")
print("Deleting variable V2 (not from data file)")
delete(V2)
psrep = systemfunc("ps u")
print("MEMORY CHECK 09: "+psrep)
list_vars()
print("")
print("Creating new variable V (from data file via function)")
f = addfile(IFILE,"r")
LON = f->$ILON$-360.
LON!0 = "lon"
LON&lon = LON
Vdims := dimsizes(getfilevardimsizes(f,findVarName("HAFS","U","850")))
VdNames := getfilevardims(f,findVarName("HAFS","U","850"))
Vdims at dNames = VdNames
V :=
getVar2d(f,"HAFS",(/"U","850","0"/),Vdims,(/40.,0.,-100.,-20./),1.,False,LON)
psrep = systemfunc("ps u")
print("MEMORY CHECK 10: "+psrep)
print("")
print("Deleting variable V (from data file via function)")
delete([/V,f,Vdims,VdNames,LON/])
psrep = systemfunc("ps u")
print("MEMORY CHECK 11: "+psrep)
list_vars()
end
*OUTPUT*
[Ghassan.Alaka at v139 ~]$ ncl ~/NCL/scripts/memoryleak.ncl
Copyright (C) 1995-2019 - All Rights Reserved
University Corporation for Atmospheric Research
NCAR Command Language Version 6.6.2
The use of this software is governed by a License Agreement.
See http://www.ncl.ucar.edu/ for more details.
(0) Checking original memory usage
(0) MEMORY CHECK 01: USER PID %CPU %MEM VSZ RSS TTY
STAT START TIME COMMAND
(1) MEMORY CHECK 01: Ghassan+ 7761 0.0 0.0 119692 2116 pts/0 Ss
20:46 0:00 -bin/tcsh
(2) MEMORY CHECK 01: Ghassan+ 8129 33.0 0.1 232392 126320 pts/0 S+
20:50 0:00 ncl /home/Ghassan.Alaka/NCL/scripts/memoryleak.ncl
(3) MEMORY CHECK 01: Ghassan+ 8134 0.0 0.1 232392 122128 pts/0 S+
20:50 0:00 ncl /home/Ghassan.Alaka/NCL/scripts/memoryleak.ncl
(4) MEMORY CHECK 01: Ghassan+ 8135 0.0 0.0 155372 1860 pts/0 R+
20:50 0:00 ps u
(0)
(0) Using addfile to read the data file
(0) MEMORY CHECK 02: USER PID %CPU %MEM VSZ RSS TTY
STAT START TIME COMMAND
(1) MEMORY CHECK 02: Ghassan+ 7761 0.0 0.0 119692 2116 pts/0 Ss
20:46 0:00 -bin/tcsh
(2) MEMORY CHECK 02: Ghassan+ 8129 38.0 0.2 241416 135612 pts/0 S+
20:50 0:00 ncl /home/Ghassan.Alaka/NCL/scripts/memoryleak.ncl
(3) MEMORY CHECK 02: Ghassan+ 8137 0.0 0.1 241416 131100 pts/0 S+
20:50 0:00 ncl /home/Ghassan.Alaka/NCL/scripts/memoryleak.ncl
(4) MEMORY CHECK 02: Ghassan+ 8138 0.0 0.0 155372 1856 pts/0 R+
20:50 0:00 ps u
(0)
(0) Creating new variable V (from data file)
(0) MEMORY CHECK 03: USER PID %CPU %MEM VSZ RSS TTY
STAT START TIME COMMAND
(1) MEMORY CHECK 03: Ghassan+ 7761 0.0 0.0 119692 2116 pts/0 Ss
20:46 0:00 -bin/tcsh
(2) MEMORY CHECK 03: Ghassan+ 8129 95.4 2.2 1597980 1492360 pts/0 S+
20:50 0:14 ncl /home/Ghassan.Alaka/NCL/scripts/memoryleak.ncl
(3) MEMORY CHECK 03: Ghassan+ 8150 0.0 2.2 1597980 1487716 pts/0 S+
20:50 0:00 ncl /home/Ghassan.Alaka/NCL/scripts/memoryleak.ncl
(4) MEMORY CHECK 03: Ghassan+ 8151 0.0 0.0 155372 1856 pts/0 R+
20:50 0:00 ps u
(0)
(0) Deleting variable V (from data file)
(0) MEMORY CHECK 04: USER PID %CPU %MEM VSZ RSS TTY
STAT START TIME COMMAND
(1) MEMORY CHECK 04: Ghassan+ 7761 0.0 0.0 119692 2116 pts/0 Ss
20:46 0:00 -bin/tcsh
(2) MEMORY CHECK 04: Ghassan+ 8129 95.5 0.3 339176 233560 pts/0 S+
20:50 0:14 ncl /home/Ghassan.Alaka/NCL/scripts/memoryleak.ncl
(3) MEMORY CHECK 04: Ghassan+ 8152 0.0 0.3 339176 228984 pts/0 S+
20:50 0:00 ncl /home/Ghassan.Alaka/NCL/scripts/memoryleak.ncl
(4) MEMORY CHECK 04: Ghassan+ 8153 0.0 0.0 155372 1864 pts/0 R+
20:50 0:00 ps u
(0)
(0) Deleting variable f (data file)
(0) MEMORY CHECK 05: USER PID %CPU %MEM VSZ RSS TTY
STAT START TIME COMMAND
(1) MEMORY CHECK 05: Ghassan+ 7761 0.0 0.0 119692 2116 pts/0 Ss
20:46 0:00 -bin/tcsh
(2) MEMORY CHECK 05: Ghassan+ 8129 95.6 0.1 235108 129496 pts/0 S+
20:50 0:14 ncl /home/Ghassan.Alaka/NCL/scripts/memoryleak.ncl
(3) MEMORY CHECK 05: Ghassan+ 8154 0.0 0.1 235108 124916 pts/0 S+
20:50 0:00 ncl /home/Ghassan.Alaka/NCL/scripts/memoryleak.ncl
(4) MEMORY CHECK 05: Ghassan+ 8155 0.0 0.0 155372 1860 pts/0 R+
20:50 0:00 ps u
(0)
(0) Creating new variable V (not from data file)
(0) MEMORY CHECK 06: USER PID %CPU %MEM VSZ RSS TTY
STAT START TIME COMMAND
(1) MEMORY CHECK 06: Ghassan+ 7761 0.0 0.0 119692 2116 pts/0 Ss
20:46 0:00 -bin/tcsh
(2) MEMORY CHECK 06: Ghassan+ 8129 93.3 2.1 1493912 1388304 pts/0 S+
20:50 0:14 ncl /home/Ghassan.Alaka/NCL/scripts/memoryleak.ncl
(3) MEMORY CHECK 06: Ghassan+ 8156 0.0 2.0 1493912 1383720 pts/0 S+
20:50 0:00 ncl /home/Ghassan.Alaka/NCL/scripts/memoryleak.ncl
(4) MEMORY CHECK 06: Ghassan+ 8157 0.0 0.0 155372 1864 pts/0 R+
20:50 0:00 ps u
(0)
(0) Deleting variable V (not from data file)
(0) MEMORY CHECK 07: USER PID %CPU %MEM VSZ RSS TTY
STAT START TIME COMMAND
(1) MEMORY CHECK 07: Ghassan+ 7761 0.0 0.0 119692 2116 pts/0 Ss
20:46 0:00 -bin/tcsh
(2) MEMORY CHECK 07: Ghassan+ 8129 93.3 0.1 235108 129500 pts/0 S+
20:50 0:14 ncl /home/Ghassan.Alaka/NCL/scripts/memoryleak.ncl
(3) MEMORY CHECK 07: Ghassan+ 8158 0.0 0.1 235108 124916 pts/0 S+
20:50 0:00 ncl /home/Ghassan.Alaka/NCL/scripts/memoryleak.ncl
(4) MEMORY CHECK 07: Ghassan+ 8159 0.0 0.0 155372 1860 pts/0 R+
20:50 0:00 ps u
(0)
(0) Creating new variable V2 (not from data file)
(0) MEMORY CHECK 08: USER PID %CPU %MEM VSZ RSS TTY
STAT START TIME COMMAND
(1) MEMORY CHECK 08: Ghassan+ 7761 0.0 0.0 119692 2116 pts/0 Ss
20:46 0:00 -bin/tcsh
(2) MEMORY CHECK 08: Ghassan+ 8129 91.4 2.1 1493912 1388304 pts/0 S+
20:50 0:15 ncl /home/Ghassan.Alaka/NCL/scripts/memoryleak.ncl
(3) MEMORY CHECK 08: Ghassan+ 8161 0.0 2.0 1493912 1383720 pts/0 S+
20:51 0:00 ncl /home/Ghassan.Alaka/NCL/scripts/memoryleak.ncl
(4) MEMORY CHECK 08: Ghassan+ 8162 0.0 0.0 155372 1860 pts/0 R+
20:51 0:00 ps u
(0)
(0) Deleting variable V2 (not from data file)
(0) MEMORY CHECK 09: USER PID %CPU %MEM VSZ RSS TTY
STAT START TIME COMMAND
(1) MEMORY CHECK 09: Ghassan+ 7761 0.0 0.0 119692 2116 pts/0 Ss
20:46 0:00 -bin/tcsh
(2) MEMORY CHECK 09: Ghassan+ 8129 91.4 0.1 235108 129500 pts/0 S+
20:50 0:15 ncl /home/Ghassan.Alaka/NCL/scripts/memoryleak.ncl
(3) MEMORY CHECK 09: Ghassan+ 8163 0.0 0.1 235108 124916 pts/0 S+
20:51 0:00 ncl /home/Ghassan.Alaka/NCL/scripts/memoryleak.ncl
(4) MEMORY CHECK 09: Ghassan+ 8164 0.0 0.0 155372 1860 pts/0 R+
20:51 0:00 ps u
float pi [ 1 ]
string GPLOT [ 1 ]
float ms2kts [ 1 ]
string psrep [ 5 ]
float fval3 [ 1 ]
integer fval2 [ 1 ]
string ILON [ 1 ]
float d2r [ 1 ]
string VAR [ 1 ]
float r [ 1 ]
float g [ 1 ]
logical GSN_OLD_LABELBAR [ 1 ]
string IFILE [ 1 ]
float fval [ 1 ]
float omega [ 1 ]
(0)
(0) Creating new variable V (from data file via function)
(0) MEMORY CHECK 10: USER PID %CPU %MEM VSZ RSS TTY
STAT START TIME COMMAND
(1) MEMORY CHECK 10: Ghassan+ 7761 0.0 0.0 119692 2116 pts/0 Ss
20:46 0:00 -bin/tcsh
(2) MEMORY CHECK 10: Ghassan+ 8129 90.5 0.4 384024 277172 pts/0 S+
20:50 0:16 ncl /home/Ghassan.Alaka/NCL/scripts/memoryleak.ncl
(3) MEMORY CHECK 10: Ghassan+ 8166 0.0 0.4 384024 272304 pts/0 S+
20:51 0:00 ncl /home/Ghassan.Alaka/NCL/scripts/memoryleak.ncl
(4) MEMORY CHECK 10: Ghassan+ 8167 0.0 0.0 155372 1860 pts/0 R+
20:51 0:00 ps u
(0)
(0) Deleting variable V (from data file via function)
(0) MEMORY CHECK 11: USER PID %CPU %MEM VSZ RSS TTY
STAT START TIME COMMAND
(1) MEMORY CHECK 11: Ghassan+ 7761 0.0 0.0 119692 2116 pts/0 Ss
20:46 0:00 -bin/tcsh
(2) MEMORY CHECK 11: Ghassan+ 8129 90.6 0.2 267228 161908 pts/0 S+
20:50 0:16 ncl /home/Ghassan.Alaka/NCL/scripts/memoryleak.ncl
(3) MEMORY CHECK 11: Ghassan+ 8168 0.0 0.2 267228 157040 pts/0 S+
20:51 0:00 ncl /home/Ghassan.Alaka/NCL/scripts/memoryleak.ncl
(4) MEMORY CHECK 11: Ghassan+ 8169 0.0 0.0 155372 1860 pts/0 R+
20:51 0:00 ps u
float pi [ 1 ]
string GPLOT [ 1 ]
float ms2kts [ 1 ]
string psrep [ 5 ]
float fval3 [ 1 ]
integer fval2 [ 1 ]
string ILON [ 1 ]
float d2r [ 1 ]
string VAR [ 1 ]
float r [ 1 ]
float g [ 1 ]
logical GSN_OLD_LABELBAR [ 1 ]
string IFILE [ 1 ]
float fval [ 1 ]
float omega [ 1 ]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20191211/cd55bb94/attachment.html>
More information about the ncl-talk
mailing list