[pyngl-talk] PyNIO not closing grib codetables?

[Root, Benjamin]

Dived into the code, I think I found the problem. The particular RAP 
files I am reading does emit a warning:
warning: Entry (242) not found in code table file 
/nas/home/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table

This particular issue doesn't impact me for the data I am dealing with, 
so I ignore it. Well, it seems that in that particular code branch, the 
file pointer does not get closed.

In Grib2ReadCodeTable(), at line 7980, the function returns after 
emitting that warning, but fails to close the fp pointer. I am going to 
try adding a "(void) fclose(fp);" line to it, rebuild and install and 
try it out.

Cheers!
Ben Root

On Mon 03 Nov 2014 10:03:49 AM EST, Root, Benjamin wrote:
> I had left a long-running process over the weekend to analyze about a
> year's worth of hourly RAP data (grib2 format). I would load up a day's
> worth using PyNIO, process them, and then close the files before moving
> onto the next day. When I got into work this morning, the analysis
> errorred out saying that too many files were open.
>
> I restarted it for a smaller analysis, and ran "lsof" while it executed.
> Here is a snippet of its output:
>
> 3
> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
> python2.7 17092     broot  140r      REG              253,4
> 4998    5909053
> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
> python2.7 17092     broot  141r      REG              253,4
> 4998    5909053
> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
> python2.7 17092     broot  142r      REG              253,4
> 4998    5909053
> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
> python2.7 17092     broot  143r      REG              253,4
> 4998    5909053
> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
> python2.7 17092     broot  144r      REG              253,4
> 4998    5909053
> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
> python2.7 17092     broot  145r      REG              253,4
> 4998    5909053
> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
> python2.7 17092     broot  146r      REG              253,4
> 4998    5909053
> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
> python2.7 17092     broot  147r      REG              253,4
> 4998    5909053
> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
> python2.7 17092     broot  148r      REG              253,4
> 4998    5909053
> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
> python2.7 17092     broot  149r      REG              253,4
> 4998    5909053
> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
> python2.7 17092     broot  150r      REG              253,4
> 4998    5909053
> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>
> It would seem that closing the grib file doesn't close the associated
> grib2 code table. As a side note, is there any reason why the codetable
> is not simply cached?
>
> Over a period of about 5 days worth of RAP files, I ran the lsof
> command, counting the number of times the table file appears:
>
> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
> 121
> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
> 135
> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
> 159
> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
> 179
> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
> 199
> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
> 217
> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
> 241
>
> My version of PyNIO is 1.4.1.
>
> Cheers!
> Ben Root
>
> ________________________________
>
> This email is intended solely for the recipient. It may contain privileged, proprietary or confidential information or material. If you are not the intended recipient, please delete this email and any attachments and notify the sender of the error.
>
> _______________________________________________
> pyngl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
>