[pyngl-talk] PyNIO not closing grib codetables?

[Root, Benjamin]

I had left a long-running process over the weekend to analyze about a
year's worth of hourly RAP data (grib2 format). I would load up a day's
worth using PyNIO, process them, and then close the files before moving
onto the next day. When I got into work this morning, the analysis
errorred out saying that too many files were open.

I restarted it for a smaller analysis, and ran "lsof" while it executed.
Here is a snippet of its output:

3
/rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
python2.7 17092     broot  140r      REG              253,4
4998    5909053
/rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
python2.7 17092     broot  141r      REG              253,4
4998    5909053
/rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
python2.7 17092     broot  142r      REG              253,4
4998    5909053
/rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
python2.7 17092     broot  143r      REG              253,4
4998    5909053
/rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
python2.7 17092     broot  144r      REG              253,4
4998    5909053
/rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
python2.7 17092     broot  145r      REG              253,4
4998    5909053
/rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
python2.7 17092     broot  146r      REG              253,4
4998    5909053
/rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
python2.7 17092     broot  147r      REG              253,4
4998    5909053
/rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
python2.7 17092     broot  148r      REG              253,4
4998    5909053
/rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
python2.7 17092     broot  149r      REG              253,4
4998    5909053
/rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
python2.7 17092     broot  150r      REG              253,4
4998    5909053
/rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table

It would seem that closing the grib file doesn't close the associated
grib2 code table. As a side note, is there any reason why the codetable
is not simply cached?

Over a period of about 5 days worth of RAP files, I ran the lsof
command, counting the number of times the table file appears:

[centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
121
[centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
135
[centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
159
[centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
179
[centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
199
[centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
217
[centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
241

My version of PyNIO is 1.4.1.

Cheers!
Ben Root

________________________________

This email is intended solely for the recipient. It may contain privileged, proprietary or confidential information or material. If you are not the intended recipient, please delete this email and any attachments and notify the sender of the error.