[pyngl-talk] PyNIO not closing grib codetables?

[David Brown]

Hi Ben,
Thanks very much for your sleuthing and actually locating the problem.
In fact, I believe this problem has been fixed in the development
code, but it has been quite awhile since we released a version of
PyNIO. We are planning a new release within the next couple of months
that will have quite a few new features (including support for much
more of the NetCDF4 specification). There are also many updates to the
GRIB1 parameter tables and the GRIB2 code tables along with numerous
bug fixes.

If you would like to get a development version of PyNIO let us know.
 -dave

On Mon, Nov 3, 2014 at 9:13 AM, Root, Benjamin <broot at aer.com> wrote:
> And I just realized that I didn't mention which file I was talking
> about: libsrc/NclGRIB2.c
>
> Ben
>
>
> On Mon 03 Nov 2014 10:58:33 AM EST, Root, Benjamin wrote:
>> Dived into the code, I think I found the problem. The particular RAP
>> files I am reading does emit a warning:
>> warning: Entry (242) not found in code table file
>> /nas/home/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>
>> This particular issue doesn't impact me for the data I am dealing with,
>> so I ignore it. Well, it seems that in that particular code branch, the
>> file pointer does not get closed.
>>
>> In Grib2ReadCodeTable(), at line 7980, the function returns after
>> emitting that warning, but fails to close the fp pointer. I am going to
>> try adding a "(void) fclose(fp);" line to it, rebuild and install and
>> try it out.
>>
>> Cheers!
>> Ben Root
>>
>> On Mon 03 Nov 2014 10:03:49 AM EST, Root, Benjamin wrote:
>>> I had left a long-running process over the weekend to analyze about a
>>> year's worth of hourly RAP data (grib2 format). I would load up a day's
>>> worth using PyNIO, process them, and then close the files before moving
>>> onto the next day. When I got into work this morning, the analysis
>>> errorred out saying that too many files were open.
>>>
>>> I restarted it for a smaller analysis, and ran "lsof" while it executed.
>>> Here is a snippet of its output:
>>>
>>> 3
>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>> python2.7 17092     broot  140r      REG              253,4
>>> 4998    5909053
>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>> python2.7 17092     broot  141r      REG              253,4
>>> 4998    5909053
>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>> python2.7 17092     broot  142r      REG              253,4
>>> 4998    5909053
>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>> python2.7 17092     broot  143r      REG              253,4
>>> 4998    5909053
>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>> python2.7 17092     broot  144r      REG              253,4
>>> 4998    5909053
>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>> python2.7 17092     broot  145r      REG              253,4
>>> 4998    5909053
>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>> python2.7 17092     broot  146r      REG              253,4
>>> 4998    5909053
>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>> python2.7 17092     broot  147r      REG              253,4
>>> 4998    5909053
>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>> python2.7 17092     broot  148r      REG              253,4
>>> 4998    5909053
>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>> python2.7 17092     broot  149r      REG              253,4
>>> 4998    5909053
>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>> python2.7 17092     broot  150r      REG              253,4
>>> 4998    5909053
>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>>
>>> It would seem that closing the grib file doesn't close the associated
>>> grib2 code table. As a side note, is there any reason why the codetable
>>> is not simply cached?
>>>
>>> Over a period of about 5 days worth of RAP files, I ran the lsof
>>> command, counting the number of times the table file appears:
>>>
>>> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
>>> 121
>>> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
>>> 135
>>> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
>>> 159
>>> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
>>> 179
>>> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
>>> 199
>>> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
>>> 217
>>> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
>>> 241
>>>
>>> My version of PyNIO is 1.4.1.
>>>
>>> Cheers!
>>> Ben Root
>>>
>>> ________________________________
>>>
>>> This email is intended solely for the recipient. It may contain privileged, proprietary or confidential information or material. If you are not the intended recipient, please delete this email and any attachments and notify the sender of the error.
>>>
>>> _______________________________________________
>>> pyngl-talk mailing list
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
>>>
>>
>
> _______________________________________________
> pyngl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/pyngl-talk