[pyngl-talk] PyNIO not closing grib codetables?
Root, Benjamin
broot at aer.com
Fri Nov 7 11:40:29 MST 2014
So, I can verify that the fix I made does solve the problem for me. I
would be interested in evaluating the next version of PyNIO. We will be
able to see if there are any regressions for the grib/grib2 reader by
trying it out on our products' test servers. Let me know what I need to
do to get a hold of this.
Cheers!
Ben Root
On Mon 03 Nov 2014 12:56:48 PM EST, David Brown wrote:
> Hi Ben,
> Thanks very much for your sleuthing and actually locating the problem.
> In fact, I believe this problem has been fixed in the development
> code, but it has been quite awhile since we released a version of
> PyNIO. We are planning a new release within the next couple of months
> that will have quite a few new features (including support for much
> more of the NetCDF4 specification). There are also many updates to the
> GRIB1 parameter tables and the GRIB2 code tables along with numerous
> bug fixes.
>
> If you would like to get a development version of PyNIO let us know.
> -dave
>
> On Mon, Nov 3, 2014 at 9:13 AM, Root, Benjamin <broot at aer.com> wrote:
>> And I just realized that I didn't mention which file I was talking
>> about: libsrc/NclGRIB2.c
>>
>> Ben
>>
>>
>> On Mon 03 Nov 2014 10:58:33 AM EST, Root, Benjamin wrote:
>>> Dived into the code, I think I found the problem. The particular RAP
>>> files I am reading does emit a warning:
>>> warning: Entry (242) not found in code table file
>>> /nas/home/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>>
>>> This particular issue doesn't impact me for the data I am dealing with,
>>> so I ignore it. Well, it seems that in that particular code branch, the
>>> file pointer does not get closed.
>>>
>>> In Grib2ReadCodeTable(), at line 7980, the function returns after
>>> emitting that warning, but fails to close the fp pointer. I am going to
>>> try adding a "(void) fclose(fp);" line to it, rebuild and install and
>>> try it out.
>>>
>>> Cheers!
>>> Ben Root
>>>
>>> On Mon 03 Nov 2014 10:03:49 AM EST, Root, Benjamin wrote:
>>>> I had left a long-running process over the weekend to analyze about a
>>>> year's worth of hourly RAP data (grib2 format). I would load up a day's
>>>> worth using PyNIO, process them, and then close the files before moving
>>>> onto the next day. When I got into work this morning, the analysis
>>>> errorred out saying that too many files were open.
>>>>
>>>> I restarted it for a smaller analysis, and ran "lsof" while it executed.
>>>> Here is a snippet of its output:
>>>>
>>>> 3
>>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>>> python2.7 17092 broot 140r REG 253,4
>>>> 4998 5909053
>>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>>> python2.7 17092 broot 141r REG 253,4
>>>> 4998 5909053
>>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>>> python2.7 17092 broot 142r REG 253,4
>>>> 4998 5909053
>>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>>> python2.7 17092 broot 143r REG 253,4
>>>> 4998 5909053
>>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>>> python2.7 17092 broot 144r REG 253,4
>>>> 4998 5909053
>>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>>> python2.7 17092 broot 145r REG 253,4
>>>> 4998 5909053
>>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>>> python2.7 17092 broot 146r REG 253,4
>>>> 4998 5909053
>>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>>> python2.7 17092 broot 147r REG 253,4
>>>> 4998 5909053
>>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>>> python2.7 17092 broot 148r REG 253,4
>>>> 4998 5909053
>>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>>> python2.7 17092 broot 149r REG 253,4
>>>> 4998 5909053
>>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>>> python2.7 17092 broot 150r REG 253,4
>>>> 4998 5909053
>>>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>>>
>>>> It would seem that closing the grib file doesn't close the associated
>>>> grib2 code table. As a side note, is there any reason why the codetable
>>>> is not simply cached?
>>>>
>>>> Over a period of about 5 days worth of RAP files, I ran the lsof
>>>> command, counting the number of times the table file appears:
>>>>
>>>> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
>>>> 121
>>>> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
>>>> 135
>>>> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
>>>> 159
>>>> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
>>>> 179
>>>> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
>>>> 199
>>>> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
>>>> 217
>>>> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
>>>> 241
>>>>
>>>> My version of PyNIO is 1.4.1.
>>>>
>>>> Cheers!
>>>> Ben Root
>>>>
>>>> ________________________________
>>>>
>>>> This email is intended solely for the recipient. It may contain privileged, proprietary or confidential information or material. If you are not the intended recipient, please delete this email and any attachments and notify the sender of the error.
>>>>
>>>> _______________________________________________
>>>> pyngl-talk mailing list
>>>> List instructions, subscriber options, unsubscribe:
>>>> http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
>>>>
>>>
>>
>> _______________________________________________
>> pyngl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
>
More information about the pyngl-talk
mailing list