[pyngl-talk] PyNIO not closing grib codetables?

[Root, Benjamin]

And I just realized that I didn't mention which file I was talking 
about: libsrc/NclGRIB2.c

Ben


On Mon 03 Nov 2014 10:58:33 AM EST, Root, Benjamin wrote:
> Dived into the code, I think I found the problem. The particular RAP
> files I am reading does emit a warning:
> warning: Entry (242) not found in code table file
> /nas/home/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>
> This particular issue doesn't impact me for the data I am dealing with,
> so I ignore it. Well, it seems that in that particular code branch, the
> file pointer does not get closed.
>
> In Grib2ReadCodeTable(), at line 7980, the function returns after
> emitting that warning, but fails to close the fp pointer. I am going to
> try adding a "(void) fclose(fp);" line to it, rebuild and install and
> try it out.
>
> Cheers!
> Ben Root
>
> On Mon 03 Nov 2014 10:03:49 AM EST, Root, Benjamin wrote:
>> I had left a long-running process over the weekend to analyze about a
>> year's worth of hourly RAP data (grib2 format). I would load up a day's
>> worth using PyNIO, process them, and then close the files before moving
>> onto the next day. When I got into work this morning, the analysis
>> errorred out saying that too many files were open.
>>
>> I restarted it for a smaller analysis, and ran "lsof" while it executed.
>> Here is a snippet of its output:
>>
>> 3
>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>> python2.7 17092     broot  140r      REG              253,4
>> 4998    5909053
>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>> python2.7 17092     broot  141r      REG              253,4
>> 4998    5909053
>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>> python2.7 17092     broot  142r      REG              253,4
>> 4998    5909053
>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>> python2.7 17092     broot  143r      REG              253,4
>> 4998    5909053
>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>> python2.7 17092     broot  144r      REG              253,4
>> 4998    5909053
>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>> python2.7 17092     broot  145r      REG              253,4
>> 4998    5909053
>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>> python2.7 17092     broot  146r      REG              253,4
>> 4998    5909053
>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>> python2.7 17092     broot  147r      REG              253,4
>> 4998    5909053
>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>> python2.7 17092     broot  148r      REG              253,4
>> 4998    5909053
>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>> python2.7 17092     broot  149r      REG              253,4
>> 4998    5909053
>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>> python2.7 17092     broot  150r      REG              253,4
>> 4998    5909053
>> /rd22/scratch/broot/centos6/lib/python2.7/site-packages/PyNIO/ncarg/grib2_codetables/ncep/4/4.2.0.1.table
>>
>> It would seem that closing the grib file doesn't close the associated
>> grib2 code table. As a side note, is there any reason why the codetable
>> is not simply cached?
>>
>> Over a period of about 5 days worth of RAP files, I ran the lsof
>> command, counting the number of times the table file appears:
>>
>> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
>> 121
>> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
>> 135
>> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
>> 159
>> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
>> 179
>> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
>> 199
>> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
>> 217
>> [centos6] [broot at rd22 squarepeg]$ lsof | grep table | wc -l
>> 241
>>
>> My version of PyNIO is 1.4.1.
>>
>> Cheers!
>> Ben Root
>>
>> ________________________________
>>
>> This email is intended solely for the recipient. It may contain privileged, proprietary or confidential information or material. If you are not the intended recipient, please delete this email and any attachments and notify the sender of the error.
>>
>> _______________________________________________
>> pyngl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/pyngl-talk
>>
>