[Met_help] [rt.rap.ucar.edu #63440] History for

[John Halley Gotway via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------


Dear MET Help:

I'm reporting a bug in pcp_combine (in function search_pcp_dir) that prevents a number of opened input files from being closed. This bug appears in MET 4.0 and 4.1. This bug can cause a fatal error on Mac OS X where too many files are opened simultaneously when running the test_pcp_combine.sh script.

In src/tools/core/pcp_combine/pcp_combine.cc, lines 671 to near 726 contains code for looping through the files in a directory, opening the files, testing if a given file matches the desired time period, and returning an indicator
if an appropriate file is found. A key component is near line 695, where factory.new_met_2d_data_file is called and returns a Met2dDataFile object (datafile) associated with an opened file. The object includes a destructor
method that will close the associated file when called; however, the destructor is *never* called in this loop. Instead, the pointer is reassigned with each pass through the loop, and once the loop is finished the pointer falls out of scope. 

My proposed fix is to use the delete operator on the datafile pointer near the end of the loop. Specifically, near line 722:

         //  look for a VarInfo record match in the data file
         // EMK DEBUG
         //if( -1 != (i_rec = datafile->index(*var)) ) break;
         if( -1 != (i_rec = datafile->index(*var)) ) {
           delete datafile;
           datafile = (Met2dDataFile *) 0;
           break;
         }

         // EMK DEBUG
         delete datafile;
         datafile = (Met2dDataFile *) 0;

The delete operator will call the destructor of the object being pointed to by datafile, and this will close the file. The datafile pointer is then reset to 0 matching similar code earlier in the function. With this change, the test_pcp_combine.sh script runs correctly.

I suspect the Linux operating system permits a larger number of open files than OS X does, or else this bug would have been found earlier.

System details: I'm running OS X 10.7.5 and using GNU 4.8 compilers.

Regards,

Eric Kemp


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Re: [rt.rap.ucar.edu #63440]
From: John Halley Gotway
Time: Thu Oct 17 16:15:04 2013

Eric,

Thank you very much for bringing this to our attention!  You're right,
it's leaving those file handles open.

I modified your fix slightly (see below) and have posted it as a
bugfix for METv4.0 and METv4.1:
    http://www.dtcenter.org/met/users/support/known_issues/METv4.0/index.php
    http://www.dtcenter.org/met/users/support/known_issues/METv4.1/index.php

Thanks again, and please let us know if any more questions or issues
arise in your use of MET.

Thanks,
John Halley Gotway
met_help at ucar.edu

          //  look for a VarInfo record match in the data file
          i_rec = datafile->index(*var);

          //  delete allocated data file
          if( datafile ) {
             delete datafile;
             datafile = (Met2dDataFile *) 0;
          }

          //  check for a valid match
          if( -1 != i_rec ) break;


On 10/16/2013 12:43 PM, Eric Kemp via RT wrote:
>
> Wed Oct 16 12:43:59 2013: Request 63440 was acted upon.
> Transaction: Ticket created by eric.kemp at ssaihq.com
>         Queue: met_help
>       Subject: (No subject given)
>         Owner: Nobody
>    Requestors: eric.kemp at ssaihq.com
>        Status: new
>   Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=63440 >
>
>
>
> Dear MET Help:
>
> I'm reporting a bug in pcp_combine (in function search_pcp_dir) that
prevents a number of opened input files from being closed. This bug
appears in MET 4.0 and 4.1. This bug can cause a fatal error on Mac OS
X where too many files are opened simultaneously when running the
test_pcp_combine.sh script.
>
> In src/tools/core/pcp_combine/pcp_combine.cc, lines 671 to near 726
contains code for looping through the files in a directory, opening
the files, testing if a given file matches the desired time period,
and returning an indicator
> if an appropriate file is found. A key component is near line 695,
where factory.new_met_2d_data_file is called and returns a
Met2dDataFile object (datafile) associated with an opened file. The
object includes a destructor
> method that will close the associated file when called; however, the
destructor is *never* called in this loop. Instead, the pointer is
reassigned with each pass through the loop, and once the loop is
finished the pointer falls out of scope.
>
> My proposed fix is to use the delete operator on the datafile
pointer near the end of the loop. Specifically, near line 722:
>
>           //  look for a VarInfo record match in the data file
>           // EMK DEBUG
>           //if( -1 != (i_rec = datafile->index(*var)) ) break;
>           if( -1 != (i_rec = datafile->index(*var)) ) {
>             delete datafile;
>             datafile = (Met2dDataFile *) 0;
>             break;
>           }
>
>           // EMK DEBUG
>           delete datafile;
>           datafile = (Met2dDataFile *) 0;
>
> The delete operator will call the destructor of the object being
pointed to by datafile, and this will close the file. The datafile
pointer is then reset to 0 matching similar code earlier in the
function. With this change, the test_pcp_combine.sh script runs
correctly.
>
> I suspect the Linux operating system permits a larger number of open
files than OS X does, or else this bug would have been found earlier.
>
> System details: I'm running OS X 10.7.5 and using GNU 4.8 compilers.
>
> Regards,
>
> Eric Kemp
>

------------------------------------------------