[Met_help] met and conditional verification across variables

Tue Apr 13 08:09:57 MDT 2010

Works with -add, but still not with -sum.  The problem is with similar idiosyncrasies with the files as were found with some TR values of 0 I'd dealt with before, even though each file is a 3 hour precip accumulation.  It'd be nice if I could circumvent some of the checks for these and just sum the APCP (code 61) values for all the files in the directory given, or using the date in the files.  I'm just going in and making some modifications to the code to skip the checks and process that way and doing what I can to check that the sums make sense.  Easier than modifying the grib files (I think).  If you have another idea beyond going in and modifying all the grib files to work with MET, feel free to let me know!

Thanks,
Mike

-----Original Message-----
From: John Halley Gotway [mailto:johnhg at ucar.edu] 
Sent: Monday, April 12, 2010 2:59 PM
To: Shaw, Michael J CTR USAF AFWA 16 WS/WXE
Cc: 'met_help'
Subject: Re: [Met_help] met and conditional verification across variables

Mike,

I'd suggest trying to run a different pcp_combine command to narrow in on where the problem actually lies.  For example, try using the "-add" command:
   pcp_combine -add file1 3 file2 3 test_add.nc

Where file1 and file2 are the paths to two files containing 3-hourly accumulations.  If you get the same error when you run this, there's likely a problem in the GRIB file formatting.  And if that's
the case, feel free to post a couple of sample files to our anonymous ftp site.  I'd be happy to run pcp-combine through the debugger and try to figure out where the problem is.  You can post to our
ftp site as follows:

ftp ftp.rap.ucar.edu
username=anonymous
password="your email address"
cd incoming/irap/met_help/shaw_data
put "your files"
bye

Thanks,
John

Shaw, Michael J CTR USAF AFWA 16 WS/WXE wrote:
> Thanks, John.
> 
> Got the exact same error with the line you suggested below.  Thanks for the notice of the missing output file in what I'd put in that email.  I'm going to try without the pcprx part and see how that goes.  The 0p25 was the extension of the files in the pcpdir, of course....
> 
> Any other thoughts welcome in the mean time.
> 
> -Mike
> 
> -----Original Message-----
> From: John Halley Gotway [mailto:johnhg at ucar.edu] 
> Sent: Monday, April 12, 2010 2:06 PM
> To: Shaw, Michael J CTR USAF AFWA 16 WS/WXE
> Cc: met_help
> Subject: Re: [Met_help] met and conditional verification across variables
> 
> Mike,
> 
> I think it may be a problem with how you're calling PCP-Combine.  Listed below is the usage statement for PCP-Combine, when summing up files:
> 
>         SUM_ARGS:
>                 init_time
>                 in_accum
>                 valid_time
>                 out_accum
>                 out_file
>                 [-pcpdir path]
>                 [-pcprx reg_exp]
> 
>                 where   "init_time" indicates the initialization time of the input Grib files in YYYYMMDD_HHMMSS format (required).
>                         "in_accum" indicates the accumulation interval of the input Grib files in HH format (required).
>                         "valid_time" indicates the desired valid time in YYYYMMDD_HHMMSS format (required).
>                         "out_accum" indicates the desired accumulation interval for the output NetCDF file in HH format (required).
>                         "out_file" indicates the name of the output NetCDF file to be written consisting of the sum of the accumulation intervals (required).
>                         "-pcpdir path" overrides the default precipitation directory (/d1/johnhg/MET/MET_releases/METv2.0_patch/data/sample_obs/ST2ml) (optional).
>                         "-pcprx reg_exp" overrides the default regular expression for precipitation file naming convention (.*) (optional).
> 
> And here's the command line you're running:
>  pcp_combine -sum 00000000_000000 3 20100318_000000 168 -pcpdir <the dir> -pcprx *0p25
> 
> It looks like you've failed to list an output file name to be used in the correct place in the command line.  Try changing it something like:
>  pcp_combine -sum 00000000_000000 3 20100318_000000 168 test_output.nc -pcpdir <the dir> -pcprx 0p25
> 
> Also notice that I've removed the '*' from the -pcprx argument.  Just using '0p25' should do what you want.  Otherwise, the shell will expand that '*' on the command line and you'll get unexpected
> behavior.
> 
> Let me know if that helps.
> 
> Thanks,
> John
> 
> Shaw, Michael J CTR USAF AFWA 16 WS/WXE wrote:
>> Hi John.
>>
>> Trying to accumulate precip of a bunch of agrmet grib files.  Here's what I get:
>>
>> pcp_combine -sum 00000000_000000 3 20100318_000000 168 -pcpdir <the dir> -pcprx *0p25
>>
>> [1]Reading input file:
>>
>>    GribFile::open(const char *) -> unable to open grib file
>>
>> While I look around for the culprit and fix, maybe you can help me before I get there.  Of course, these are the airforce grib files...
>>
>> Thanks,
>> Mike
>>
>> -----Original Message-----
>> From: John Halley Gotway [mailto:johnhg at ucar.edu] 
>> Sent: Tuesday, March 23, 2010 1:14 PM
>> To: Shaw, Michael J CTR USAF AFWA 16 WS/WXE; met_help
>> Subject: Re: [Met_help] met and conditional verification across variables
>>
>> Mike,
>>
>> Great, thanks.  Please send me the changes when you get a chance, and I'll look into getting them incorporated into the next version of MET.
>>
>> John
>>
>> Shaw, Michael J CTR USAF AFWA 16 WS/WXE wrote:
>>> Hi John.
>>>
>>> I think I was able to make the mods I needed and rebuild MET.  I'm out
>>> of town for much of the week, but I can pass those on to you early next
>>> week if you want.
>>>
>>> Thanks,
>>> Mike
>>>
>>>
>>> -----Original Message-----
>>> From: John Halley Gotway [mailto:johnhg at rap.ucar.edu] 
>>> Sent: Friday, March 19, 2010 10:23 PM
>>> To: Shaw, Michael J CTR USAF AFWA 16 WS/WXE
>>> Cc: met_help at ucar.edu
>>> Subject: Re: [Met_help] met and conditional verification across
>>> variables
>>>
>>> Mike,
>>>
>>> You may be able to do what you'd like to do using what we've called
>>> "data
>>> masking".  Listed below is an excerpt from the Point-Stat configuration
>>> file describing the "mask_poly" option:
>>>
>>> //
>>> // Specify a comma-separated list of masking regions to be applied.
>>> // An empty list indicates that no additional masks should be used.
>>> // The masking regions may be defined in one of 4 ways:
>>> //
>>> // (1) An ASCII file containing a lat/lon polygon.
>>> //     Latitude in degrees north and longitude in degrees east.
>>> //     By default, the first and last polygon points are connected.
>>> //     e.g. "MET_BASE/data/poly/EAST.poly" which consists of n points:
>>> //          "poly_name lat1 lon1 lat2 lon2... latn lonn"
>>> //
>>> // (2) The NetCDF output of the gen_poly_mask tool.
>>> //
>>> // (3) A NetCDF data file, followed by the name of the NetCDF variable
>>> //     to be used, and optionally, a threshold to be applied to the
>>> field.
>>> //     e.g. "sample.nc var_name gt0.00"
>>> //
>>> // (4) A GRIB data file, followed by a description of the field
>>> //     to be used, and optionally, a threshold to be applied to the
>>> field.
>>> //     e.g. "sample.grb APCP/A3 gt0.00"
>>> //
>>> // Any NetCDF or GRIB file used must have the same grid dimensions as
>>> the
>>> // data being verified.
>>> //
>>> // MET_BASE may be used in the path for the files above.
>>> //
>>> // e.g. mask_poly[] = [ "MET_BASE/data/poly/EAST.poly",
>>> //                      "poly_mask.ncf",
>>> //                      "sample.nc APCP",
>>> //                      "sample.grb HGT/Z0 gt100.0" ];
>>> //
>>> mask_poly[] = [ "MET_BASE/data/poly/EAST.poly" ];
>>>
>>> Option (4) listed above may do what you'd like.  You'd need the land use
>>> field on the same grid as the data you're verifying.  Basically, you'd
>>> specify the GRIB file you'd like to use, the field you want (land use in
>>> your case), and then the threshold you'd like applied to that field -
>>> and
>>> you could specify that threshold as "eq0" or "eq1" - or whatever the
>>> land
>>> use values are.  That threshold defines the data mask that's applied to
>>> the data.  You may specify as many of those masking regions as you'd
>>> like
>>> - probably one per land use category you'd like to verify.
>>>
>>> Hope that helps.
>>>
>>> John Halley Gotway
>>>
>>>
>>>> Hi.
>>>>
>>>>
>>>>
>>>> I'm interested in verifying conditionally;
>>> if(landuse==suchandsuch)then
>>>> "how does surface temperature verify across cases and against
>>>> observations/retrievals?" kind of thing.  Basically, doing what MET
>>> does
>>>> with thresholds, but applying the threshold to other variables
>>> (preferably
>>>> multiple - if(x<landuse<y)elseif(y<landuse<z) for instance) with a
>>> grib
>>>> file, e.g..  Not finding that in MET: Am I missing something or is
>>> that
>>>> something that's planned for a (near-)future release?
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Mike
>>>>
>>>>
>>>>
>>>> ______________________________________________
>>>>
>>>> Michael Shaw, Contractor
>>>>
>>>> SAIC/NASA Support Scientist
>>>>
>>>> michael.shaw.ctr at offutt.af.mil
>>>>
>>>> 16WS Environmental Characterization
>>>>
>>>> HQ AFWA-SAIC
>>>>
>>>> 101 Nelson Drive
>>>>
>>>> Offutt AFB, NE 68113-1023
>>>>
>>>> 402-232-7690 Comm * 402-272-7690 DSN * 402-294-8230 Fax
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Met_help mailing list
>>>> Met_help at mailman.ucar.edu
>>>> http://mailman.ucar.edu/mailman/listinfo/met_help
>>>>