[Met_help] [rt.rap.ucar.edu #91563] History for FW: Stat Anal Filtering
John Halley Gotway via RT
met_help at ucar.edu
Wed Aug 21 16:52:48 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
Sent: Friday, August 16, 2019 3:04 PM
To: met_help at ucar.edu
Subject: Stat Anal Filtering
John, I am having trouble trying to filter a stat file using the following
command line:
/h/MET/bin/stat_analysis -lookin
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/test -job
filter -interp_mthd NBRHD_SQUARE -line_type MPR -mask_sid /h/data/global/
WXQC/data/met/sid/baltic1p5km.txt -out_stat
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
I also tried:
/h/MET/bin/stat_analysis -lookin
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/test -job
filter -line_type MPR -column_str OBS_SID /h/data/global/WXQC/data/met/si
d/baltic1p5km.txt -dump_row
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
The goal is to reduce the stat file to just those mpr lines that contain
station ids in the file baltic1p5km.txt and
The data file I am trying to filter is attached. I have tried lots of
combinations of things from the users manual to filter the data but keep
getting the following error:
DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
/h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
/h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
DEBUG 1: Default Config File:
/h/MET/share/met/config/STATAnalysisConfig_default
DEBUG 1: Station ID Mask: /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162 points
ERROR :
" in job command line: -job filter -interp_mthd NBRHD_SQUARE -line_type MPR
-mask_sid /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -out_stat
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
ERROR :
ERROR :
ERROR : main() -> encountered an error value of 1. Calling clean_up() and
usage() before exiting.
What I am doing wrong?
Thanks
Bob
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: FW: Stat Anal Filtering
From: John Halley Gotway
Time: Fri Aug 16 14:52:33 2019
Bob,
You should also be seeing this error message in the output:
ERROR : do_job_filter() -> this function may only be called when
using the*
-dump_row *option in the job command line: -job filter -line_type MPR
-mask_sid baltic1p5km.txt
The output for the "filter" job is specified by the "-dump_row"
option, not
the "-out_stat" option.
Please try running:
*/h/MET/bin/stat_analysis \*
*-lookin
/h/data/global/WXQC/data/met/p**tstat/mdlob_pairs/global_17km/**CB/test
\*
*-job filter -interp_mthd NBRHD_SQUARE -line_type MPR \*
*-mask_sid /h/data/global/WXQC/data/met/sid/baltic1p5km.**txt \*
*-dump_row
/h/data/global/WXQC/data/met/s**ummary/CB_global_17km_le3000_d*
*ump.stat*
Does that work?
Thanks,
John
On Fri, Aug 16, 2019 at 2:43 PM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Fri Aug 16 14:43:07 2019: Request 91563 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: FW: Stat Anal Filtering
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563 >
>
>
>
>
>
>
> From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> Sent: Friday, August 16, 2019 3:04 PM
> To: met_help at ucar.edu
> Subject: Stat Anal Filtering
>
>
>
> John, I am having trouble trying to filter a stat file using the
following
> command line:
>
>
>
> /h/MET/bin/stat_analysis -lookin
> /h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/test
-job
> filter -interp_mthd NBRHD_SQUARE -line_type MPR -mask_sid
/h/data/global/
> WXQC/data/met/sid/baltic1p5km.txt -out_stat
> /h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
>
>
>
> I also tried:
>
>
>
> /h/MET/bin/stat_analysis -lookin
> /h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/test
-job
> filter -line_type MPR -column_str OBS_SID
/h/data/global/WXQC/data/met/si
> d/baltic1p5km.txt -dump_row
> /h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
>
>
>
> The goal is to reduce the stat file to just those mpr lines that
contain
> station ids in the file baltic1p5km.txt and
>
> The data file I am trying to filter is attached. I have tried lots
of
> combinations of things from the users manual to filter the data but
keep
> getting the following error:
>
>
>
> DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
>
> DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
>
> DEBUG 1: Default Config File:
> /h/MET/share/met/config/STATAnalysisConfig_default
>
> DEBUG 1: Station ID Mask:
/h/data/global/WXQC/data/met/sid/baltic1p5km.txt
>
> DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162 points
>
> ERROR :
>
> " in job command line: -job filter -interp_mthd NBRHD_SQUARE
-line_type MPR
> -mask_sid /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -out_stat
> /h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
>
> ERROR :
>
> ERROR :
>
> ERROR : main() -> encountered an error value of 1. Calling
clean_up() and
> usage() before exiting.
>
>
>
> What I am doing wrong?
>
>
>
> Thanks
>
> Bob
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat Anal Filtering
From: robert.craig.2 at us.af.mil
Time: Fri Aug 16 15:00:04 2019
John, that gives me the error:
DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
/h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
/h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
DEBUG 1: Default Config File:
/h/MET/share/met/config/STATAnalysisConfig_default
DEBUG 1: Station ID Mask:
/h/data/global/WXQC/data/met/sid/baltic1p5km.txt
DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162 points
ERROR :
" in job command line: -job filter -interp_mthd NBRHD_SQUARE
-line_type MPR -mask_sid
/h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
ERROR :
ERROR :
ERROR : main() -> encountered an error value of 1. Calling
clean_up() and usage() before exiting.
ERROR :
I sent you the data file via amerdec if you would like to try it.
Thanks
Bob
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Friday, August 16, 2019 3:53 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat Anal
Filtering
Bob,
You should also be seeing this error message in the output:
ERROR : do_job_filter() -> this function may only be called when
using the* -dump_row *option in the job command line: -job filter
-line_type MPR -mask_sid baltic1p5km.txt
The output for the "filter" job is specified by the "-dump_row"
option, not the "-out_stat" option.
Please try running:
*/h/MET/bin/stat_analysis \*
*-lookin
/h/data/global/WXQC/data/met/p**tstat/mdlob_pairs/global_17km/**CB/test
\*
*-job filter -interp_mthd NBRHD_SQUARE -line_type MPR \* *-mask_sid
/h/data/global/WXQC/data/met/sid/baltic1p5km.**txt \* *-dump_row
/h/data/global/WXQC/data/met/s**ummary/CB_global_17km_le3000_d*
*ump.stat*
Does that work?
Thanks,
John
On Fri, Aug 16, 2019 at 2:43 PM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Fri Aug 16 14:43:07 2019: Request 91563 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: FW: Stat Anal Filtering
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563
> >
>
>
>
>
>
>
> From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> Sent: Friday, August 16, 2019 3:04 PM
> To: met_help at ucar.edu
> Subject: Stat Anal Filtering
>
>
>
> John, I am having trouble trying to filter a stat file using the
> following command line:
>
>
>
> /h/MET/bin/stat_analysis -lookin
> /h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/test
> -job filter -interp_mthd NBRHD_SQUARE -line_type MPR -mask_sid
> /h/data/global/ WXQC/data/met/sid/baltic1p5km.txt -out_stat
> /h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
>
>
>
> I also tried:
>
>
>
> /h/MET/bin/stat_analysis -lookin
> /h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/test
> -job filter -line_type MPR -column_str OBS_SID
> /h/data/global/WXQC/data/met/si d/baltic1p5km.txt -dump_row
> /h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
>
>
>
> The goal is to reduce the stat file to just those mpr lines that
> contain station ids in the file baltic1p5km.txt and
>
> The data file I am trying to filter is attached. I have tried lots
of
> combinations of things from the users manual to filter the data but
> keep getting the following error:
>
>
>
> DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
>
> DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
>
> DEBUG 1: Default Config File:
> /h/MET/share/met/config/STATAnalysisConfig_default
>
> DEBUG 1: Station ID Mask:
> /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
>
> DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162 points
>
> ERROR :
>
> " in job command line: -job filter -interp_mthd NBRHD_SQUARE
> -line_type MPR -mask_sid
> /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -out_stat
> /h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
>
> ERROR :
>
> ERROR :
>
> ERROR : main() -> encountered an error value of 1. Calling
> clean_up() and
> usage() before exiting.
>
>
>
> What I am doing wrong?
>
>
>
> Thanks
>
> Bob
>
>
>
------------------------------------------------
Subject: FW: Stat Anal Filtering
From: John Halley Gotway
Time: Fri Aug 16 16:19:29 2019
Bob,
I never saw an email come through from amerdec. So to test, I used
some
sample data that's generated when you run 'make test'.
*cd met-8.1*
*bin/stat_analysis -lookin
out/point_stat/point_stat_360000L_20070331_120000V.stat \*
*-job filter -line_type MPR -interp_mthd NEAREST -dump_row
filter.stat*
DEBUG 2: STAT Lines read = 28140
DEBUG 2: STAT Lines retained = 9288
And then I picked out the first 500 unique station id names from the
MRP
lines and wrote them to a file:
*echo "SID_MASK" > my_sid_list.txt*
*grep MPR out/point_stat/point_stat_360000L_20070331_120000V.stat |
awk
'{print $27}' | sort -u | head -500 >> my_sid_list.txt*
And then I reran using that station ID list:
*bin/stat_analysis -lookin
out/point_stat/point_stat_360000L_20070331_120000V.stat -job filter
-line_type MPR -interp_mthd NEAREST -dump_row filter.stat -mask_sid
my_sid_list.txt*
DEBUG 2: STAT Lines read = 28140
DEBUG 2: STAT Lines retained = 1451
The 9288 MPR lines for NEAREST were reduced down to 1451. So this
functionality *should* be working. But I'm not sure where the
disconnect
is.
Thanks,
John
On Fri, Aug 16, 2019 at 3:00 PM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563 >
>
> John, that gives me the error:
>
> DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> DEBUG 1: Default Config File:
> /h/MET/share/met/config/STATAnalysisConfig_default
> DEBUG 1: Station ID Mask:
/h/data/global/WXQC/data/met/sid/baltic1p5km.txt
> DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162 points
> ERROR :
> " in job command line: -job filter -interp_mthd NBRHD_SQUARE
-line_type
> MPR -mask_sid /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
-dump_row
> /h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> ERROR :
> ERROR :
> ERROR : main() -> encountered an error value of 1. Calling
clean_up()
> and usage() before exiting.
> ERROR :
>
> I sent you the data file via amerdec if you would like to try it.
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Friday, August 16, 2019 3:53 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat Anal
> Filtering
>
> Bob,
>
> You should also be seeing this error message in the output:
>
> ERROR : do_job_filter() -> this function may only be called when
using
> the* -dump_row *option in the job command line: -job filter
-line_type MPR
> -mask_sid baltic1p5km.txt
>
> The output for the "filter" job is specified by the "-dump_row"
option,
> not the "-out_stat" option.
>
> Please try running:
>
> */h/MET/bin/stat_analysis \*
> *-lookin
>
/h/data/global/WXQC/data/met/p**tstat/mdlob_pairs/global_17km/**CB/test
> \*
> *-job filter -interp_mthd NBRHD_SQUARE -line_type MPR \* *-mask_sid
> /h/data/global/WXQC/data/met/sid/baltic1p5km.**txt \* *-dump_row
> /h/data/global/WXQC/data/met/s**ummary/CB_global_17km_le3000_d*
> *ump.stat*
>
> Does that work?
>
> Thanks,
> John
>
> On Fri, Aug 16, 2019 at 2:43 PM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Fri Aug 16 14:43:07 2019: Request 91563 was acted upon.
> > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > Queue: met_help
> > Subject: FW: Stat Anal Filtering
> > Owner: Nobody
> > Requestors: robert.craig.2 at us.af.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563
> > >
> >
> >
> >
> >
> >
> >
> > From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > Sent: Friday, August 16, 2019 3:04 PM
> > To: met_help at ucar.edu
> > Subject: Stat Anal Filtering
> >
> >
> >
> > John, I am having trouble trying to filter a stat file using the
> > following command line:
> >
> >
> >
> > /h/MET/bin/stat_analysis -lookin
> >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/test
> > -job filter -interp_mthd NBRHD_SQUARE -line_type MPR -mask_sid
> > /h/data/global/ WXQC/data/met/sid/baltic1p5km.txt -out_stat
> >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> >
> >
> >
> > I also tried:
> >
> >
> >
> > /h/MET/bin/stat_analysis -lookin
> >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/test
> > -job filter -line_type MPR -column_str OBS_SID
> > /h/data/global/WXQC/data/met/si d/baltic1p5km.txt -dump_row
> >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> >
> >
> >
> > The goal is to reduce the stat file to just those mpr lines that
> > contain station ids in the file baltic1p5km.txt and
> >
> > The data file I am trying to filter is attached. I have tried
lots of
> > combinations of things from the users manual to filter the data
but
> > keep getting the following error:
> >
> >
> >
> > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> > /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> >
> > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> >
> > DEBUG 1: Default Config File:
> > /h/MET/share/met/config/STATAnalysisConfig_default
> >
> > DEBUG 1: Station ID Mask:
> > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
> >
> > DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162 points
> >
> > ERROR :
> >
> > " in job command line: -job filter -interp_mthd NBRHD_SQUARE
> > -line_type MPR -mask_sid
> > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -out_stat
> >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> >
> > ERROR :
> >
> > ERROR :
> >
> > ERROR : main() -> encountered an error value of 1. Calling
> > clean_up() and
> > usage() before exiting.
> >
> >
> >
> > What I am doing wrong?
> >
> >
> >
> > Thanks
> >
> > Bob
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat Anal Filtering
From: robert.craig.2 at us.af.mil
Time: Mon Aug 19 07:05:14 2019
John, it has got to be something in my data file that MET doesn't
like. I resent the file and it should make it this time. This was a
file created using the HIRA option.
Maybe you could enhance the error message to give more detail on what
caused the error.
Thanks
Bob
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Friday, August 16, 2019 5:19 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat
Anal Filtering
Bob,
I never saw an email come through from amerdec. So to test, I used
some sample data that's generated when you run 'make test'.
*cd met-8.1*
*bin/stat_analysis -lookin
out/point_stat/point_stat_360000L_20070331_120000V.stat \* *-job
filter -line_type MPR -interp_mthd NEAREST -dump_row filter.stat*
DEBUG 2: STAT Lines read = 28140
DEBUG 2: STAT Lines retained = 9288
And then I picked out the first 500 unique station id names from the
MRP lines and wrote them to a file:
*echo "SID_MASK" > my_sid_list.txt*
*grep MPR out/point_stat/point_stat_360000L_20070331_120000V.stat |
awk '{print $27}' | sort -u | head -500 >> my_sid_list.txt*
And then I reran using that station ID list:
*bin/stat_analysis -lookin
out/point_stat/point_stat_360000L_20070331_120000V.stat -job filter
-line_type MPR -interp_mthd NEAREST -dump_row filter.stat -mask_sid
my_sid_list.txt*
DEBUG 2: STAT Lines read = 28140
DEBUG 2: STAT Lines retained = 1451
The 9288 MPR lines for NEAREST were reduced down to 1451. So this
functionality *should* be working. But I'm not sure where the
disconnect is.
Thanks,
John
On Fri, Aug 16, 2019 at 3:00 PM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563 >
>
> John, that gives me the error:
>
> DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> DEBUG 1: Default Config File:
> /h/MET/share/met/config/STATAnalysisConfig_default
> DEBUG 1: Station ID Mask:
> /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
> DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162 points
ERROR
> :
> " in job command line: -job filter -interp_mthd NBRHD_SQUARE
> -line_type MPR -mask_sid
> /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
> /h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> ERROR :
> ERROR :
> ERROR : main() -> encountered an error value of 1. Calling
> clean_up() and usage() before exiting.
> ERROR :
>
> I sent you the data file via amerdec if you would like to try it.
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Friday, August 16, 2019 3:53 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> <robert.craig.2 at us.af.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat Anal
> Filtering
>
> Bob,
>
> You should also be seeing this error message in the output:
>
> ERROR : do_job_filter() -> this function may only be called when
> using
> the* -dump_row *option in the job command line: -job filter
-line_type
> MPR -mask_sid baltic1p5km.txt
>
> The output for the "filter" job is specified by the "-dump_row"
> option, not the "-out_stat" option.
>
> Please try running:
>
> */h/MET/bin/stat_analysis \*
> *-lookin
>
/h/data/global/WXQC/data/met/p**tstat/mdlob_pairs/global_17km/**CB/tes
> t
> \*
> *-job filter -interp_mthd NBRHD_SQUARE -line_type MPR \* *-mask_sid
> /h/data/global/WXQC/data/met/sid/baltic1p5km.**txt \* *-dump_row
> /h/data/global/WXQC/data/met/s**ummary/CB_global_17km_le3000_d*
> *ump.stat*
>
> Does that work?
>
> Thanks,
> John
>
> On Fri, Aug 16, 2019 at 2:43 PM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Fri Aug 16 14:43:07 2019: Request 91563 was acted upon.
> > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > Queue: met_help
> > Subject: FW: Stat Anal Filtering
> > Owner: Nobody
> > Requestors: robert.craig.2 at us.af.mil
> > Status: new
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563
> > >
> >
> >
> >
> >
> >
> >
> > From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > Sent: Friday, August 16, 2019 3:04 PM
> > To: met_help at ucar.edu
> > Subject: Stat Anal Filtering
> >
> >
> >
> > John, I am having trouble trying to filter a stat file using the
> > following command line:
> >
> >
> >
> > /h/MET/bin/stat_analysis -lookin
> >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/test
> > -job filter -interp_mthd NBRHD_SQUARE -line_type MPR -mask_sid
> > /h/data/global/ WXQC/data/met/sid/baltic1p5km.txt -out_stat
> >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> >
> >
> >
> > I also tried:
> >
> >
> >
> > /h/MET/bin/stat_analysis -lookin
> >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/test
> > -job filter -line_type MPR -column_str OBS_SID
> > /h/data/global/WXQC/data/met/si d/baltic1p5km.txt -dump_row
> >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> >
> >
> >
> > The goal is to reduce the stat file to just those mpr lines that
> > contain station ids in the file baltic1p5km.txt and
> >
> > The data file I am trying to filter is attached. I have tried
lots
> > of combinations of things from the users manual to filter the data
> > but keep getting the following error:
> >
> >
> >
> > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> > /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> >
> > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> >
> > DEBUG 1: Default Config File:
> > /h/MET/share/met/config/STATAnalysisConfig_default
> >
> > DEBUG 1: Station ID Mask:
> > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
> >
> > DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162 points
> >
> > ERROR :
> >
> > " in job command line: -job filter -interp_mthd NBRHD_SQUARE
> > -line_type MPR -mask_sid
> > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -out_stat
> >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> >
> > ERROR :
> >
> > ERROR :
> >
> > ERROR : main() -> encountered an error value of 1. Calling
> > clean_up() and
> > usage() before exiting.
> >
> >
> >
> > What I am doing wrong?
> >
> >
> >
> > Thanks
> >
> > Bob
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat Anal Filtering
From: robert.craig.2 at us.af.mil
Time: Mon Aug 19 09:01:11 2019
John, a related question. In stat_analysis I would like to exclude
obs from being considered - is there a command switch for that? I
tried -sid_exc but that didn't work.
Thanks
Bob
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Friday, August 16, 2019 5:19 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat
Anal Filtering
Bob,
I never saw an email come through from amerdec. So to test, I used
some sample data that's generated when you run 'make test'.
*cd met-8.1*
*bin/stat_analysis -lookin
out/point_stat/point_stat_360000L_20070331_120000V.stat \* *-job
filter -line_type MPR -interp_mthd NEAREST -dump_row filter.stat*
DEBUG 2: STAT Lines read = 28140
DEBUG 2: STAT Lines retained = 9288
And then I picked out the first 500 unique station id names from the
MRP lines and wrote them to a file:
*echo "SID_MASK" > my_sid_list.txt*
*grep MPR out/point_stat/point_stat_360000L_20070331_120000V.stat |
awk '{print $27}' | sort -u | head -500 >> my_sid_list.txt*
And then I reran using that station ID list:
*bin/stat_analysis -lookin
out/point_stat/point_stat_360000L_20070331_120000V.stat -job filter
-line_type MPR -interp_mthd NEAREST -dump_row filter.stat -mask_sid
my_sid_list.txt*
DEBUG 2: STAT Lines read = 28140
DEBUG 2: STAT Lines retained = 1451
The 9288 MPR lines for NEAREST were reduced down to 1451. So this
functionality *should* be working. But I'm not sure where the
disconnect is.
Thanks,
John
On Fri, Aug 16, 2019 at 3:00 PM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563 >
>
> John, that gives me the error:
>
> DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> DEBUG 1: Default Config File:
> /h/MET/share/met/config/STATAnalysisConfig_default
> DEBUG 1: Station ID Mask:
> /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
> DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162 points
ERROR
> :
> " in job command line: -job filter -interp_mthd NBRHD_SQUARE
> -line_type MPR -mask_sid
> /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
> /h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> ERROR :
> ERROR :
> ERROR : main() -> encountered an error value of 1. Calling
> clean_up() and usage() before exiting.
> ERROR :
>
> I sent you the data file via amerdec if you would like to try it.
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Friday, August 16, 2019 3:53 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> <robert.craig.2 at us.af.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat Anal
> Filtering
>
> Bob,
>
> You should also be seeing this error message in the output:
>
> ERROR : do_job_filter() -> this function may only be called when
> using
> the* -dump_row *option in the job command line: -job filter
-line_type
> MPR -mask_sid baltic1p5km.txt
>
> The output for the "filter" job is specified by the "-dump_row"
> option, not the "-out_stat" option.
>
> Please try running:
>
> */h/MET/bin/stat_analysis \*
> *-lookin
>
/h/data/global/WXQC/data/met/p**tstat/mdlob_pairs/global_17km/**CB/tes
> t
> \*
> *-job filter -interp_mthd NBRHD_SQUARE -line_type MPR \* *-mask_sid
> /h/data/global/WXQC/data/met/sid/baltic1p5km.**txt \* *-dump_row
> /h/data/global/WXQC/data/met/s**ummary/CB_global_17km_le3000_d*
> *ump.stat*
>
> Does that work?
>
> Thanks,
> John
>
> On Fri, Aug 16, 2019 at 2:43 PM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Fri Aug 16 14:43:07 2019: Request 91563 was acted upon.
> > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > Queue: met_help
> > Subject: FW: Stat Anal Filtering
> > Owner: Nobody
> > Requestors: robert.craig.2 at us.af.mil
> > Status: new
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563
> > >
> >
> >
> >
> >
> >
> >
> > From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > Sent: Friday, August 16, 2019 3:04 PM
> > To: met_help at ucar.edu
> > Subject: Stat Anal Filtering
> >
> >
> >
> > John, I am having trouble trying to filter a stat file using the
> > following command line:
> >
> >
> >
> > /h/MET/bin/stat_analysis -lookin
> >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/test
> > -job filter -interp_mthd NBRHD_SQUARE -line_type MPR -mask_sid
> > /h/data/global/ WXQC/data/met/sid/baltic1p5km.txt -out_stat
> >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> >
> >
> >
> > I also tried:
> >
> >
> >
> > /h/MET/bin/stat_analysis -lookin
> >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/test
> > -job filter -line_type MPR -column_str OBS_SID
> > /h/data/global/WXQC/data/met/si d/baltic1p5km.txt -dump_row
> >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> >
> >
> >
> > The goal is to reduce the stat file to just those mpr lines that
> > contain station ids in the file baltic1p5km.txt and
> >
> > The data file I am trying to filter is attached. I have tried
lots
> > of combinations of things from the users manual to filter the data
> > but keep getting the following error:
> >
> >
> >
> > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> > /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> >
> > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> >
> > DEBUG 1: Default Config File:
> > /h/MET/share/met/config/STATAnalysisConfig_default
> >
> > DEBUG 1: Station ID Mask:
> > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
> >
> > DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162 points
> >
> > ERROR :
> >
> > " in job command line: -job filter -interp_mthd NBRHD_SQUARE
> > -line_type MPR -mask_sid
> > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -out_stat
> >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> >
> > ERROR :
> >
> > ERROR :
> >
> > ERROR : main() -> encountered an error value of 1. Calling
> > clean_up() and
> > usage() before exiting.
> >
> >
> >
> > What I am doing wrong?
> >
> >
> >
> > Thanks
> >
> > Bob
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: FW: Stat Anal Filtering
From: John Halley Gotway
Time: Mon Aug 19 09:57:18 2019
Bob,
Still no luck. I'm not seeing anything from ARMDEC coming to my email
address "johnhg at ucar.edu".
One big mystery is the error message you sent to me:
ERROR :
" in job command line: -job filter -interp_mthd NBRHD_SQUARE
-line_type MPR
-mask_sid /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
ERROR :
The whole first part of it is truncated. There should be an actual
message
before it says " in job command...". So why/how is that happening?
One
additional thing to consider... you use Windows and that inserts a ^M
character at the end of every line of text as the line break marker.
In
the past, those ^M characters have caused trouble. I'm not sure they
are
here, but it's always a possibility. You could try running that ascii
file
through the dos2unix utility which strips off those ^M characters:
dos2unix baltic1p5km.txt
And then see if anything changes.
For your second question, no, there currently isn't any way to tell
STAT-Analysis which stations to skip. The closest thing is the
"-column_str" option.
For example, "-column MPR -column_str OBS_SID KDEN,690150" tells
STAT-Analysis to only use lines MPR lines where the OBS_SID column
contains
"KDEN" or "690150". But there is no negation for this option. We
can't
tell it which strings to exclude.
For numeric values, this is easy. The "-column_thresh" option defines
which numbers to keep. For example, "-column MPR -column_thresh FCST
gt100" keeps only MPR lines where the FCST value is > 100. Negating
this
is easy... just switch the threshold "-column_thresh FCST le100".
We could potentially enhance stat_analysis (and tc_stat) to support
this in
the future. I see a few implementation options...
(1) Explicitly add a "-sid_exc" option which only works when
processing MPR
or ORANK lines.
(2) Add a new "-column_str_exc" option to do the opposite of the
existing
"-column_str" option. This is generally applied to any column
containing
strings.
(3) Just use the existing "-column_str" option but switch to
interpreting
the strings as regular expressions, and define the negation as a
regular
expression.
I think I lean toward option (2). While (3) is nice, regular
expressions
can be difficult to set up well and produce unintended matches.
Please let
me know if you'd like me to write up this as a potential GitHub
development
issue.
For now, I'd recommend just running egrep prior to running stat-
analysis to
throw out the stations you don't want. For example:
egrep -v " 690150 | 710630 "
point_stat_meps_conus_480000L_20190806_000000V.stat > filter.stat
This will discard those 2 stations from that file and write the output
to a
new file named "filter.stat".
Thanks,
John
On Mon, Aug 19, 2019 at 9:01 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563 >
>
> John, a related question. In stat_analysis I would like to
exclude obs
> from being considered - is there a command switch for that? I tried
> -sid_exc but that didn't work.
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Friday, August 16, 2019 5:19 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat
Anal
> Filtering
>
> Bob,
>
> I never saw an email come through from amerdec. So to test, I used
some
> sample data that's generated when you run 'make test'.
>
> *cd met-8.1*
> *bin/stat_analysis -lookin
> out/point_stat/point_stat_360000L_20070331_120000V.stat \* *-job
filter
> -line_type MPR -interp_mthd NEAREST -dump_row filter.stat*
>
> DEBUG 2: STAT Lines read = 28140
> DEBUG 2: STAT Lines retained = 9288
>
> And then I picked out the first 500 unique station id names from the
MRP
> lines and wrote them to a file:
>
> *echo "SID_MASK" > my_sid_list.txt*
> *grep MPR out/point_stat/point_stat_360000L_20070331_120000V.stat |
awk
> '{print $27}' | sort -u | head -500 >> my_sid_list.txt*
>
> And then I reran using that station ID list:
>
>
> *bin/stat_analysis -lookin
> out/point_stat/point_stat_360000L_20070331_120000V.stat -job filter
> -line_type MPR -interp_mthd NEAREST -dump_row filter.stat -mask_sid
> my_sid_list.txt*
>
> DEBUG 2: STAT Lines read = 28140
> DEBUG 2: STAT Lines retained = 1451
>
> The 9288 MPR lines for NEAREST were reduced down to 1451. So this
> functionality *should* be working. But I'm not sure where the
disconnect
> is.
>
> Thanks,
> John
>
> On Fri, Aug 16, 2019 at 3:00 PM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563 >
> >
> > John, that gives me the error:
> >
> > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> > /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> > DEBUG 1: Default Config File:
> > /h/MET/share/met/config/STATAnalysisConfig_default
> > DEBUG 1: Station ID Mask:
> > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
> > DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162 points
ERROR
> > :
> > " in job command line: -job filter -interp_mthd NBRHD_SQUARE
> > -line_type MPR -mask_sid
> > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
> >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> > ERROR :
> > ERROR :
> > ERROR : main() -> encountered an error value of 1. Calling
> > clean_up() and usage() before exiting.
> > ERROR :
> >
> > I sent you the data file via amerdec if you would like to try it.
> >
> > Thanks
> > Bob
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Friday, August 16, 2019 3:53 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat
Anal
> > Filtering
> >
> > Bob,
> >
> > You should also be seeing this error message in the output:
> >
> > ERROR : do_job_filter() -> this function may only be called when
> > using
> > the* -dump_row *option in the job command line: -job filter
-line_type
> > MPR -mask_sid baltic1p5km.txt
> >
> > The output for the "filter" job is specified by the "-dump_row"
> > option, not the "-out_stat" option.
> >
> > Please try running:
> >
> > */h/MET/bin/stat_analysis \*
> > *-lookin
> >
/h/data/global/WXQC/data/met/p**tstat/mdlob_pairs/global_17km/**CB/tes
> > t
> > \*
> > *-job filter -interp_mthd NBRHD_SQUARE -line_type MPR \* *-
mask_sid
> > /h/data/global/WXQC/data/met/sid/baltic1p5km.**txt \* *-dump_row
> > /h/data/global/WXQC/data/met/s**ummary/CB_global_17km_le3000_d*
> > *ump.stat*
> >
> > Does that work?
> >
> > Thanks,
> > John
> >
> > On Fri, Aug 16, 2019 at 2:43 PM robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Fri Aug 16 14:43:07 2019: Request 91563 was acted upon.
> > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > Queue: met_help
> > > Subject: FW: Stat Anal Filtering
> > > Owner: Nobody
> > > Requestors: robert.craig.2 at us.af.mil
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563
> > > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > Sent: Friday, August 16, 2019 3:04 PM
> > > To: met_help at ucar.edu
> > > Subject: Stat Anal Filtering
> > >
> > >
> > >
> > > John, I am having trouble trying to filter a stat file using the
> > > following command line:
> > >
> > >
> > >
> > > /h/MET/bin/stat_analysis -lookin
> > >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/test
> > > -job filter -interp_mthd NBRHD_SQUARE -line_type MPR -mask_sid
> > > /h/data/global/ WXQC/data/met/sid/baltic1p5km.txt -out_stat
> > >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> > >
> > >
> > >
> > > I also tried:
> > >
> > >
> > >
> > > /h/MET/bin/stat_analysis -lookin
> > >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/test
> > > -job filter -line_type MPR -column_str OBS_SID
> > > /h/data/global/WXQC/data/met/si d/baltic1p5km.txt -dump_row
> > >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> > >
> > >
> > >
> > > The goal is to reduce the stat file to just those mpr lines that
> > > contain station ids in the file baltic1p5km.txt and
> > >
> > > The data file I am trying to filter is attached. I have tried
lots
> > > of combinations of things from the users manual to filter the
data
> > > but keep getting the following error:
> > >
> > >
> > >
> > > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> > > /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> > >
> > > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> > >
> > > DEBUG 1: Default Config File:
> > > /h/MET/share/met/config/STATAnalysisConfig_default
> > >
> > > DEBUG 1: Station ID Mask:
> > > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
> > >
> > > DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162 points
> > >
> > > ERROR :
> > >
> > > " in job command line: -job filter -interp_mthd NBRHD_SQUARE
> > > -line_type MPR -mask_sid
> > > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -out_stat
> > >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> > >
> > > ERROR :
> > >
> > > ERROR :
> > >
> > > ERROR : main() -> encountered an error value of 1. Calling
> > > clean_up() and
> > > usage() before exiting.
> > >
> > >
> > >
> > > What I am doing wrong?
> > >
> > >
> > >
> > > Thanks
> > >
> > > Bob
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat Anal Filtering
From: robert.craig.2 at us.af.mil
Time: Mon Aug 19 10:17:14 2019
John, I noticed the email was going to methelp instead of you. So I
will try on last time going to johnhg at ucar.edu. I tried the dos2unix
command but got the same error. Not sure what you mean by a truncated
error message. I changed the verbosity to 4 and that is below:
DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
/h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
/h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
DEBUG 1: Default Config File:
/h/MET/share/met/config/STATAnalysisConfig_default
DEBUG 4: Default Job from the config file: "(nul)"
DEBUG 4: Amending default job with command line options: "-job filter
-interp_mthd NBRHD_SQUARE -line_type MPR -mask_sid
/h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
/h/"ata/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
DEBUG 1: Station ID Mask:
/h/data/global/WXQC/data/met/sid/baltic1p5km.txt
DEBUG 4: parse_sid_mask() -> parsing station ID masking file
"/h/data/global/WXQC/data/met/sid/baltic1p5km.txt"
DEBUG 4: parse_sid_mask() -> parsed 162 station ID's for the
"baltic1p5" mask from file
"/h/data/global/WXQC/data/met/sid/baltic1p5km.txt"
DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162 points
ERROR :
" in job command line: -job filter -interp_mthd NBRHD_SQUARE
-line_type MPR -mask_sid
/h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
ERROR :
ERROR :
ERROR : main() -> encountered an error value of 1. Calling
clean_up() and usage() before exiting.
ERROR :
*** Model Evaluation Tools (METV8.1) ***
Usage: stat_analysis
-lookin path
[-out file]
[-tmp_dir path]
[-log file]
[-v level]
[-config config_file] | [JOB COMMAND LINE]
where "-lookin path" specifies a STAT file or top-level
directory containing STAT files. It allows the use of wildcards (at
least one required).
"-out file" specifies a file to which output should be
written rather than the screen (optional).
"-tmp_dir path" specifies the directory into which
temporary files should be written (optional).
"-log file" outputs log messages to the specified file
(optional).
"-v level" overrides the default level of logging (4)
(optional).
"-config config_file" specifies a STATAnalysis config
file containing STATAnalysis jobs to be run.
"JOB COMMAND LINE" specifies all the arguments
necessary to perform a single STATAnalysis job.
NOTE: Refer to a STATAnalysis config file for details on how
to specify the JOB COMMAND LINE.
As for the option to limit the SID's in stat_analysis, option 2 sounds
good. I CCed Matt on this incase he has an opinion on this option.
Thanks
Bob
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Monday, August 19, 2019 10:57 AM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat
Anal Filtering
Bob,
Still no luck. I'm not seeing anything from ARMDEC coming to my email
address "johnhg at ucar.edu".
One big mystery is the error message you sent to me:
ERROR :
" in job command line: -job filter -interp_mthd NBRHD_SQUARE
-line_type MPR -mask_sid
/h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
ERROR :
The whole first part of it is truncated. There should be an actual
message before it says " in job command...". So why/how is that
happening? One additional thing to consider... you use Windows and
that inserts a ^M character at the end of every line of text as the
line break marker. In the past, those ^M characters have caused
trouble. I'm not sure they are here, but it's always a possibility.
You could try running that ascii file through the dos2unix utility
which strips off those ^M characters:
dos2unix baltic1p5km.txt
And then see if anything changes.
For your second question, no, there currently isn't any way to tell
STAT-Analysis which stations to skip. The closest thing is the "-
column_str" option.
For example, "-column MPR -column_str OBS_SID KDEN,690150" tells STAT-
Analysis to only use lines MPR lines where the OBS_SID column contains
"KDEN" or "690150". But there is no negation for this option. We
can't tell it which strings to exclude.
For numeric values, this is easy. The "-column_thresh" option defines
which numbers to keep. For example, "-column MPR -column_thresh FCST
gt100" keeps only MPR lines where the FCST value is > 100. Negating
this is easy... just switch the threshold "-column_thresh FCST le100".
We could potentially enhance stat_analysis (and tc_stat) to support
this in the future. I see a few implementation options...
(1) Explicitly add a "-sid_exc" option which only works when
processing MPR or ORANK lines.
(2) Add a new "-column_str_exc" option to do the opposite of the
existing "-column_str" option. This is generally applied to any
column containing strings.
(3) Just use the existing "-column_str" option but switch to
interpreting the strings as regular expressions, and define the
negation as a regular expression.
I think I lean toward option (2). While (3) is nice, regular
expressions can be difficult to set up well and produce unintended
matches. Please let me know if you'd like me to write up this as a
potential GitHub development issue.
For now, I'd recommend just running egrep prior to running stat-
analysis to throw out the stations you don't want. For example:
egrep -v " 690150 | 710630 "
point_stat_meps_conus_480000L_20190806_000000V.stat > filter.stat
This will discard those 2 stations from that file and write the output
to a new file named "filter.stat".
Thanks,
John
On Mon, Aug 19, 2019 at 9:01 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563 >
>
> John, a related question. In stat_analysis I would like to
exclude obs
> from being considered - is there a command switch for that? I tried
> -sid_exc but that didn't work.
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Friday, August 16, 2019 5:19 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat
> Anal Filtering
>
> Bob,
>
> I never saw an email come through from amerdec. So to test, I used
> some sample data that's generated when you run 'make test'.
>
> *cd met-8.1*
> *bin/stat_analysis -lookin
> out/point_stat/point_stat_360000L_20070331_120000V.stat \* *-job
> filter -line_type MPR -interp_mthd NEAREST -dump_row filter.stat*
>
> DEBUG 2: STAT Lines read = 28140
> DEBUG 2: STAT Lines retained = 9288
>
> And then I picked out the first 500 unique station id names from the
> MRP lines and wrote them to a file:
>
> *echo "SID_MASK" > my_sid_list.txt*
> *grep MPR out/point_stat/point_stat_360000L_20070331_120000V.stat |
> awk '{print $27}' | sort -u | head -500 >> my_sid_list.txt*
>
> And then I reran using that station ID list:
>
>
> *bin/stat_analysis -lookin
> out/point_stat/point_stat_360000L_20070331_120000V.stat -job filter
> -line_type MPR -interp_mthd NEAREST -dump_row filter.stat -mask_sid
> my_sid_list.txt*
>
> DEBUG 2: STAT Lines read = 28140
> DEBUG 2: STAT Lines retained = 1451
>
> The 9288 MPR lines for NEAREST were reduced down to 1451. So this
> functionality *should* be working. But I'm not sure where the
> disconnect is.
>
> Thanks,
> John
>
> On Fri, Aug 16, 2019 at 3:00 PM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563 >
> >
> > John, that gives me the error:
> >
> > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> > /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> > DEBUG 1: Default Config File:
> > /h/MET/share/met/config/STATAnalysisConfig_default
> > DEBUG 1: Station ID Mask:
> > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
> > DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162 points
> > ERROR
> > :
> > " in job command line: -job filter -interp_mthd NBRHD_SQUARE
> > -line_type MPR -mask_sid
> > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
> >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> > ERROR :
> > ERROR :
> > ERROR : main() -> encountered an error value of 1. Calling
> > clean_up() and usage() before exiting.
> > ERROR :
> >
> > I sent you the data file via amerdec if you would like to try it.
> >
> > Thanks
> > Bob
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Friday, August 16, 2019 3:53 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat
Anal
> > Filtering
> >
> > Bob,
> >
> > You should also be seeing this error message in the output:
> >
> > ERROR : do_job_filter() -> this function may only be called when
> > using
> > the* -dump_row *option in the job command line: -job filter
> > -line_type MPR -mask_sid baltic1p5km.txt
> >
> > The output for the "filter" job is specified by the "-dump_row"
> > option, not the "-out_stat" option.
> >
> > Please try running:
> >
> > */h/MET/bin/stat_analysis \*
> > *-lookin
> >
/h/data/global/WXQC/data/met/p**tstat/mdlob_pairs/global_17km/**CB/t
> > es
> > t
> > \*
> > *-job filter -interp_mthd NBRHD_SQUARE -line_type MPR \* *-
mask_sid
> > /h/data/global/WXQC/data/met/sid/baltic1p5km.**txt \* *-dump_row
> > /h/data/global/WXQC/data/met/s**ummary/CB_global_17km_le3000_d*
> > *ump.stat*
> >
> > Does that work?
> >
> > Thanks,
> > John
> >
> > On Fri, Aug 16, 2019 at 2:43 PM robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Fri Aug 16 14:43:07 2019: Request 91563 was acted upon.
> > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > Queue: met_help
> > > Subject: FW: Stat Anal Filtering
> > > Owner: Nobody
> > > Requestors: robert.craig.2 at us.af.mil
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563
> > > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > Sent: Friday, August 16, 2019 3:04 PM
> > > To: met_help at ucar.edu
> > > Subject: Stat Anal Filtering
> > >
> > >
> > >
> > > John, I am having trouble trying to filter a stat file using the
> > > following command line:
> > >
> > >
> > >
> > > /h/MET/bin/stat_analysis -lookin
> > >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/tes
> > > t -job filter -interp_mthd NBRHD_SQUARE -line_type MPR -mask_sid
> > > /h/data/global/ WXQC/data/met/sid/baltic1p5km.txt -out_stat
> > >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.st
> > > at
> > >
> > >
> > >
> > > I also tried:
> > >
> > >
> > >
> > > /h/MET/bin/stat_analysis -lookin
> > >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/tes
> > > t -job filter -line_type MPR -column_str OBS_SID
> > > /h/data/global/WXQC/data/met/si d/baltic1p5km.txt -dump_row
> > >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.st
> > > at
> > >
> > >
> > >
> > > The goal is to reduce the stat file to just those mpr lines that
> > > contain station ids in the file baltic1p5km.txt and
> > >
> > > The data file I am trying to filter is attached. I have tried
> > > lots of combinations of things from the users manual to filter
the
> > > data but keep getting the following error:
> > >
> > >
> > >
> > > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> > > /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> > >
> > > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> > >
> > > DEBUG 1: Default Config File:
> > > /h/MET/share/met/config/STATAnalysisConfig_default
> > >
> > > DEBUG 1: Station ID Mask:
> > > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
> > >
> > > DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162 points
> > >
> > > ERROR :
> > >
> > > " in job command line: -job filter -interp_mthd NBRHD_SQUARE
> > > -line_type MPR -mask_sid
> > > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -out_stat
> > >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.st
> > > at
> > >
> > > ERROR :
> > >
> > > ERROR :
> > >
> > > ERROR : main() -> encountered an error value of 1. Calling
> > > clean_up() and
> > > usage() before exiting.
> > >
> > >
> > >
> > > What I am doing wrong?
> > >
> > >
> > >
> > > Thanks
> > >
> > > Bob
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: FW: Stat Anal Filtering
From: John Halley Gotway
Time: Mon Aug 19 11:29:16 2019
Bob,
OK, I got the file when you sent it to johnhg at ucar.edu. I wonder if
perhaps the ones to met-help get treated as spam?
Anyway, below I've listed the results of my successful run of met-8.1
stat_analysis. It reads 31,674 input lines and kept 414 of them which
meet
the filtering criteria. So I'm not getting the error you're getting.
If you run the exact same command but with the "-mask_sid" option
removed,
does that work? Hopefully we can narrow down a bit more what's
actually
causing the problem.
FYI, here's what I mean about truncated error message... If I rerun
WITHOUT
the "-dump_row" option, stat_analysis correctly errors out with the
following message:
ERROR : do_job_filter() -> this function may only be called when
using the
-dump_row option in the job command line: -job filter -interp_mthd
NBRHD_SQUARE -line_type MPR -mask_sid baltic1p5km.txt
You're seeing a similar message, BUT the first part of that message is
missing from your output (do_job_filter() -> this function may only be
called when using the -dump_row option). That's why I called it
truncated.
John
/usr/local/met-8.1/bin/stat_analysis -lookin
point_stat_global_17km_all_240000L_20180301_000000V.stat -job filter
-interp_mthd NBRHD_SQUARE -line_type MPR -mask_sid baltic1p5km.txt
-dump_row CB_global_17km_le3000_dump.stat
DEBUG 1: Default Config File:
/usr/local/met-8.1/share/met/config/STATAnalysisConfig_default
DEBUG 1: Station ID Mask: baltic1p5km.txt
DEBUG 2: Parsed Station ID Mask: baltic1p5km containing 673 points
DEBUG 2: Processing 1 STAT files.
DEBUG 2: STAT Lines read = 31674
DEBUG 2: STAT Lines retained = 414
DEBUG 2:
DEBUG 2: Processing Job 1: -job filter -interp_mthd NBRHD_SQUARE
-line_type
MPR -dump_row CB_global_17km_le3000_dump.stat -mask_sid
baltic1p5km.txt
DEBUG 1: Creating dump row output file
"CB_global_17km_le3000_dump.stat"
FILTER: -job filter -interp_mthd NBRHD_SQUARE -line_type MPR
-dump_row CB_global_17km_le3000_dump.stat -mask_sid baltic1p5km.txt
DEBUG 2: Job 1 used 414 out of 414 STAT lines.
On Mon, Aug 19, 2019 at 10:17 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563 >
>
> John, I noticed the email was going to methelp instead of you. So I
will
> try on last time going to johnhg at ucar.edu. I tried the dos2unix
command
> but got the same error. Not sure what you mean by a truncated error
> message. I changed the verbosity to 4 and that is below:
>
> DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> DEBUG 1: Default Config File:
> /h/MET/share/met/config/STATAnalysisConfig_default
> DEBUG 4: Default Job from the config file: "(nul)"
> DEBUG 4: Amending default job with command line options: "-job
filter
> -interp_mthd NBRHD_SQUARE -line_type MPR -mask_sid
> /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
> /h/"ata/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> DEBUG 1: Station ID Mask:
/h/data/global/WXQC/data/met/sid/baltic1p5km.txt
> DEBUG 4: parse_sid_mask() -> parsing station ID masking file
> "/h/data/global/WXQC/data/met/sid/baltic1p5km.txt"
> DEBUG 4: parse_sid_mask() -> parsed 162 station ID's for the
"baltic1p5"
> mask from file "/h/data/global/WXQC/data/met/sid/baltic1p5km.txt"
> DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162 points
> ERROR :
> " in job command line: -job filter -interp_mthd NBRHD_SQUARE
-line_type
> MPR -mask_sid /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
-dump_row
> /h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> ERROR :
> ERROR :
> ERROR : main() -> encountered an error value of 1. Calling
clean_up()
> and usage() before exiting.
> ERROR :
>
> *** Model Evaluation Tools (METV8.1) ***
>
> Usage: stat_analysis
> -lookin path
> [-out file]
> [-tmp_dir path]
> [-log file]
> [-v level]
> [-config config_file] | [JOB COMMAND LINE]
>
> where "-lookin path" specifies a STAT file or top-level
> directory containing STAT files. It allows the use of wildcards (at
least
> one required).
> "-out file" specifies a file to which output should
be
> written rather than the screen (optional).
> "-tmp_dir path" specifies the directory into which
> temporary files should be written (optional).
> "-log file" outputs log messages to the specified
file
> (optional).
> "-v level" overrides the default level of logging
(4)
> (optional).
> "-config config_file" specifies a STATAnalysis
config file
> containing STATAnalysis jobs to be run.
> "JOB COMMAND LINE" specifies all the arguments
necessary
> to perform a single STATAnalysis job.
>
> NOTE: Refer to a STATAnalysis config file for details on how
to
> specify the JOB COMMAND LINE.
>
>
> As for the option to limit the SID's in stat_analysis, option 2
sounds
> good. I CCed Matt on this incase he has an opinion on this option.
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Monday, August 19, 2019 10:57 AM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat
Anal
> Filtering
>
> Bob,
>
> Still no luck. I'm not seeing anything from ARMDEC coming to my
email
> address "johnhg at ucar.edu".
>
> One big mystery is the error message you sent to me:
> ERROR :
> " in job command line: -job filter -interp_mthd NBRHD_SQUARE
-line_type
> MPR -mask_sid /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
-dump_row
> /h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> ERROR :
>
> The whole first part of it is truncated. There should be an actual
> message before it says " in job command...". So why/how is that
> happening? One additional thing to consider... you use Windows and
that
> inserts a ^M character at the end of every line of text as the line
break
> marker. In the past, those ^M characters have caused trouble. I'm
not
> sure they are here, but it's always a possibility. You could try
running
> that ascii file through the dos2unix utility which strips off those
^M
> characters:
>
> dos2unix baltic1p5km.txt
>
> And then see if anything changes.
>
> For your second question, no, there currently isn't any way to tell
> STAT-Analysis which stations to skip. The closest thing is the
> "-column_str" option.
> For example, "-column MPR -column_str OBS_SID KDEN,690150" tells
> STAT-Analysis to only use lines MPR lines where the OBS_SID column
contains
> "KDEN" or "690150". But there is no negation for this option. We
can't
> tell it which strings to exclude.
>
> For numeric values, this is easy. The "-column_thresh" option
defines
> which numbers to keep. For example, "-column MPR -column_thresh
FCST
> gt100" keeps only MPR lines where the FCST value is > 100. Negating
this
> is easy... just switch the threshold "-column_thresh FCST le100".
>
> We could potentially enhance stat_analysis (and tc_stat) to support
this
> in the future. I see a few implementation options...
> (1) Explicitly add a "-sid_exc" option which only works when
processing
> MPR or ORANK lines.
> (2) Add a new "-column_str_exc" option to do the opposite of the
existing
> "-column_str" option. This is generally applied to any column
containing
> strings.
> (3) Just use the existing "-column_str" option but switch to
interpreting
> the strings as regular expressions, and define the negation as a
regular
> expression.
>
> I think I lean toward option (2). While (3) is nice, regular
expressions
> can be difficult to set up well and produce unintended matches.
Please let
> me know if you'd like me to write up this as a potential GitHub
development
> issue.
>
> For now, I'd recommend just running egrep prior to running stat-
analysis
> to throw out the stations you don't want. For example:
>
> egrep -v " 690150 | 710630 "
> point_stat_meps_conus_480000L_20190806_000000V.stat > filter.stat
>
> This will discard those 2 stations from that file and write the
output to
> a new file named "filter.stat".
>
> Thanks,
> John
>
> On Mon, Aug 19, 2019 at 9:01 AM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563 >
> >
> > John, a related question. In stat_analysis I would like to
exclude obs
> > from being considered - is there a command switch for that? I
tried
> > -sid_exc but that didn't work.
> >
> > Thanks
> > Bob
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Friday, August 16, 2019 5:19 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW:
Stat
> > Anal Filtering
> >
> > Bob,
> >
> > I never saw an email come through from amerdec. So to test, I
used
> > some sample data that's generated when you run 'make test'.
> >
> > *cd met-8.1*
> > *bin/stat_analysis -lookin
> > out/point_stat/point_stat_360000L_20070331_120000V.stat \* *-job
> > filter -line_type MPR -interp_mthd NEAREST -dump_row filter.stat*
> >
> > DEBUG 2: STAT Lines read = 28140
> > DEBUG 2: STAT Lines retained = 9288
> >
> > And then I picked out the first 500 unique station id names from
the
> > MRP lines and wrote them to a file:
> >
> > *echo "SID_MASK" > my_sid_list.txt*
> > *grep MPR out/point_stat/point_stat_360000L_20070331_120000V.stat
|
> > awk '{print $27}' | sort -u | head -500 >> my_sid_list.txt*
> >
> > And then I reran using that station ID list:
> >
> >
> > *bin/stat_analysis -lookin
> > out/point_stat/point_stat_360000L_20070331_120000V.stat -job
filter
> > -line_type MPR -interp_mthd NEAREST -dump_row filter.stat
-mask_sid
> > my_sid_list.txt*
> >
> > DEBUG 2: STAT Lines read = 28140
> > DEBUG 2: STAT Lines retained = 1451
> >
> > The 9288 MPR lines for NEAREST were reduced down to 1451. So this
> > functionality *should* be working. But I'm not sure where the
> > disconnect is.
> >
> > Thanks,
> > John
> >
> > On Fri, Aug 16, 2019 at 3:00 PM robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563 >
> > >
> > > John, that gives me the error:
> > >
> > > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> > > /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> > > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> > > DEBUG 1: Default Config File:
> > > /h/MET/share/met/config/STATAnalysisConfig_default
> > > DEBUG 1: Station ID Mask:
> > > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
> > > DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162 points
> > > ERROR
> > > :
> > > " in job command line: -job filter -interp_mthd NBRHD_SQUARE
> > > -line_type MPR -mask_sid
> > > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
> > >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> > > ERROR :
> > > ERROR :
> > > ERROR : main() -> encountered an error value of 1. Calling
> > > clean_up() and usage() before exiting.
> > > ERROR :
> > >
> > > I sent you the data file via amerdec if you would like to try
it.
> > >
> > > Thanks
> > > Bob
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: Friday, August 16, 2019 3:53 PM
> > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > <robert.craig.2 at us.af.mil>
> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat
Anal
> > > Filtering
> > >
> > > Bob,
> > >
> > > You should also be seeing this error message in the output:
> > >
> > > ERROR : do_job_filter() -> this function may only be called
when
> > > using
> > > the* -dump_row *option in the job command line: -job filter
> > > -line_type MPR -mask_sid baltic1p5km.txt
> > >
> > > The output for the "filter" job is specified by the "-dump_row"
> > > option, not the "-out_stat" option.
> > >
> > > Please try running:
> > >
> > > */h/MET/bin/stat_analysis \*
> > > *-lookin
> > >
/h/data/global/WXQC/data/met/p**tstat/mdlob_pairs/global_17km/**CB/t
> > > es
> > > t
> > > \*
> > > *-job filter -interp_mthd NBRHD_SQUARE -line_type MPR \* *-
mask_sid
> > > /h/data/global/WXQC/data/met/sid/baltic1p5km.**txt \* *-dump_row
> > > /h/data/global/WXQC/data/met/s**ummary/CB_global_17km_le3000_d*
> > > *ump.stat*
> > >
> > > Does that work?
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Aug 16, 2019 at 2:43 PM robert.craig.2 at us.af.mil via RT
<
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Fri Aug 16 14:43:07 2019: Request 91563 was acted upon.
> > > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > > Queue: met_help
> > > > Subject: FW: Stat Anal Filtering
> > > > Owner: Nobody
> > > > Requestors: robert.craig.2 at us.af.mil
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > > Sent: Friday, August 16, 2019 3:04 PM
> > > > To: met_help at ucar.edu
> > > > Subject: Stat Anal Filtering
> > > >
> > > >
> > > >
> > > > John, I am having trouble trying to filter a stat file using
the
> > > > following command line:
> > > >
> > > >
> > > >
> > > > /h/MET/bin/stat_analysis -lookin
> > > >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/tes
> > > > t -job filter -interp_mthd NBRHD_SQUARE -line_type MPR
-mask_sid
> > > > /h/data/global/ WXQC/data/met/sid/baltic1p5km.txt -out_stat
> > > >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.st
> > > > at
> > > >
> > > >
> > > >
> > > > I also tried:
> > > >
> > > >
> > > >
> > > > /h/MET/bin/stat_analysis -lookin
> > > >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/tes
> > > > t -job filter -line_type MPR -column_str OBS_SID
> > > > /h/data/global/WXQC/data/met/si d/baltic1p5km.txt -dump_row
> > > >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.st
> > > > at
> > > >
> > > >
> > > >
> > > > The goal is to reduce the stat file to just those mpr lines
that
> > > > contain station ids in the file baltic1p5km.txt and
> > > >
> > > > The data file I am trying to filter is attached. I have tried
> > > > lots of combinations of things from the users manual to filter
the
> > > > data but keep getting the following error:
> > > >
> > > >
> > > >
> > > > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> > > > /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> > > >
> > > > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > > > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> > > >
> > > > DEBUG 1: Default Config File:
> > > > /h/MET/share/met/config/STATAnalysisConfig_default
> > > >
> > > > DEBUG 1: Station ID Mask:
> > > > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
> > > >
> > > > DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162
points
> > > >
> > > > ERROR :
> > > >
> > > > " in job command line: -job filter -interp_mthd NBRHD_SQUARE
> > > > -line_type MPR -mask_sid
> > > > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -out_stat
> > > >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.st
> > > > at
> > > >
> > > > ERROR :
> > > >
> > > > ERROR :
> > > >
> > > > ERROR : main() -> encountered an error value of 1. Calling
> > > > clean_up() and
> > > > usage() before exiting.
> > > >
> > > >
> > > >
> > > > What I am doing wrong?
> > > >
> > > >
> > > >
> > > > Thanks
> > > >
> > > > Bob
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat Anal Filtering
From: robert.craig.2 at us.af.mil
Time: Tue Aug 20 13:50:55 2019
John, I believe I found the problem. Taking your advice, I tried it
without the -mask_sid command and it worked. So checking the file I
was calling using -mask_sid, I found I had one extra space at the end
of the line of station ids. Removing that space allowed the code to
run without error.
Thanks
Bob
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Monday, August 19, 2019 12:29 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat
Anal Filtering
Bob,
OK, I got the file when you sent it to johnhg at ucar.edu. I wonder if
perhaps the ones to met-help get treated as spam?
Anyway, below I've listed the results of my successful run of met-8.1
stat_analysis. It reads 31,674 input lines and kept 414 of them which
meet the filtering criteria. So I'm not getting the error you're
getting.
If you run the exact same command but with the "-mask_sid" option
removed, does that work? Hopefully we can narrow down a bit more
what's actually causing the problem.
FYI, here's what I mean about truncated error message... If I rerun
WITHOUT the "-dump_row" option, stat_analysis correctly errors out
with the following message:
ERROR : do_job_filter() -> this function may only be called when
using the -dump_row option in the job command line: -job filter
-interp_mthd NBRHD_SQUARE -line_type MPR -mask_sid baltic1p5km.txt
You're seeing a similar message, BUT the first part of that message is
missing from your output (do_job_filter() -> this function may only be
called when using the -dump_row option). That's why I called it
truncated.
John
/usr/local/met-8.1/bin/stat_analysis -lookin
point_stat_global_17km_all_240000L_20180301_000000V.stat -job filter
-interp_mthd NBRHD_SQUARE -line_type MPR -mask_sid baltic1p5km.txt
-dump_row CB_global_17km_le3000_dump.stat DEBUG 1: Default Config
File:
/usr/local/met-8.1/share/met/config/STATAnalysisConfig_default
DEBUG 1: Station ID Mask: baltic1p5km.txt DEBUG 2: Parsed Station ID
Mask: baltic1p5km containing 673 points DEBUG 2: Processing 1 STAT
files.
DEBUG 2: STAT Lines read = 31674
DEBUG 2: STAT Lines retained = 414
DEBUG 2:
DEBUG 2: Processing Job 1: -job filter -interp_mthd NBRHD_SQUARE
-line_type MPR -dump_row CB_global_17km_le3000_dump.stat -mask_sid
baltic1p5km.txt DEBUG 1: Creating dump row output file
"CB_global_17km_le3000_dump.stat"
FILTER: -job filter -interp_mthd NBRHD_SQUARE -line_type MPR
-dump_row CB_global_17km_le3000_dump.stat -mask_sid baltic1p5km.txt
DEBUG 2: Job 1 used 414 out of 414 STAT lines.
On Mon, Aug 19, 2019 at 10:17 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563 >
>
> John, I noticed the email was going to methelp instead of you. So I
will
> try on last time going to johnhg at ucar.edu. I tried the dos2unix
command
> but got the same error. Not sure what you mean by a truncated error
> message. I changed the verbosity to 4 and that is below:
>
> DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> DEBUG 1: Default Config File:
> /h/MET/share/met/config/STATAnalysisConfig_default
> DEBUG 4: Default Job from the config file: "(nul)"
> DEBUG 4: Amending default job with command line options: "-job
filter
> -interp_mthd NBRHD_SQUARE -line_type MPR -mask_sid
> /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
> /h/"ata/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> DEBUG 1: Station ID Mask:
> /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
> DEBUG 4: parse_sid_mask() -> parsing station ID masking file
> "/h/data/global/WXQC/data/met/sid/baltic1p5km.txt"
> DEBUG 4: parse_sid_mask() -> parsed 162 station ID's for the
"baltic1p5"
> mask from file "/h/data/global/WXQC/data/met/sid/baltic1p5km.txt"
> DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162 points
ERROR
> :
> " in job command line: -job filter -interp_mthd NBRHD_SQUARE
> -line_type MPR -mask_sid
> /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
> /h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> ERROR :
> ERROR :
> ERROR : main() -> encountered an error value of 1. Calling
> clean_up() and usage() before exiting.
> ERROR :
>
> *** Model Evaluation Tools (METV8.1) ***
>
> Usage: stat_analysis
> -lookin path
> [-out file]
> [-tmp_dir path]
> [-log file]
> [-v level]
> [-config config_file] | [JOB COMMAND LINE]
>
> where "-lookin path" specifies a STAT file or top-level
> directory containing STAT files. It allows the use of wildcards (at
> least one required).
> "-out file" specifies a file to which output should
be
> written rather than the screen (optional).
> "-tmp_dir path" specifies the directory into which
> temporary files should be written (optional).
> "-log file" outputs log messages to the specified
file
> (optional).
> "-v level" overrides the default level of logging
(4)
> (optional).
> "-config config_file" specifies a STATAnalysis
config
> file containing STATAnalysis jobs to be run.
> "JOB COMMAND LINE" specifies all the arguments
> necessary to perform a single STATAnalysis job.
>
> NOTE: Refer to a STATAnalysis config file for details on how
> to specify the JOB COMMAND LINE.
>
>
> As for the option to limit the SID's in stat_analysis, option 2
sounds
> good. I CCed Matt on this incase he has an opinion on this option.
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Monday, August 19, 2019 10:57 AM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat
> Anal Filtering
>
> Bob,
>
> Still no luck. I'm not seeing anything from ARMDEC coming to my
email
> address "johnhg at ucar.edu".
>
> One big mystery is the error message you sent to me:
> ERROR :
> " in job command line: -job filter -interp_mthd NBRHD_SQUARE
> -line_type MPR -mask_sid
> /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
> /h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> ERROR :
>
> The whole first part of it is truncated. There should be an actual
> message before it says " in job command...". So why/how is that
> happening? One additional thing to consider... you use Windows and
> that inserts a ^M character at the end of every line of text as the
> line break marker. In the past, those ^M characters have caused
> trouble. I'm not sure they are here, but it's always a possibility.
> You could try running that ascii file through the dos2unix utility
> which strips off those ^M
> characters:
>
> dos2unix baltic1p5km.txt
>
> And then see if anything changes.
>
> For your second question, no, there currently isn't any way to tell
> STAT-Analysis which stations to skip. The closest thing is the
> "-column_str" option.
> For example, "-column MPR -column_str OBS_SID KDEN,690150" tells
> STAT-Analysis to only use lines MPR lines where the OBS_SID column
> contains "KDEN" or "690150". But there is no negation for this
> option. We can't tell it which strings to exclude.
>
> For numeric values, this is easy. The "-column_thresh" option
defines
> which numbers to keep. For example, "-column MPR -column_thresh
FCST
> gt100" keeps only MPR lines where the FCST value is > 100. Negating
> this is easy... just switch the threshold "-column_thresh FCST
le100".
>
> We could potentially enhance stat_analysis (and tc_stat) to support
> this in the future. I see a few implementation options...
> (1) Explicitly add a "-sid_exc" option which only works when
> processing MPR or ORANK lines.
> (2) Add a new "-column_str_exc" option to do the opposite of the
> existing "-column_str" option. This is generally applied to any
> column containing strings.
> (3) Just use the existing "-column_str" option but switch to
> interpreting the strings as regular expressions, and define the
> negation as a regular expression.
>
> I think I lean toward option (2). While (3) is nice, regular
> expressions can be difficult to set up well and produce unintended
> matches. Please let me know if you'd like me to write up this as a
> potential GitHub development issue.
>
> For now, I'd recommend just running egrep prior to running
> stat-analysis to throw out the stations you don't want. For
example:
>
> egrep -v " 690150 | 710630 "
> point_stat_meps_conus_480000L_20190806_000000V.stat > filter.stat
>
> This will discard those 2 stations from that file and write the
output
> to a new file named "filter.stat".
>
> Thanks,
> John
>
> On Mon, Aug 19, 2019 at 9:01 AM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563 >
> >
> > John, a related question. In stat_analysis I would like to
exclude obs
> > from being considered - is there a command switch for that? I
tried
> > -sid_exc but that didn't work.
> >
> > Thanks
> > Bob
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Friday, August 16, 2019 5:19 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW:
Stat
> > Anal Filtering
> >
> > Bob,
> >
> > I never saw an email come through from amerdec. So to test, I
used
> > some sample data that's generated when you run 'make test'.
> >
> > *cd met-8.1*
> > *bin/stat_analysis -lookin
> > out/point_stat/point_stat_360000L_20070331_120000V.stat \* *-job
> > filter -line_type MPR -interp_mthd NEAREST -dump_row filter.stat*
> >
> > DEBUG 2: STAT Lines read = 28140
> > DEBUG 2: STAT Lines retained = 9288
> >
> > And then I picked out the first 500 unique station id names from
the
> > MRP lines and wrote them to a file:
> >
> > *echo "SID_MASK" > my_sid_list.txt*
> > *grep MPR out/point_stat/point_stat_360000L_20070331_120000V.stat
|
> > awk '{print $27}' | sort -u | head -500 >> my_sid_list.txt*
> >
> > And then I reran using that station ID list:
> >
> >
> > *bin/stat_analysis -lookin
> > out/point_stat/point_stat_360000L_20070331_120000V.stat -job
filter
> > -line_type MPR -interp_mthd NEAREST -dump_row filter.stat
-mask_sid
> > my_sid_list.txt*
> >
> > DEBUG 2: STAT Lines read = 28140
> > DEBUG 2: STAT Lines retained = 1451
> >
> > The 9288 MPR lines for NEAREST were reduced down to 1451. So this
> > functionality *should* be working. But I'm not sure where the
> > disconnect is.
> >
> > Thanks,
> > John
> >
> > On Fri, Aug 16, 2019 at 3:00 PM robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563 >
> > >
> > > John, that gives me the error:
> > >
> > > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> > > /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> > > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> > > DEBUG 1: Default Config File:
> > > /h/MET/share/met/config/STATAnalysisConfig_default
> > > DEBUG 1: Station ID Mask:
> > > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
> > > DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162 points
> > > ERROR
> > > :
> > > " in job command line: -job filter -interp_mthd NBRHD_SQUARE
> > > -line_type MPR -mask_sid
> > > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
> > >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.st
> > > at
> > > ERROR :
> > > ERROR :
> > > ERROR : main() -> encountered an error value of 1. Calling
> > > clean_up() and usage() before exiting.
> > > ERROR :
> > >
> > > I sent you the data file via amerdec if you would like to try
it.
> > >
> > > Thanks
> > > Bob
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: Friday, August 16, 2019 3:53 PM
> > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > <robert.craig.2 at us.af.mil>
> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat
> > > Anal Filtering
> > >
> > > Bob,
> > >
> > > You should also be seeing this error message in the output:
> > >
> > > ERROR : do_job_filter() -> this function may only be called
when
> > > using
> > > the* -dump_row *option in the job command line: -job filter
> > > -line_type MPR -mask_sid baltic1p5km.txt
> > >
> > > The output for the "filter" job is specified by the "-dump_row"
> > > option, not the "-out_stat" option.
> > >
> > > Please try running:
> > >
> > > */h/MET/bin/stat_analysis \*
> > > *-lookin
> > >
/h/data/global/WXQC/data/met/p**tstat/mdlob_pairs/global_17km/**CB
> > > /t
> > > es
> > > t
> > > \*
> > > *-job filter -interp_mthd NBRHD_SQUARE -line_type MPR \*
> > > *-mask_sid /h/data/global/WXQC/data/met/sid/baltic1p5km.**txt \*
> > > *-dump_row
> > > /h/data/global/WXQC/data/met/s**ummary/CB_global_17km_le3000_d*
> > > *ump.stat*
> > >
> > > Does that work?
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Aug 16, 2019 at 2:43 PM robert.craig.2 at us.af.mil via RT
<
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Fri Aug 16 14:43:07 2019: Request 91563 was acted upon.
> > > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > > Queue: met_help
> > > > Subject: FW: Stat Anal Filtering
> > > > Owner: Nobody
> > > > Requestors: robert.craig.2 at us.af.mil
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > > Sent: Friday, August 16, 2019 3:04 PM
> > > > To: met_help at ucar.edu
> > > > Subject: Stat Anal Filtering
> > > >
> > > >
> > > >
> > > > John, I am having trouble trying to filter a stat file using
the
> > > > following command line:
> > > >
> > > >
> > > >
> > > > /h/MET/bin/stat_analysis -lookin
> > > >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/t
> > > > es t -job filter -interp_mthd NBRHD_SQUARE -line_type MPR
> > > > -mask_sid /h/data/global/ WXQC/data/met/sid/baltic1p5km.txt
> > > > -out_stat
> > > >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.
> > > > st
> > > > at
> > > >
> > > >
> > > >
> > > > I also tried:
> > > >
> > > >
> > > >
> > > > /h/MET/bin/stat_analysis -lookin
> > > >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/t
> > > > es t -job filter -line_type MPR -column_str OBS_SID
> > > > /h/data/global/WXQC/data/met/si d/baltic1p5km.txt -dump_row
> > > >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.
> > > > st
> > > > at
> > > >
> > > >
> > > >
> > > > The goal is to reduce the stat file to just those mpr lines
that
> > > > contain station ids in the file baltic1p5km.txt and
> > > >
> > > > The data file I am trying to filter is attached. I have tried
> > > > lots of combinations of things from the users manual to filter
> > > > the data but keep getting the following error:
> > > >
> > > >
> > > >
> > > > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> > > > /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> > > >
> > > > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > > > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> > > >
> > > > DEBUG 1: Default Config File:
> > > > /h/MET/share/met/config/STATAnalysisConfig_default
> > > >
> > > > DEBUG 1: Station ID Mask:
> > > > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
> > > >
> > > > DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162
points
> > > >
> > > > ERROR :
> > > >
> > > > " in job command line: -job filter -interp_mthd NBRHD_SQUARE
> > > > -line_type MPR -mask_sid
> > > > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -out_stat
> > > >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.
> > > > st
> > > > at
> > > >
> > > > ERROR :
> > > >
> > > > ERROR :
> > > >
> > > > ERROR : main() -> encountered an error value of 1. Calling
> > > > clean_up() and
> > > > usage() before exiting.
> > > >
> > > >
> > > >
> > > > What I am doing wrong?
> > > >
> > > >
> > > >
> > > > Thanks
> > > >
> > > > Bob
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: FW: Stat Anal Filtering
From: John Halley Gotway
Time: Tue Aug 20 23:43:50 2019
Bob,
That’s great info. Thanks for letting me know. I’ll try making the
same
change locally in hopes of replicating that same error. If I’m able
to, I
should be able to fix it for the next release.
Thanks,
John
On Tue, Aug 20, 2019 at 1:51 PM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563 >
>
> John, I believe I found the problem. Taking your advice, I tried it
> without the -mask_sid command and it worked. So checking the file I
was
> calling using -mask_sid, I found I had one extra space at the end of
the
> line of station ids. Removing that space allowed the code to run
without
> error.
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Monday, August 19, 2019 12:29 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat
Anal
> Filtering
>
> Bob,
>
> OK, I got the file when you sent it to johnhg at ucar.edu. I wonder if
> perhaps the ones to met-help get treated as spam?
>
> Anyway, below I've listed the results of my successful run of met-
8.1
> stat_analysis. It reads 31,674 input lines and kept 414 of them
which meet
> the filtering criteria. So I'm not getting the error you're
getting.
>
> If you run the exact same command but with the "-mask_sid" option
removed,
> does that work? Hopefully we can narrow down a bit more what's
actually
> causing the problem.
>
> FYI, here's what I mean about truncated error message... If I rerun
> WITHOUT the "-dump_row" option, stat_analysis correctly errors out
with the
> following message:
> ERROR : do_job_filter() -> this function may only be called when
using
> the -dump_row option in the job command line: -job filter
-interp_mthd
> NBRHD_SQUARE -line_type MPR -mask_sid baltic1p5km.txt
>
> You're seeing a similar message, BUT the first part of that message
is
> missing from your output (do_job_filter() -> this function may only
be
> called when using the -dump_row option). That's why I called it
> truncated.
>
> John
>
> /usr/local/met-8.1/bin/stat_analysis -lookin
> point_stat_global_17km_all_240000L_20180301_000000V.stat -job filter
> -interp_mthd NBRHD_SQUARE -line_type MPR -mask_sid baltic1p5km.txt
> -dump_row CB_global_17km_le3000_dump.stat DEBUG 1: Default Config
File:
> /usr/local/met-8.1/share/met/config/STATAnalysisConfig_default
> DEBUG 1: Station ID Mask: baltic1p5km.txt DEBUG 2: Parsed Station ID
Mask:
> baltic1p5km containing 673 points DEBUG 2: Processing 1 STAT files.
> DEBUG 2: STAT Lines read = 31674
> DEBUG 2: STAT Lines retained = 414
> DEBUG 2:
> DEBUG 2: Processing Job 1: -job filter -interp_mthd NBRHD_SQUARE
> -line_type MPR -dump_row CB_global_17km_le3000_dump.stat -mask_sid
> baltic1p5km.txt DEBUG 1: Creating dump row output file
> "CB_global_17km_le3000_dump.stat"
> FILTER: -job filter -interp_mthd NBRHD_SQUARE -line_type MPR
> -dump_row CB_global_17km_le3000_dump.stat -mask_sid baltic1p5km.txt
DEBUG
> 2: Job 1 used 414 out of 414 STAT lines.
>
>
> On Mon, Aug 19, 2019 at 10:17 AM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563 >
> >
> > John, I noticed the email was going to methelp instead of you. So
I will
> > try on last time going to johnhg at ucar.edu. I tried the dos2unix
> command
> > but got the same error. Not sure what you mean by a truncated
error
> > message. I changed the verbosity to 4 and that is below:
> >
> > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> > /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> > DEBUG 1: Default Config File:
> > /h/MET/share/met/config/STATAnalysisConfig_default
> > DEBUG 4: Default Job from the config file: "(nul)"
> > DEBUG 4: Amending default job with command line options: "-job
filter
> > -interp_mthd NBRHD_SQUARE -line_type MPR -mask_sid
> > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
> >
/h/"ata/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> > DEBUG 1: Station ID Mask:
> > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
> > DEBUG 4: parse_sid_mask() -> parsing station ID masking file
> > "/h/data/global/WXQC/data/met/sid/baltic1p5km.txt"
> > DEBUG 4: parse_sid_mask() -> parsed 162 station ID's for the
"baltic1p5"
> > mask from file "/h/data/global/WXQC/data/met/sid/baltic1p5km.txt"
> > DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162 points
ERROR
> > :
> > " in job command line: -job filter -interp_mthd NBRHD_SQUARE
> > -line_type MPR -mask_sid
> > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
> >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> > ERROR :
> > ERROR :
> > ERROR : main() -> encountered an error value of 1. Calling
> > clean_up() and usage() before exiting.
> > ERROR :
> >
> > *** Model Evaluation Tools (METV8.1) ***
> >
> > Usage: stat_analysis
> > -lookin path
> > [-out file]
> > [-tmp_dir path]
> > [-log file]
> > [-v level]
> > [-config config_file] | [JOB COMMAND LINE]
> >
> > where "-lookin path" specifies a STAT file or top-level
> > directory containing STAT files. It allows the use of wildcards
(at
> > least one required).
> > "-out file" specifies a file to which output
should be
> > written rather than the screen (optional).
> > "-tmp_dir path" specifies the directory into which
> > temporary files should be written (optional).
> > "-log file" outputs log messages to the specified
file
> > (optional).
> > "-v level" overrides the default level of logging
(4)
> > (optional).
> > "-config config_file" specifies a STATAnalysis
config
> > file containing STATAnalysis jobs to be run.
> > "JOB COMMAND LINE" specifies all the arguments
> > necessary to perform a single STATAnalysis job.
> >
> > NOTE: Refer to a STATAnalysis config file for details on
how
> > to specify the JOB COMMAND LINE.
> >
> >
> > As for the option to limit the SID's in stat_analysis, option 2
sounds
> > good. I CCed Matt on this incase he has an opinion on this
option.
> >
> > Thanks
> > Bob
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Monday, August 19, 2019 10:57 AM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW:
Stat
> > Anal Filtering
> >
> > Bob,
> >
> > Still no luck. I'm not seeing anything from ARMDEC coming to my
email
> > address "johnhg at ucar.edu".
> >
> > One big mystery is the error message you sent to me:
> > ERROR :
> > " in job command line: -job filter -interp_mthd NBRHD_SQUARE
> > -line_type MPR -mask_sid
> > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
> >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
> > ERROR :
> >
> > The whole first part of it is truncated. There should be an
actual
> > message before it says " in job command...". So why/how is that
> > happening? One additional thing to consider... you use Windows
and
> > that inserts a ^M character at the end of every line of text as
the
> > line break marker. In the past, those ^M characters have caused
> > trouble. I'm not sure they are here, but it's always a
possibility.
> > You could try running that ascii file through the dos2unix utility
> > which strips off those ^M
> > characters:
> >
> > dos2unix baltic1p5km.txt
> >
> > And then see if anything changes.
> >
> > For your second question, no, there currently isn't any way to
tell
> > STAT-Analysis which stations to skip. The closest thing is the
> > "-column_str" option.
> > For example, "-column MPR -column_str OBS_SID KDEN,690150" tells
> > STAT-Analysis to only use lines MPR lines where the OBS_SID column
> > contains "KDEN" or "690150". But there is no negation for this
> > option. We can't tell it which strings to exclude.
> >
> > For numeric values, this is easy. The "-column_thresh" option
defines
> > which numbers to keep. For example, "-column MPR -column_thresh
FCST
> > gt100" keeps only MPR lines where the FCST value is > 100.
Negating
> > this is easy... just switch the threshold "-column_thresh FCST
le100".
> >
> > We could potentially enhance stat_analysis (and tc_stat) to
support
> > this in the future. I see a few implementation options...
> > (1) Explicitly add a "-sid_exc" option which only works when
> > processing MPR or ORANK lines.
> > (2) Add a new "-column_str_exc" option to do the opposite of the
> > existing "-column_str" option. This is generally applied to any
> > column containing strings.
> > (3) Just use the existing "-column_str" option but switch to
> > interpreting the strings as regular expressions, and define the
> > negation as a regular expression.
> >
> > I think I lean toward option (2). While (3) is nice, regular
> > expressions can be difficult to set up well and produce unintended
> > matches. Please let me know if you'd like me to write up this as
a
> > potential GitHub development issue.
> >
> > For now, I'd recommend just running egrep prior to running
> > stat-analysis to throw out the stations you don't want. For
example:
> >
> > egrep -v " 690150 | 710630 "
> > point_stat_meps_conus_480000L_20190806_000000V.stat > filter.stat
> >
> > This will discard those 2 stations from that file and write the
output
> > to a new file named "filter.stat".
> >
> > Thanks,
> > John
> >
> > On Mon, Aug 19, 2019 at 9:01 AM robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563 >
> > >
> > > John, a related question. In stat_analysis I would like to
exclude
> obs
> > > from being considered - is there a command switch for that? I
tried
> > > -sid_exc but that didn't work.
> > >
> > > Thanks
> > > Bob
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: Friday, August 16, 2019 5:19 PM
> > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > <robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW:
Stat
> > > Anal Filtering
> > >
> > > Bob,
> > >
> > > I never saw an email come through from amerdec. So to test, I
used
> > > some sample data that's generated when you run 'make test'.
> > >
> > > *cd met-8.1*
> > > *bin/stat_analysis -lookin
> > > out/point_stat/point_stat_360000L_20070331_120000V.stat \* *-job
> > > filter -line_type MPR -interp_mthd NEAREST -dump_row
filter.stat*
> > >
> > > DEBUG 2: STAT Lines read = 28140
> > > DEBUG 2: STAT Lines retained = 9288
> > >
> > > And then I picked out the first 500 unique station id names from
the
> > > MRP lines and wrote them to a file:
> > >
> > > *echo "SID_MASK" > my_sid_list.txt*
> > > *grep MPR
out/point_stat/point_stat_360000L_20070331_120000V.stat |
> > > awk '{print $27}' | sort -u | head -500 >> my_sid_list.txt*
> > >
> > > And then I reran using that station ID list:
> > >
> > >
> > > *bin/stat_analysis -lookin
> > > out/point_stat/point_stat_360000L_20070331_120000V.stat -job
filter
> > > -line_type MPR -interp_mthd NEAREST -dump_row filter.stat
-mask_sid
> > > my_sid_list.txt*
> > >
> > > DEBUG 2: STAT Lines read = 28140
> > > DEBUG 2: STAT Lines retained = 1451
> > >
> > > The 9288 MPR lines for NEAREST were reduced down to 1451. So
this
> > > functionality *should* be working. But I'm not sure where the
> > > disconnect is.
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Aug 16, 2019 at 3:00 PM robert.craig.2 at us.af.mil via RT
<
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563
>
> > > >
> > > > John, that gives me the error:
> > > >
> > > > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> > > > /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> > > > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > > > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> > > > DEBUG 1: Default Config File:
> > > > /h/MET/share/met/config/STATAnalysisConfig_default
> > > > DEBUG 1: Station ID Mask:
> > > > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
> > > > DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162
points
> > > > ERROR
> > > > :
> > > > " in job command line: -job filter -interp_mthd NBRHD_SQUARE
> > > > -line_type MPR -mask_sid
> > > > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
> > > >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.st
> > > > at
> > > > ERROR :
> > > > ERROR :
> > > > ERROR : main() -> encountered an error value of 1. Calling
> > > > clean_up() and usage() before exiting.
> > > > ERROR :
> > > >
> > > > I sent you the data file via amerdec if you would like to try
it.
> > > >
> > > > Thanks
> > > > Bob
> > > >
> > > > -----Original Message-----
> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > > Sent: Friday, August 16, 2019 3:53 PM
> > > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > > <robert.craig.2 at us.af.mil>
> > > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW:
Stat
> > > > Anal Filtering
> > > >
> > > > Bob,
> > > >
> > > > You should also be seeing this error message in the output:
> > > >
> > > > ERROR : do_job_filter() -> this function may only be called
when
> > > > using
> > > > the* -dump_row *option in the job command line: -job filter
> > > > -line_type MPR -mask_sid baltic1p5km.txt
> > > >
> > > > The output for the "filter" job is specified by the "-
dump_row"
> > > > option, not the "-out_stat" option.
> > > >
> > > > Please try running:
> > > >
> > > > */h/MET/bin/stat_analysis \*
> > > > *-lookin
> > > >
/h/data/global/WXQC/data/met/p**tstat/mdlob_pairs/global_17km/**CB
> > > > /t
> > > > es
> > > > t
> > > > \*
> > > > *-job filter -interp_mthd NBRHD_SQUARE -line_type MPR \*
> > > > *-mask_sid /h/data/global/WXQC/data/met/sid/baltic1p5km.**txt
\*
> > > > *-dump_row
> > > >
/h/data/global/WXQC/data/met/s**ummary/CB_global_17km_le3000_d*
> > > > *ump.stat*
> > > >
> > > > Does that work?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Fri, Aug 16, 2019 at 2:43 PM robert.craig.2 at us.af.mil via
RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Fri Aug 16 14:43:07 2019: Request 91563 was acted upon.
> > > > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > > > Queue: met_help
> > > > > Subject: FW: Stat Anal Filtering
> > > > > Owner: Nobody
> > > > > Requestors: robert.craig.2 at us.af.mil
> > > > > Status: new
> > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > > > > Sent: Friday, August 16, 2019 3:04 PM
> > > > > To: met_help at ucar.edu
> > > > > Subject: Stat Anal Filtering
> > > > >
> > > > >
> > > > >
> > > > > John, I am having trouble trying to filter a stat file using
the
> > > > > following command line:
> > > > >
> > > > >
> > > > >
> > > > > /h/MET/bin/stat_analysis -lookin
> > > > >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/t
> > > > > es t -job filter -interp_mthd NBRHD_SQUARE -line_type MPR
> > > > > -mask_sid /h/data/global/ WXQC/data/met/sid/baltic1p5km.txt
> > > > > -out_stat
> > > > >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.
> > > > > st
> > > > > at
> > > > >
> > > > >
> > > > >
> > > > > I also tried:
> > > > >
> > > > >
> > > > >
> > > > > /h/MET/bin/stat_analysis -lookin
> > > > >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/t
> > > > > es t -job filter -line_type MPR -column_str OBS_SID
> > > > > /h/data/global/WXQC/data/met/si d/baltic1p5km.txt -dump_row
> > > > >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.
> > > > > st
> > > > > at
> > > > >
> > > > >
> > > > >
> > > > > The goal is to reduce the stat file to just those mpr lines
that
> > > > > contain station ids in the file baltic1p5km.txt and
> > > > >
> > > > > The data file I am trying to filter is attached. I have
tried
> > > > > lots of combinations of things from the users manual to
filter
> > > > > the data but keep getting the following error:
> > > > >
> > > > >
> > > > >
> > > > > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> > > > >
/h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> > > > >
> > > > > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > > > > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> > > > >
> > > > > DEBUG 1: Default Config File:
> > > > > /h/MET/share/met/config/STATAnalysisConfig_default
> > > > >
> > > > > DEBUG 1: Station ID Mask:
> > > > > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
> > > > >
> > > > > DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162
points
> > > > >
> > > > > ERROR :
> > > > >
> > > > > " in job command line: -job filter -interp_mthd NBRHD_SQUARE
> > > > > -line_type MPR -mask_sid
> > > > > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -out_stat
> > > > >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.
> > > > > st
> > > > > at
> > > > >
> > > > > ERROR :
> > > > >
> > > > > ERROR :
> > > > >
> > > > > ERROR : main() -> encountered an error value of 1. Calling
> > > > > clean_up() and
> > > > > usage() before exiting.
> > > > >
> > > > >
> > > > >
> > > > > What I am doing wrong?
> > > > >
> > > > >
> > > > >
> > > > > Thanks
> > > > >
> > > > > Bob
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: FW: Stat Anal Filtering
From: John Halley Gotway
Time: Wed Aug 21 16:52:44 2019
Bob,
Well I tried, but wasn't able to replicate the problem you described
by
adding extra whitespace. So it's nothing that I know how to fix.
I'll go
ahead and resolve this ticket now.
Thanks,
John
On Tue, Aug 20, 2019 at 11:43 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Bob,
>
> That’s great info. Thanks for letting me know. I’ll try making the
same
> change locally in hopes of replicating that same error. If I’m able
to, I
> should be able to fix it for the next release.
>
> Thanks,
> John
>
> On Tue, Aug 20, 2019 at 1:51 PM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563 >
>>
>> John, I believe I found the problem. Taking your advice, I tried
it
>> without the -mask_sid command and it worked. So checking the file
I was
>> calling using -mask_sid, I found I had one extra space at the end
of the
>> line of station ids. Removing that space allowed the code to run
without
>> error.
>>
>> Thanks
>> Bob
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT <met_help at ucar.edu>
>> Sent: Monday, August 19, 2019 12:29 PM
>> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
>> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW: Stat
Anal
>> Filtering
>>
>> Bob,
>>
>> OK, I got the file when you sent it to johnhg at ucar.edu. I wonder
if
>> perhaps the ones to met-help get treated as spam?
>>
>> Anyway, below I've listed the results of my successful run of met-
8.1
>> stat_analysis. It reads 31,674 input lines and kept 414 of them
which meet
>> the filtering criteria. So I'm not getting the error you're
getting.
>>
>> If you run the exact same command but with the "-mask_sid" option
>> removed, does that work? Hopefully we can narrow down a bit more
what's
>> actually causing the problem.
>>
>> FYI, here's what I mean about truncated error message... If I rerun
>> WITHOUT the "-dump_row" option, stat_analysis correctly errors out
with the
>> following message:
>> ERROR : do_job_filter() -> this function may only be called when
using
>> the -dump_row option in the job command line: -job filter
-interp_mthd
>> NBRHD_SQUARE -line_type MPR -mask_sid baltic1p5km.txt
>>
>> You're seeing a similar message, BUT the first part of that message
is
>> missing from your output (do_job_filter() -> this function may only
be
>> called when using the -dump_row option). That's why I called it
>> truncated.
>>
>> John
>>
>> /usr/local/met-8.1/bin/stat_analysis -lookin
>> point_stat_global_17km_all_240000L_20180301_000000V.stat -job
filter
>> -interp_mthd NBRHD_SQUARE -line_type MPR -mask_sid baltic1p5km.txt
>> -dump_row CB_global_17km_le3000_dump.stat DEBUG 1: Default Config
File:
>> /usr/local/met-8.1/share/met/config/STATAnalysisConfig_default
>> DEBUG 1: Station ID Mask: baltic1p5km.txt DEBUG 2: Parsed Station
ID
>> Mask: baltic1p5km containing 673 points DEBUG 2: Processing 1 STAT
files.
>> DEBUG 2: STAT Lines read = 31674
>> DEBUG 2: STAT Lines retained = 414
>> DEBUG 2:
>> DEBUG 2: Processing Job 1: -job filter -interp_mthd NBRHD_SQUARE
>> -line_type MPR -dump_row CB_global_17km_le3000_dump.stat -mask_sid
>> baltic1p5km.txt DEBUG 1: Creating dump row output file
>> "CB_global_17km_le3000_dump.stat"
>> FILTER: -job filter -interp_mthd NBRHD_SQUARE -line_type MPR
>> -dump_row CB_global_17km_le3000_dump.stat -mask_sid baltic1p5km.txt
DEBUG
>> 2: Job 1 used 414 out of 414 STAT lines.
>>
>>
>> On Mon, Aug 19, 2019 at 10:17 AM robert.craig.2 at us.af.mil via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563 >
>> >
>> > John, I noticed the email was going to methelp instead of you.
So I
>> will
>> > try on last time going to johnhg at ucar.edu. I tried the dos2unix
>> command
>> > but got the same error. Not sure what you mean by a truncated
error
>> > message. I changed the verbosity to 4 and that is below:
>> >
>> > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
>> > /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
>> > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
>> > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
>> > DEBUG 1: Default Config File:
>> > /h/MET/share/met/config/STATAnalysisConfig_default
>> > DEBUG 4: Default Job from the config file: "(nul)"
>> > DEBUG 4: Amending default job with command line options: "-job
filter
>> > -interp_mthd NBRHD_SQUARE -line_type MPR -mask_sid
>> > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
>> >
/h/"ata/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
>> > DEBUG 1: Station ID Mask:
>> > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
>> > DEBUG 4: parse_sid_mask() -> parsing station ID masking file
>> > "/h/data/global/WXQC/data/met/sid/baltic1p5km.txt"
>> > DEBUG 4: parse_sid_mask() -> parsed 162 station ID's for the
"baltic1p5"
>> > mask from file "/h/data/global/WXQC/data/met/sid/baltic1p5km.txt"
>> > DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162 points
ERROR
>> > :
>> > " in job command line: -job filter -interp_mthd NBRHD_SQUARE
>> > -line_type MPR -mask_sid
>> > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
>> >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
>> > ERROR :
>> > ERROR :
>> > ERROR : main() -> encountered an error value of 1. Calling
>> > clean_up() and usage() before exiting.
>> > ERROR :
>> >
>> > *** Model Evaluation Tools (METV8.1) ***
>> >
>> > Usage: stat_analysis
>> > -lookin path
>> > [-out file]
>> > [-tmp_dir path]
>> > [-log file]
>> > [-v level]
>> > [-config config_file] | [JOB COMMAND LINE]
>> >
>> > where "-lookin path" specifies a STAT file or top-level
>> > directory containing STAT files. It allows the use of wildcards
(at
>> > least one required).
>> > "-out file" specifies a file to which output
should be
>> > written rather than the screen (optional).
>> > "-tmp_dir path" specifies the directory into
which
>> > temporary files should be written (optional).
>> > "-log file" outputs log messages to the specified
file
>> > (optional).
>> > "-v level" overrides the default level of logging
(4)
>> > (optional).
>> > "-config config_file" specifies a STATAnalysis
config
>> > file containing STATAnalysis jobs to be run.
>> > "JOB COMMAND LINE" specifies all the arguments
>> > necessary to perform a single STATAnalysis job.
>> >
>> > NOTE: Refer to a STATAnalysis config file for details on
how
>> > to specify the JOB COMMAND LINE.
>> >
>> >
>> > As for the option to limit the SID's in stat_analysis, option 2
sounds
>> > good. I CCed Matt on this incase he has an opinion on this
option.
>> >
>> > Thanks
>> > Bob
>> >
>> > -----Original Message-----
>> > From: John Halley Gotway via RT <met_help at ucar.edu>
>> > Sent: Monday, August 19, 2019 10:57 AM
>> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
>> > <robert.craig.2 at us.af.mil>
>> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW:
Stat
>> > Anal Filtering
>> >
>> > Bob,
>> >
>> > Still no luck. I'm not seeing anything from ARMDEC coming to my
email
>> > address "johnhg at ucar.edu".
>> >
>> > One big mystery is the error message you sent to me:
>> > ERROR :
>> > " in job command line: -job filter -interp_mthd NBRHD_SQUARE
>> > -line_type MPR -mask_sid
>> > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
>> >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.stat
>> > ERROR :
>> >
>> > The whole first part of it is truncated. There should be an
actual
>> > message before it says " in job command...". So why/how is that
>> > happening? One additional thing to consider... you use Windows
and
>> > that inserts a ^M character at the end of every line of text as
the
>> > line break marker. In the past, those ^M characters have caused
>> > trouble. I'm not sure they are here, but it's always a
possibility.
>> > You could try running that ascii file through the dos2unix
utility
>> > which strips off those ^M
>> > characters:
>> >
>> > dos2unix baltic1p5km.txt
>> >
>> > And then see if anything changes.
>> >
>> > For your second question, no, there currently isn't any way to
tell
>> > STAT-Analysis which stations to skip. The closest thing is the
>> > "-column_str" option.
>> > For example, "-column MPR -column_str OBS_SID KDEN,690150" tells
>> > STAT-Analysis to only use lines MPR lines where the OBS_SID
column
>> > contains "KDEN" or "690150". But there is no negation for this
>> > option. We can't tell it which strings to exclude.
>> >
>> > For numeric values, this is easy. The "-column_thresh" option
defines
>> > which numbers to keep. For example, "-column MPR -column_thresh
FCST
>> > gt100" keeps only MPR lines where the FCST value is > 100.
Negating
>> > this is easy... just switch the threshold "-column_thresh FCST
le100".
>> >
>> > We could potentially enhance stat_analysis (and tc_stat) to
support
>> > this in the future. I see a few implementation options...
>> > (1) Explicitly add a "-sid_exc" option which only works when
>> > processing MPR or ORANK lines.
>> > (2) Add a new "-column_str_exc" option to do the opposite of the
>> > existing "-column_str" option. This is generally applied to any
>> > column containing strings.
>> > (3) Just use the existing "-column_str" option but switch to
>> > interpreting the strings as regular expressions, and define the
>> > negation as a regular expression.
>> >
>> > I think I lean toward option (2). While (3) is nice, regular
>> > expressions can be difficult to set up well and produce
unintended
>> > matches. Please let me know if you'd like me to write up this as
a
>> > potential GitHub development issue.
>> >
>> > For now, I'd recommend just running egrep prior to running
>> > stat-analysis to throw out the stations you don't want. For
example:
>> >
>> > egrep -v " 690150 | 710630 "
>> > point_stat_meps_conus_480000L_20190806_000000V.stat > filter.stat
>> >
>> > This will discard those 2 stations from that file and write the
output
>> > to a new file named "filter.stat".
>> >
>> > Thanks,
>> > John
>> >
>> > On Mon, Aug 19, 2019 at 9:01 AM robert.craig.2 at us.af.mil via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563 >
>> > >
>> > > John, a related question. In stat_analysis I would like to
exclude
>> obs
>> > > from being considered - is there a command switch for that? I
tried
>> > > -sid_exc but that didn't work.
>> > >
>> > > Thanks
>> > > Bob
>> > >
>> > > -----Original Message-----
>> > > From: John Halley Gotway via RT <met_help at ucar.edu>
>> > > Sent: Friday, August 16, 2019 5:19 PM
>> > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
>> > > <robert.craig.2 at us.af.mil>
>> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW:
Stat
>> > > Anal Filtering
>> > >
>> > > Bob,
>> > >
>> > > I never saw an email come through from amerdec. So to test, I
used
>> > > some sample data that's generated when you run 'make test'.
>> > >
>> > > *cd met-8.1*
>> > > *bin/stat_analysis -lookin
>> > > out/point_stat/point_stat_360000L_20070331_120000V.stat \* *-
job
>> > > filter -line_type MPR -interp_mthd NEAREST -dump_row
filter.stat*
>> > >
>> > > DEBUG 2: STAT Lines read = 28140
>> > > DEBUG 2: STAT Lines retained = 9288
>> > >
>> > > And then I picked out the first 500 unique station id names
from the
>> > > MRP lines and wrote them to a file:
>> > >
>> > > *echo "SID_MASK" > my_sid_list.txt*
>> > > *grep MPR
out/point_stat/point_stat_360000L_20070331_120000V.stat |
>> > > awk '{print $27}' | sort -u | head -500 >> my_sid_list.txt*
>> > >
>> > > And then I reran using that station ID list:
>> > >
>> > >
>> > > *bin/stat_analysis -lookin
>> > > out/point_stat/point_stat_360000L_20070331_120000V.stat -job
filter
>> > > -line_type MPR -interp_mthd NEAREST -dump_row filter.stat
-mask_sid
>> > > my_sid_list.txt*
>> > >
>> > > DEBUG 2: STAT Lines read = 28140
>> > > DEBUG 2: STAT Lines retained = 1451
>> > >
>> > > The 9288 MPR lines for NEAREST were reduced down to 1451. So
this
>> > > functionality *should* be working. But I'm not sure where the
>> > > disconnect is.
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > > On Fri, Aug 16, 2019 at 3:00 PM robert.craig.2 at us.af.mil via RT
<
>> > > met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563
>
>> > > >
>> > > > John, that gives me the error:
>> > > >
>> > > > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
>> > > > /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
>> > > > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
>> > > > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
>> > > > DEBUG 1: Default Config File:
>> > > > /h/MET/share/met/config/STATAnalysisConfig_default
>> > > > DEBUG 1: Station ID Mask:
>> > > > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
>> > > > DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162
points
>> > > > ERROR
>> > > > :
>> > > > " in job command line: -job filter -interp_mthd NBRHD_SQUARE
>> > > > -line_type MPR -mask_sid
>> > > > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -dump_row
>> > > >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.st
>> > > > at
>> > > > ERROR :
>> > > > ERROR :
>> > > > ERROR : main() -> encountered an error value of 1. Calling
>> > > > clean_up() and usage() before exiting.
>> > > > ERROR :
>> > > >
>> > > > I sent you the data file via amerdec if you would like to try
it.
>> > > >
>> > > > Thanks
>> > > > Bob
>> > > >
>> > > > -----Original Message-----
>> > > > From: John Halley Gotway via RT <met_help at ucar.edu>
>> > > > Sent: Friday, August 16, 2019 3:53 PM
>> > > > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
>> > > > <robert.craig.2 at us.af.mil>
>> > > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #91563] FW:
Stat
>> > > > Anal Filtering
>> > > >
>> > > > Bob,
>> > > >
>> > > > You should also be seeing this error message in the output:
>> > > >
>> > > > ERROR : do_job_filter() -> this function may only be called
when
>> > > > using
>> > > > the* -dump_row *option in the job command line: -job filter
>> > > > -line_type MPR -mask_sid baltic1p5km.txt
>> > > >
>> > > > The output for the "filter" job is specified by the "-
dump_row"
>> > > > option, not the "-out_stat" option.
>> > > >
>> > > > Please try running:
>> > > >
>> > > > */h/MET/bin/stat_analysis \*
>> > > > *-lookin
>> > > >
/h/data/global/WXQC/data/met/p**tstat/mdlob_pairs/global_17km/**CB
>> > > > /t
>> > > > es
>> > > > t
>> > > > \*
>> > > > *-job filter -interp_mthd NBRHD_SQUARE -line_type MPR \*
>> > > > *-mask_sid /h/data/global/WXQC/data/met/sid/baltic1p5km.**txt
\*
>> > > > *-dump_row
>> > > >
/h/data/global/WXQC/data/met/s**ummary/CB_global_17km_le3000_d*
>> > > > *ump.stat*
>> > > >
>> > > > Does that work?
>> > > >
>> > > > Thanks,
>> > > > John
>> > > >
>> > > > On Fri, Aug 16, 2019 at 2:43 PM robert.craig.2 at us.af.mil via
RT <
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > > >
>> > > > > Fri Aug 16 14:43:07 2019: Request 91563 was acted upon.
>> > > > > Transaction: Ticket created by robert.craig.2 at us.af.mil
>> > > > > Queue: met_help
>> > > > > Subject: FW: Stat Anal Filtering
>> > > > > Owner: Nobody
>> > > > > Requestors: robert.craig.2 at us.af.mil
>> > > > > Status: new
>> > > > > Ticket <URL:
>> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91563
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
>> > > > > Sent: Friday, August 16, 2019 3:04 PM
>> > > > > To: met_help at ucar.edu
>> > > > > Subject: Stat Anal Filtering
>> > > > >
>> > > > >
>> > > > >
>> > > > > John, I am having trouble trying to filter a stat file
using the
>> > > > > following command line:
>> > > > >
>> > > > >
>> > > > >
>> > > > > /h/MET/bin/stat_analysis -lookin
>> > > > >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/t
>> > > > > es t -job filter -interp_mthd NBRHD_SQUARE -line_type MPR
>> > > > > -mask_sid /h/data/global/ WXQC/data/met/sid/baltic1p5km.txt
>> > > > > -out_stat
>> > > > >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.
>> > > > > st
>> > > > > at
>> > > > >
>> > > > >
>> > > > >
>> > > > > I also tried:
>> > > > >
>> > > > >
>> > > > >
>> > > > > /h/MET/bin/stat_analysis -lookin
>> > > > >
/h/data/global/WXQC/data/met/ptstat/mdlob_pairs/global_17km/CB/t
>> > > > > es t -job filter -line_type MPR -column_str OBS_SID
>> > > > > /h/data/global/WXQC/data/met/si d/baltic1p5km.txt -dump_row
>> > > > >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.
>> > > > > st
>> > > > > at
>> > > > >
>> > > > >
>> > > > >
>> > > > > The goal is to reduce the stat file to just those mpr lines
that
>> > > > > contain station ids in the file baltic1p5km.txt and
>> > > > >
>> > > > > The data file I am trying to filter is attached. I have
tried
>> > > > > lots of combinations of things from the users manual to
filter
>> > > > > the data but keep getting the following error:
>> > > > >
>> > > > >
>> > > > >
>> > > > > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
>> > > > >
/h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
>> > > > >
>> > > > > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
>> > > > > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
>> > > > >
>> > > > > DEBUG 1: Default Config File:
>> > > > > /h/MET/share/met/config/STATAnalysisConfig_default
>> > > > >
>> > > > > DEBUG 1: Station ID Mask:
>> > > > > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt
>> > > > >
>> > > > > DEBUG 2: Parsed Station ID Mask: baltic1p5 containing 162
points
>> > > > >
>> > > > > ERROR :
>> > > > >
>> > > > > " in job command line: -job filter -interp_mthd
NBRHD_SQUARE
>> > > > > -line_type MPR -mask_sid
>> > > > > /h/data/global/WXQC/data/met/sid/baltic1p5km.txt -out_stat
>> > > > >
/h/data/global/WXQC/data/met/summary/CB_global_17km_le3000_dump.
>> > > > > st
>> > > > > at
>> > > > >
>> > > > > ERROR :
>> > > > >
>> > > > > ERROR :
>> > > > >
>> > > > > ERROR : main() -> encountered an error value of 1.
Calling
>> > > > > clean_up() and
>> > > > > usage() before exiting.
>> > > > >
>> > > > >
>> > > > >
>> > > > > What I am doing wrong?
>> > > > >
>> > > > >
>> > > > >
>> > > > > Thanks
>> > > > >
>> > > > > Bob
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>>
>>
------------------------------------------------
More information about the Met_help
mailing list