[Met_help] [rt.rap.ucar.edu #99168] History for Filtering in Stat Anal

John Halley Gotway via RT met_help at ucar.edu
Fri Mar 26 16:38:21 MDT 2021


----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

John, we have a case where a bad RAOB got into our mdlob_pairs and partial sums.  In an attempt to filter the bad data using the MAE field int he SL1L2 files, I am using the following stat anal job:

/h/MET/bin/stat_analysis -lookin /h/data/global/WXQC/data/met/ptstat/stats/galwem/HT -out_stat /h/data/global/WXQC/data/met/ptstat/stats/monthly/stat_anal_sl1l2_galwem_HT_UPR_20201201_20210228.txt -tmp_dir /h/data/global/WXQC/data/met/tmp -job aggregate_stat -line_type SL1L2 -out_line_type CNT -fcst_init_beg 20201201_000000 -fcst_init_end 20210228_180000 -by fcst_lead -fcst_lev P1000,P925,P850,P700,P500,P400,P300,P250,P200,P150,P100 -by FCST_LEV -by model -vx_mask FULL,uss,usn,usa,usp,usc,use -by vx_mask -by fcst_init_hour -column_str OBS_LEV P400, -column_max MAE 17 -column_str OBS_LEV P300, -column_max MAE 12 -column_str OBS_LEV P150, -column_max MAE 14 -config /h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_bootstrap_aggstat -v 4.0

The desire is to filter out MAEs for the 400mb level >17, 300mb level > 12, and 150mb level > 14.  Unfortunately, the resulting file has only P400, P300, and P150 in it.  Am I interpeting the logic here correctly?

Thanks



----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Filtering in Stat Anal
From: John Halley Gotway
Time: Mon Mar 22 20:28:01 2021

Bob,

All of the options specified for a single job are grouped together
into one
filter. Since you requested OBS_LEVEL = P150, P300, P400 in the job,
you
get the union of those in the output.

Unfortunately you’d need to do this with 3 different jobs, one for
each
level/threshold combo.

If it were me, instead of sticking together stat-analysis jobs, I’d
probably write a shell or Python script to do this filtering.

Sorry I don’t have an easier answer for you.

John

On Mon, Mar 22, 2021 at 10:33 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:

>
> Mon Mar 22 10:33:09 2021: Request 99168 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
>        Queue: met_help
>      Subject: Filtering in Stat Anal
>        Owner: Nobody
>   Requestors: robert.craig.2 at us.af.mil
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99168 >
>
>
> John, we have a case where a bad RAOB got into our mdlob_pairs and
partial
> sums.  In an attempt to filter the bad data using the MAE field int
he
> SL1L2 files, I am using the following stat anal job:
>
> /h/MET/bin/stat_analysis -lookin
> /h/data/global/WXQC/data/met/ptstat/stats/galwem/HT -out_stat
>
/h/data/global/WXQC/data/met/ptstat/stats/monthly/stat_anal_sl1l2_galwem_HT_UPR_20201201_20210228.txt
> -tmp_dir /h/data/global/WXQC/data/met/tmp -job aggregate_stat
-line_type
> SL1L2 -out_line_type CNT -fcst_init_beg 20201201_000000
-fcst_init_end
> 20210228_180000 -by fcst_lead -fcst_lev
> P1000,P925,P850,P700,P500,P400,P300,P250,P200,P150,P100 -by FCST_LEV
-by
> model -vx_mask FULL,uss,usn,usa,usp,usc,use -by vx_mask -by
fcst_init_hour
> -column_str OBS_LEV P400, -column_max MAE 17 -column_str OBS_LEV
P300,
> -column_max MAE 12 -column_str OBS_LEV P150, -column_max MAE 14
-config
>
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_bootstrap_aggstat
> -v 4.0
>
> The desire is to filter out MAEs for the 400mb level >17, 300mb
level >
> 12, and 150mb level > 14.  Unfortunately, the resulting file has
only P400,
> P300, and P150 in it.  Am I interpeting the logic here correctly?
>
> Thanks
>
>
>

------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #99168] Filtering in Stat Anal
From: robert.craig.2 at us.af.mil
Time: Tue Mar 23 07:18:07 2021


Thanks for that.  One reason this can come up is that there is not a
ob exclude option in the stat_analysis job.  You can specify which obs
to use but you can't exclude an ob (at least that is my understanding
for met 8.1 - maybe remedied in later versions).   You can exclude in
the point stat config file but sometimes a bad level in a ob can occur
and you don't discover this until you are plotting the data.

Bob
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Monday, March 22, 2021 9:28 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXD
<robert.craig.2 at us.af.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #99168] Filtering in
Stat Anal

Bob,

All of the options specified for a single job are grouped together
into one filter. Since you requested OBS_LEVEL = P150, P300, P400 in
the job, you get the union of those in the output.

Unfortunately you’d need to do this with 3 different jobs, one for
each level/threshold combo.

If it were me, instead of sticking together stat-analysis jobs, I’d
probably write a shell or Python script to do this filtering.

Sorry I don’t have an easier answer for you.

John

On Mon, Mar 22, 2021 at 10:33 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:

>
> Mon Mar 22 10:33:09 2021: Request 99168 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
>        Queue: met_help
>      Subject: Filtering in Stat Anal
>        Owner: Nobody
>   Requestors: robert.craig.2 at us.af.mil
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99168
> >
>
>
> John, we have a case where a bad RAOB got into our mdlob_pairs and
> partial sums.  In an attempt to filter the bad data using the MAE
> field int he
> SL1L2 files, I am using the following stat anal job:
>
> /h/MET/bin/stat_analysis -lookin
> /h/data/global/WXQC/data/met/ptstat/stats/galwem/HT -out_stat
>
/h/data/global/WXQC/data/met/ptstat/stats/monthly/stat_anal_sl1l2_galw
> em_HT_UPR_20201201_20210228.txt -tmp_dir
> /h/data/global/WXQC/data/met/tmp -job aggregate_stat -line_type
> SL1L2 -out_line_type CNT -fcst_init_beg 20201201_000000
-fcst_init_end
> 20210228_180000 -by fcst_lead -fcst_lev
> P1000,P925,P850,P700,P500,P400,P300,P250,P200,P150,P100 -by FCST_LEV
> -by model -vx_mask FULL,uss,usn,usa,usp,usc,use -by vx_mask -by
> fcst_init_hour -column_str OBS_LEV P400, -column_max MAE 17
> -column_str OBS_LEV P300, -column_max MAE 12 -column_str OBS_LEV
P150,
> -column_max MAE 14 -config
>
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_boot
> strap_aggstat
> -v 4.0
>
> The desire is to filter out MAEs for the 400mb level >17, 300mb
level
> > 12, and 150mb level > 14.  Unfortunately, the resulting file has
> only P400, P300, and P150 in it.  Am I interpeting the logic here
correctly?
>
> Thanks
>
>
>



------------------------------------------------
Subject: FW: [Non-DoD Source] Re: [rt.rap.ucar.edu #99168] Filtering in Stat Anal
From: robert.craig.2 at us.af.mil
Time: Tue Mar 23 07:59:35 2021

I will go the python script route on this.

Thanks

-----Original Message-----
From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXD
Sent: Tuesday, March 23, 2021 8:17 AM
To: met_help at ucar.edu
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #99168] Filtering
in Stat Anal


Thanks for that.  One reason this can come up is that there is not a
ob exclude option in the stat_analysis job.  You can specify which obs
to use but you can't exclude an ob (at least that is my understanding
for met 8.1 - maybe remedied in later versions).   You can exclude in
the point stat config file but sometimes a bad level in a ob can occur
and you don't discover this until you are plotting the data.

Bob
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Monday, March 22, 2021 9:28 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXD
<robert.craig.2 at us.af.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #99168] Filtering in
Stat Anal

Bob,

All of the options specified for a single job are grouped together
into one filter. Since you requested OBS_LEVEL = P150, P300, P400 in
the job, you get the union of those in the output.

Unfortunately you’d need to do this with 3 different jobs, one for
each level/threshold combo.

If it were me, instead of sticking together stat-analysis jobs, I’d
probably write a shell or Python script to do this filtering.

Sorry I don’t have an easier answer for you.

John

On Mon, Mar 22, 2021 at 10:33 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:

>
> Mon Mar 22 10:33:09 2021: Request 99168 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
>        Queue: met_help
>      Subject: Filtering in Stat Anal
>        Owner: Nobody
>   Requestors: robert.craig.2 at us.af.mil
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99168
> >
>
>
> John, we have a case where a bad RAOB got into our mdlob_pairs and
> partial sums.  In an attempt to filter the bad data using the MAE
> field int he
> SL1L2 files, I am using the following stat anal job:
>
> /h/MET/bin/stat_analysis -lookin
> /h/data/global/WXQC/data/met/ptstat/stats/galwem/HT -out_stat
>
/h/data/global/WXQC/data/met/ptstat/stats/monthly/stat_anal_sl1l2_galw
> em_HT_UPR_20201201_20210228.txt -tmp_dir
> /h/data/global/WXQC/data/met/tmp -job aggregate_stat -line_type
> SL1L2 -out_line_type CNT -fcst_init_beg 20201201_000000
-fcst_init_end
> 20210228_180000 -by fcst_lead -fcst_lev
> P1000,P925,P850,P700,P500,P400,P300,P250,P200,P150,P100 -by FCST_LEV
> -by model -vx_mask FULL,uss,usn,usa,usp,usc,use -by vx_mask -by
> fcst_init_hour -column_str OBS_LEV P400, -column_max MAE 17
> -column_str OBS_LEV P300, -column_max MAE 12 -column_str OBS_LEV
P150,
> -column_max MAE 14 -config
>
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_boot
> strap_aggstat
> -v 4.0
>
> The desire is to filter out MAEs for the 400mb level >17, 300mb
level
> > 12, and 150mb level > 14.  Unfortunately, the resulting file has
> only P400, P300, and P150 in it.  Am I interpeting the logic here
correctly?
>
> Thanks
>
>
>



------------------------------------------------
Subject: Filtering in Stat Anal
From: John Halley Gotway
Time: Tue Mar 23 15:56:11 2021

Bob,

Aha, well that certainly would be a very easy addition to the
stat_analysis
and tc_stat tools! There currently is a filtering option named
“column_str”. Setting “-line_type MPR -column_str OBS_SID KDEN,KTCL”
would
only use MPR lines for those 2 stations.

In addition, point_stat and ensemble_stat have a “sid_exc” option to
specify which stations to skip.

We could add a stat_analysis and tc_stat option named “-column_exc” to
define which strings to EXCLUDE instead of which to include.

Do you think you’d find that enhancement to be useful? That’s such an
easy
one, I might be able to squeak it in for the upcoming 10.0.0 release.

Thanks,
John

On Tue, Mar 23, 2021 at 8:00 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99168 >
>
> I will go the python script route on this.
>
> Thanks
>
> -----Original Message-----
> From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXD
> Sent: Tuesday, March 23, 2021 8:17 AM
> To: met_help at ucar.edu
> Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #99168] Filtering
in
> Stat Anal
>
>
> Thanks for that.  One reason this can come up is that there is not a
ob
> exclude option in the stat_analysis job.  You can specify which obs
to use
> but you can't exclude an ob (at least that is my understanding for
met 8.1
> - maybe remedied in later versions).   You can exclude in the point
stat
> config file but sometimes a bad level in a ob can occur and you
don't
> discover this until you are plotting the data.
>
> Bob
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Monday, March 22, 2021 9:28 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXD
<robert.craig.2 at us.af.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #99168] Filtering in
Stat
> Anal
>
> Bob,
>
> All of the options specified for a single job are grouped together
into
> one filter. Since you requested OBS_LEVEL = P150, P300, P400 in the
job,
> you get the union of those in the output.
>
> Unfortunately you’d need to do this with 3 different jobs, one for
each
> level/threshold combo.
>
> If it were me, instead of sticking together stat-analysis jobs, I’d
> probably write a shell or Python script to do this filtering.
>
> Sorry I don’t have an easier answer for you.
>
> John
>
> On Mon, Mar 22, 2021 at 10:33 AM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Mar 22 10:33:09 2021: Request 99168 was acted upon.
> > Transaction: Ticket created by robert.craig.2 at us.af.mil
> >        Queue: met_help
> >      Subject: Filtering in Stat Anal
> >        Owner: Nobody
> >   Requestors: robert.craig.2 at us.af.mil
> >       Status: new
> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99168
> > >
> >
> >
> > John, we have a case where a bad RAOB got into our mdlob_pairs and
> > partial sums.  In an attempt to filter the bad data using the MAE
> > field int he
> > SL1L2 files, I am using the following stat anal job:
> >
> > /h/MET/bin/stat_analysis -lookin
> > /h/data/global/WXQC/data/met/ptstat/stats/galwem/HT -out_stat
> >
/h/data/global/WXQC/data/met/ptstat/stats/monthly/stat_anal_sl1l2_galw
> > em_HT_UPR_20201201_20210228.txt -tmp_dir
> > /h/data/global/WXQC/data/met/tmp -job aggregate_stat -line_type
> > SL1L2 -out_line_type CNT -fcst_init_beg 20201201_000000
-fcst_init_end
> > 20210228_180000 -by fcst_lead -fcst_lev
> > P1000,P925,P850,P700,P500,P400,P300,P250,P200,P150,P100 -by
FCST_LEV
> > -by model -vx_mask FULL,uss,usn,usa,usp,usc,use -by vx_mask -by
> > fcst_init_hour -column_str OBS_LEV P400, -column_max MAE 17
> > -column_str OBS_LEV P300, -column_max MAE 12 -column_str OBS_LEV
P150,
> > -column_max MAE 14 -config
> >
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_boot
> > strap_aggstat
> > -v 4.0
> >
> > The desire is to filter out MAEs for the 400mb level >17, 300mb
level
> > > 12, and 150mb level > 14.  Unfortunately, the resulting file has
> > only P400, P300, and P150 in it.  Am I interpeting the logic here
> correctly?
> >
> > Thanks
> >
> >
> >
>
>
>
>

------------------------------------------------
Subject: Filtering in Stat Anal
From: robert.craig.2 at us.af.mil
Time: Wed Mar 24 07:22:39 2021

Yes, There are times when a bad ob gets into the model/ob pairs data
and there needs to be a way of not using it in stat analysis.  So a
-column exclude would be very useful.

________________________________
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Tuesday, March 23, 2021 4:56 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXD
<robert.craig.2 at us.af.mil>
Subject: Re: FW: [Non-DoD Source] Re: [rt.rap.ucar.edu #99168]
Filtering in Stat Anal

Bob,

Aha, well that certainly would be a very easy addition to the
stat_analysis
and tc_stat tools! There currently is a filtering option named
“column_str”. Setting “-line_type MPR -column_str OBS_SID KDEN,KTCL”
would
only use MPR lines for those 2 stations.

In addition, point_stat and ensemble_stat have a “sid_exc” option to
specify which stations to skip.

We could add a stat_analysis and tc_stat option named “-column_exc” to
define which strings to EXCLUDE instead of which to include.

Do you think you’d find that enhancement to be useful? That’s such an
easy
one, I might be able to squeak it in for the upcoming 10.0.0 release.

Thanks,
John

On Tue, Mar 23, 2021 at 8:00 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99168 >
>
> I will go the python script route on this.
>
> Thanks
>
> -----Original Message-----
> From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXD
> Sent: Tuesday, March 23, 2021 8:17 AM
> To: met_help at ucar.edu
> Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #99168] Filtering
in
> Stat Anal
>
>
> Thanks for that.  One reason this can come up is that there is not a
ob
> exclude option in the stat_analysis job.  You can specify which obs
to use
> but you can't exclude an ob (at least that is my understanding for
met 8.1
> - maybe remedied in later versions).   You can exclude in the point
stat
> config file but sometimes a bad level in a ob can occur and you
don't
> discover this until you are plotting the data.
>
> Bob
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Monday, March 22, 2021 9:28 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXD
<robert.craig.2 at us.af.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #99168] Filtering in
Stat
> Anal
>
> Bob,
>
> All of the options specified for a single job are grouped together
into
> one filter. Since you requested OBS_LEVEL = P150, P300, P400 in the
job,
> you get the union of those in the output.
>
> Unfortunately you’d need to do this with 3 different jobs, one for
each
> level/threshold combo.
>
> If it were me, instead of sticking together stat-analysis jobs, I’d
> probably write a shell or Python script to do this filtering.
>
> Sorry I don’t have an easier answer for you.
>
> John
>
> On Mon, Mar 22, 2021 at 10:33 AM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Mar 22 10:33:09 2021: Request 99168 was acted upon.
> > Transaction: Ticket created by robert.craig.2 at us.af.mil
> >        Queue: met_help
> >      Subject: Filtering in Stat Anal
> >        Owner: Nobody
> >   Requestors: robert.craig.2 at us.af.mil
> >       Status: new
> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99168
> > >
> >
> >
> > John, we have a case where a bad RAOB got into our mdlob_pairs and
> > partial sums.  In an attempt to filter the bad data using the MAE
> > field int he
> > SL1L2 files, I am using the following stat anal job:
> >
> > /h/MET/bin/stat_analysis -lookin
> > /h/data/global/WXQC/data/met/ptstat/stats/galwem/HT -out_stat
> >
/h/data/global/WXQC/data/met/ptstat/stats/monthly/stat_anal_sl1l2_galw
> > em_HT_UPR_20201201_20210228.txt -tmp_dir
> > /h/data/global/WXQC/data/met/tmp -job aggregate_stat -line_type
> > SL1L2 -out_line_type CNT -fcst_init_beg 20201201_000000
-fcst_init_end
> > 20210228_180000 -by fcst_lead -fcst_lev
> > P1000,P925,P850,P700,P500,P400,P300,P250,P200,P150,P100 -by
FCST_LEV
> > -by model -vx_mask FULL,uss,usn,usa,usp,usc,use -by vx_mask -by
> > fcst_init_hour -column_str OBS_LEV P400, -column_max MAE 17
> > -column_str OBS_LEV P300, -column_max MAE 12 -column_str OBS_LEV
P150,
> > -column_max MAE 14 -config
> >
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_boot
> > strap_aggstat
> > -v 4.0
> >
> > The desire is to filter out MAEs for the 400mb level >17, 300mb
level
> > > 12, and 150mb level > 14.  Unfortunately, the resulting file has
> > only P400, P300, and P150 in it.  Am I interpeting the logic here
> correctly?
> >
> > Thanks
> >
> >
> >
>
>
>
>


------------------------------------------------
Subject: Filtering in Stat Anal
From: John Halley Gotway
Time: Fri Mar 26 16:37:32 2021

Bob,

FYI, I wrote up this GitHub issue for this feature request:
https://github.com/dtcenter/MET/issues/1733

I'll go ahead and resolve this issue.

Thanks,
John

On Wed, Mar 24, 2021 at 7:22 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99168 >
>
> Yes, There are times when a bad ob gets into the model/ob pairs data
and
> there needs to be a way of not using it in stat analysis.  So a
-column
> exclude would be very useful.
>
> ________________________________
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Tuesday, March 23, 2021 4:56 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXD
<robert.craig.2 at us.af.mil>
> Subject: Re: FW: [Non-DoD Source] Re: [rt.rap.ucar.edu #99168]
Filtering
> in Stat Anal
>
> Bob,
>
> Aha, well that certainly would be a very easy addition to the
stat_analysis
> and tc_stat tools! There currently is a filtering option named
> “column_str”. Setting “-line_type MPR -column_str OBS_SID KDEN,KTCL”
would
> only use MPR lines for those 2 stations.
>
> In addition, point_stat and ensemble_stat have a “sid_exc” option to
> specify which stations to skip.
>
> We could add a stat_analysis and tc_stat option named “-column_exc”
to
> define which strings to EXCLUDE instead of which to include.
>
> Do you think you’d find that enhancement to be useful? That’s such
an easy
> one, I might be able to squeak it in for the upcoming 10.0.0
release.
>
> Thanks,
> John
>
> On Tue, Mar 23, 2021 at 8:00 AM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99168 >
> >
> > I will go the python script route on this.
> >
> > Thanks
> >
> > -----Original Message-----
> > From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXD
> > Sent: Tuesday, March 23, 2021 8:17 AM
> > To: met_help at ucar.edu
> > Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #99168]
Filtering in
> > Stat Anal
> >
> >
> > Thanks for that.  One reason this can come up is that there is not
a ob
> > exclude option in the stat_analysis job.  You can specify which
obs to
> use
> > but you can't exclude an ob (at least that is my understanding for
met
> 8.1
> > - maybe remedied in later versions).   You can exclude in the
point stat
> > config file but sometimes a bad level in a ob can occur and you
don't
> > discover this until you are plotting the data.
> >
> > Bob
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Monday, March 22, 2021 9:28 PM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXD
<robert.craig.2 at us.af.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #99168] Filtering
in Stat
> > Anal
> >
> > Bob,
> >
> > All of the options specified for a single job are grouped together
into
> > one filter. Since you requested OBS_LEVEL = P150, P300, P400 in
the job,
> > you get the union of those in the output.
> >
> > Unfortunately you’d need to do this with 3 different jobs, one for
each
> > level/threshold combo.
> >
> > If it were me, instead of sticking together stat-analysis jobs,
I’d
> > probably write a shell or Python script to do this filtering.
> >
> > Sorry I don’t have an easier answer for you.
> >
> > John
> >
> > On Mon, Mar 22, 2021 at 10:33 AM robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Mon Mar 22 10:33:09 2021: Request 99168 was acted upon.
> > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > >        Queue: met_help
> > >      Subject: Filtering in Stat Anal
> > >        Owner: Nobody
> > >   Requestors: robert.craig.2 at us.af.mil
> > >       Status: new
> > >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99168
> > > >
> > >
> > >
> > > John, we have a case where a bad RAOB got into our mdlob_pairs
and
> > > partial sums.  In an attempt to filter the bad data using the
MAE
> > > field int he
> > > SL1L2 files, I am using the following stat anal job:
> > >
> > > /h/MET/bin/stat_analysis -lookin
> > > /h/data/global/WXQC/data/met/ptstat/stats/galwem/HT -out_stat
> > >
/h/data/global/WXQC/data/met/ptstat/stats/monthly/stat_anal_sl1l2_galw
> > > em_HT_UPR_20201201_20210228.txt -tmp_dir
> > > /h/data/global/WXQC/data/met/tmp -job aggregate_stat -line_type
> > > SL1L2 -out_line_type CNT -fcst_init_beg 20201201_000000
-fcst_init_end
> > > 20210228_180000 -by fcst_lead -fcst_lev
> > > P1000,P925,P850,P700,P500,P400,P300,P250,P200,P150,P100 -by
FCST_LEV
> > > -by model -vx_mask FULL,uss,usn,usa,usp,usc,use -by vx_mask -by
> > > fcst_init_hour -column_str OBS_LEV P400, -column_max MAE 17
> > > -column_str OBS_LEV P300, -column_max MAE 12 -column_str OBS_LEV
P150,
> > > -column_max MAE 14 -config
> > >
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_boot
> > > strap_aggstat
> > > -v 4.0
> > >
> > > The desire is to filter out MAEs for the 400mb level >17, 300mb
level
> > > > 12, and 150mb level > 14.  Unfortunately, the resulting file
has
> > > only P400, P300, and P150 in it.  Am I interpeting the logic
here
> > correctly?
> > >
> > > Thanks
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>

------------------------------------------------


More information about the Met_help mailing list