[Met_help] [rt.rap.ucar.edu #87036] History for threshold of Stat_analysis with CTS
John Halley Gotway via RT
met_help at ucar.edu
Tue Jul 9 12:07:23 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi John,
As we talked before, when we run Stat_Analysis with CTS, we need to set threshold, if the variables like temperature has the threshold as
[<271,>= 271&&<275,>=275&&<283,>=283&&<295,>=295&&<308,>=308&&<310,>=310&&<313,>=313];
does it mean I have to run 8 times of stat_analysis with each threshold?
Thank you
Mandy
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: threshold of Stat_analysis with CTS
From: John Halley Gotway
Time: Wed Sep 19 11:30:20 2018
Mandy,
Instead of specifying "-fcst_thresh '<271'" to select a single
threshold,
please try using this setting instead: "-by FCST_THRESH".
That'll run the same job but separately for each unique value found in
the
FCST_THRESH column. And you can specify multiple columns too...
-by FCST_THRESH,FCST_LEAD
Is that what you're looking for?
Thanks,
John
On Wed, Sep 19, 2018 at 10:15 AM Zhang, Mandy via RT
<met_help at ucar.edu>
wrote:
>
> Wed Sep 19 10:14:40 2018: Request 87036 was acted upon.
> Transaction: Ticket created by Man.Zhang-1 at ou.edu
> Queue: met_help
> Subject: threshold of Stat_analysis with CTS
> Owner: Nobody
> Requestors: Man.Zhang-1 at ou.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036 >
>
>
> Hi John,
>
> As we talked before, when we run Stat_Analysis with CTS, we need
to
> set threshold, if the variables like temperature has the threshold
as
> [<271,>=
>
271&&<275,>=275&&<283,>=283&&<295,>=295&&<308,>=308&&<310,>=310&&<313,>=313];
> does it mean I have to run 8 times of stat_analysis with each
threshold?
> Thank you
> Mandy
>
>
------------------------------------------------
Subject: threshold of Stat_analysis with CTS
From: Zhang, Mandy
Time: Wed Sep 19 15:24:02 2018
Zhang, Mandy has shared OneDrive for Business files with you. To view
them, click the links below.
<https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw>
[https://r1.res.office365.com/owa/prem/images/dc-
txt_20.png]<https://sooners-
my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw>
CTSAPCP_A1_1mon_ge0.1le1.5.txt<https://sooners-
my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw>
<https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg>
[https://r1.res.office365.com/owa/prem/images/dc-
txt_20.png]<https://sooners-
my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg>
CTSAPCP_A1_1mon.txt<https://sooners-
my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg>
Hi John,
I calculate the precipitation by using option "-by FCST_THRESH" and
compared to the result with option "-fcst_thresh ", they show
different value of GSS. the value of "-by FCST_THRESH" is negative and
seems unreasonable.
please find my attachment.
Thank you very much.
Mandy
________________________________
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Wednesday, September 19, 2018 12:30:21 PM
To: Zhang, Mandy
Subject: Re: [rt.rap.ucar.edu #87036] threshold of Stat_analysis with
CTS
Mandy,
Instead of specifying "-fcst_thresh '<271'" to select a single
threshold,
please try using this setting instead: "-by FCST_THRESH".
That'll run the same job but separately for each unique value found in
the
FCST_THRESH column. And you can specify multiple columns too...
-by FCST_THRESH,FCST_LEAD
Is that what you're looking for?
Thanks,
John
On Wed, Sep 19, 2018 at 10:15 AM Zhang, Mandy via RT
<met_help at ucar.edu>
wrote:
>
> Wed Sep 19 10:14:40 2018: Request 87036 was acted upon.
> Transaction: Ticket created by Man.Zhang-1 at ou.edu
> Queue: met_help
> Subject: threshold of Stat_analysis with CTS
> Owner: Nobody
> Requestors: Man.Zhang-1 at ou.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036 >
>
>
> Hi John,
>
> As we talked before, when we run Stat_Analysis with CTS, we need
to
> set threshold, if the variables like temperature has the threshold
as
> [<271,>=
>
271&&<275,>=275&&<283,>=283&&<295,>=295&&<308,>=308&&<310,>=310&&<313,>=313];
> does it mean I have to run 8 times of stat_analysis with each
threshold?
> Thank you
> Mandy
>
>
------------------------------------------------
Subject: threshold of Stat_analysis with CTS
From: John Halley Gotway
Time: Thu Sep 20 10:34:08 2018
Mandy,
Thanks for sending your sample data. I see the discrepancy you
describe,
but there's a good explanation for it. With the commands listed
below, I
grabbed the columns for the FCST_THRESH, TOTAL, and GSS. You pointed
out
that with the "-by" option you got a GSS value of -0.00047107 but with
"-fcst_thresh" you got "0.0034811".
But play close attention to the TOTAL column. In the first case,
TOTAL is
800 but in the second case it's 4000. That indicates to me that the
these
jobs were not run on the same set of .stat data.
FYI, you may find using the "-dump_row" job command option to be
helpful
(e.g. -dump_row data_for_agg_by_fcst_thresh.stat). The "-dump_row"
option
tells stat_analysis to write all the input files that were used for
that
job to the specified output file. That enables you to confirm that
you ran
the job on the exact set of data you intended.
> egrep -v JOB_LIST CTSAPCP_A1_1mon.txt | awk '{print $2,$3,$47}'
FCST_THRESH TOTAL GSS
>=0.1&&<=1.5 800 -0.00047107
>=1.6&&<=6.9 800 NA
>=15.0&&<=39.9 800 NA
>=40.0&&<=49.9 800 NA
>=50.0 800 NA
>=7.0&&<=14.9 800 NA
> egrep -v JOB_LIST CTSAPCP_A1_1mon_ge0.1le1.5.txt | awk '{print
$1,$2,$46}'
COL_NAME: TOTAL GSS
CTS: 4000 0.0034811
So the real question is why are you getting 800 vs 4000 in the TOTAL
column? Perhaps setting "-fcst_thresh '>=0.1&&<=1.5'" isn't filtering
the
data the way we expect. You could run a "filter" job on your data to
test
this...
-job filter -fcst_var APCP -fcst_lev A1 -interp_mthd DW_MEAN
-fcst_thresh
'>=0.1&&<=1.5' -line_type CTC \
-dump_row filter.stat
Then look to see if stat-analysis was successful in filtering out the
lines
you want.
Hope this helps.
Thanks,
John
On Wed, Sep 19, 2018 at 3:24 PM Zhang, Mandy via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036 >
>
> Zhang, Mandy has shared OneDrive for Business files with you. To
view
> them, click the links below.
>
>
> <
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
> >
> [https://r1.res.office365.com/owa/prem/images/dc-txt_20.png]<
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
> >
>
> CTSAPCP_A1_1mon_ge0.1le1.5.txt<
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
> >
>
> <
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
> >
> [https://r1.res.office365.com/owa/prem/images/dc-txt_20.png]<
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
> >
>
> CTSAPCP_A1_1mon.txt<
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
> >
>
>
>
>
> Hi John,
>
> I calculate the precipitation by using option "-by FCST_THRESH"
and
> compared to the result with option "-fcst_thresh ", they show
different
> value of GSS. the value of "-by FCST_THRESH" is negative and seems
> unreasonable.
>
> please find my attachment.
>
> Thank you very much.
>
> Mandy
>
> ________________________________
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Wednesday, September 19, 2018 12:30:21 PM
> To: Zhang, Mandy
> Subject: Re: [rt.rap.ucar.edu #87036] threshold of Stat_analysis
with CTS
>
> Mandy,
>
> Instead of specifying "-fcst_thresh '<271'" to select a single
threshold,
> please try using this setting instead: "-by FCST_THRESH".
>
> That'll run the same job but separately for each unique value found
in the
> FCST_THRESH column. And you can specify multiple columns too...
> -by FCST_THRESH,FCST_LEAD
>
> Is that what you're looking for?
>
> Thanks,
> John
>
> On Wed, Sep 19, 2018 at 10:15 AM Zhang, Mandy via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Wed Sep 19 10:14:40 2018: Request 87036 was acted upon.
> > Transaction: Ticket created by Man.Zhang-1 at ou.edu
> > Queue: met_help
> > Subject: threshold of Stat_analysis with CTS
> > Owner: Nobody
> > Requestors: Man.Zhang-1 at ou.edu
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036 >
> >
> >
> > Hi John,
> >
> > As we talked before, when we run Stat_Analysis with CTS, we
need to
> > set threshold, if the variables like temperature has the threshold
as
> > [<271,>=
> >
>
271&&<275,>=275&&<283,>=283&&<295,>=295&&<308,>=308&&<310,>=310&&<313,>=313];
> > does it mean I have to run 8 times of stat_analysis with each
threshold?
> > Thank you
> > Mandy
> >
> >
>
>
>
------------------------------------------------
Subject: threshold of Stat_analysis with CTS
From: Zhang, Mandy
Time: Thu Sep 20 14:24:19 2018
Hi John,
As you said, after I ran job filter with "-job filter -fcst_var
APCP -fcst_lev A1 -interp_mthd DW_MEAN -fcst_thresh
'>=0.1&&<=1.5' -line_type CTC \", it shows 800 total column,
and my question now is:
if i only do stat_analysis on specified threshold, do i need to run it
with 2 steps: 1. job filter the data and get the data of filter.stat
2. -job aggregate_stat with output of job_filter (filter.stat).
I try to write job filter and job aggregate_stat in one script as:
#!/bin/sh
echo
echo "*** Running STAT-Analysis ***"
/home-gs/users/SZQXG152/METOOLS/met/bin/stat_analysis \
-lookin /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801*/0600/point_stat/02
\
-job filter -line_type CTC -fcst_var APCP -fcst_lev A1 -interp_mthd
DW_MEAN -fcst_thresh 'ge0.1&&le1.5' -dump_row /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
\
-job aggregate_stat -line_type CTC -out_line_type CTS -fcst_var
APCP -fcst_lev A1 -interp_mthd DW_MEAN -fcst_thresh 'ge0.1&&le1.5'
-dump_row /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_aggregate_stat_'ge0.1&&le1.5'.stat
-out /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/CTCAPCP_A1_1mon_filter.txt
\
-v 2
##############
but the error is :
*** Running STAT-Analysis ***
DEBUG 2: Processing 9 STAT files.
DEBUG 2: STAT Lines read = 66672
DEBUG 2: STAT Lines retained = 20
DEBUG 2:
DEBUG 2: Processing Job 1: -job aggregate_stat -fcst_var APCP
-fcst_var APCP -fcst_lev A1 -fcst_lev A1 -interp_mthd DW_MEAN
-interp_mthd DW_MEAN -fcst_thresh >=0.1&&<=1.5 -fcst_thresh
>=0.1&&<=1.5 -line_type CTC -line_type CTC -dump_row /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/01/job_aggregate_stat_ge0.1&&le1.5.stat
-out_line_type CTS -out_alpha 0.05000
DEBUG 1: Creating dump row output file "/home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/01/job_aggregate_stat_ge0.1&&le1.5.stat"
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=18446744072668357467
ERROR :
ERROR : do_job_aggr_stat() -> this function may only be called when
the "-line_type" and "-out_line_type" options have been used exactly
once: -job aggregate_stat -fcst_var APCP -fcst_var APCP -fcst_lev A1
-fcst_lev A1 -interp_mthd DW_MEAN -interp_mthd DW_MEAN -fcst_thresh
>=0.1&&<=1.5 -fcst_thresh >=0.1&&<=1.5 -line_type CTC -line_type CTC
-dump_row /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/01/job_aggregate_stat_ge0.1&&le1.5.stat
-out_line_type CTS -out_alpha 0.05000
ERROR :
ERROR :
ERROR : main() -> encountered an error value of 1. Calling
clean_up() and usage() before exiting.
ERROR :
#############
Do you know what's the issue?
Thank you!
Mandy
________________________________
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Thursday, September 20, 2018 11:34:09 AM
To: Zhang, Mandy
Subject: Re: [rt.rap.ucar.edu #87036] threshold of Stat_analysis with
CTS
Mandy,
Thanks for sending your sample data. I see the discrepancy you
describe,
but there's a good explanation for it. With the commands listed
below, I
grabbed the columns for the FCST_THRESH, TOTAL, and GSS. You pointed
out
that with the "-by" option you got a GSS value of -0.00047107 but with
"-fcst_thresh" you got "0.0034811".
But play close attention to the TOTAL column. In the first case,
TOTAL is
800 but in the second case it's 4000. That indicates to me that the
these
jobs were not run on the same set of .stat data.
FYI, you may find using the "-dump_row" job command option to be
helpful
(e.g. -dump_row data_for_agg_by_fcst_thresh.stat). The "-dump_row"
option
tells stat_analysis to write all the input files that were used for
that
job to the specified output file. That enables you to confirm that
you ran
the job on the exact set of data you intended.
> egrep -v JOB_LIST CTSAPCP_A1_1mon.txt | awk '{print $2,$3,$47}'
FCST_THRESH TOTAL GSS
>=0.1&&<=1.5 800 -0.00047107
>=1.6&&<=6.9 800 NA
>=15.0&&<=39.9 800 NA
>=40.0&&<=49.9 800 NA
>=50.0 800 NA
>=7.0&&<=14.9 800 NA
> egrep -v JOB_LIST CTSAPCP_A1_1mon_ge0.1le1.5.txt | awk '{print
$1,$2,$46}'
COL_NAME: TOTAL GSS
CTS: 4000 0.0034811
So the real question is why are you getting 800 vs 4000 in the TOTAL
column? Perhaps setting "-fcst_thresh '>=0.1&&<=1.5'" isn't filtering
the
data the way we expect. You could run a "filter" job on your data to
test
this...
-job filter -fcst_var APCP -fcst_lev A1 -interp_mthd DW_MEAN
-fcst_thresh
'>=0.1&&<=1.5' -line_type CTC \
-dump_row filter.stat
Then look to see if stat-analysis was successful in filtering out the
lines
you want.
Hope this helps.
Thanks,
John
On Wed, Sep 19, 2018 at 3:24 PM Zhang, Mandy via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036 >
>
> Zhang, Mandy has shared OneDrive for Business files with you. To
view
> them, click the links below.
>
>
> <
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
> >
> [https://r1.res.office365.com/owa/prem/images/dc-txt_20.png]<
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
> >
>
> CTSAPCP_A1_1mon_ge0.1le1.5.txt<
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
> >
>
> <
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
> >
> [https://r1.res.office365.com/owa/prem/images/dc-txt_20.png]<
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
> >
>
> CTSAPCP_A1_1mon.txt<
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
> >
>
>
>
>
> Hi John,
>
> I calculate the precipitation by using option "-by FCST_THRESH"
and
> compared to the result with option "-fcst_thresh ", they show
different
> value of GSS. the value of "-by FCST_THRESH" is negative and seems
> unreasonable.
>
> please find my attachment.
>
> Thank you very much.
>
> Mandy
>
> ________________________________
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Wednesday, September 19, 2018 12:30:21 PM
> To: Zhang, Mandy
> Subject: Re: [rt.rap.ucar.edu #87036] threshold of Stat_analysis
with CTS
>
> Mandy,
>
> Instead of specifying "-fcst_thresh '<271'" to select a single
threshold,
> please try using this setting instead: "-by FCST_THRESH".
>
> That'll run the same job but separately for each unique value found
in the
> FCST_THRESH column. And you can specify multiple columns too...
> -by FCST_THRESH,FCST_LEAD
>
> Is that what you're looking for?
>
> Thanks,
> John
>
> On Wed, Sep 19, 2018 at 10:15 AM Zhang, Mandy via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Wed Sep 19 10:14:40 2018: Request 87036 was acted upon.
> > Transaction: Ticket created by Man.Zhang-1 at ou.edu
> > Queue: met_help
> > Subject: threshold of Stat_analysis with CTS
> > Owner: Nobody
> > Requestors: Man.Zhang-1 at ou.edu
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036 >
> >
> >
> > Hi John,
> >
> > As we talked before, when we run Stat_Analysis with CTS, we
need to
> > set threshold, if the variables like temperature has the threshold
as
> > [<271,>=
> >
>
271&&<275,>=275&&<283,>=283&&<295,>=295&&<308,>=308&&<310,>=310&&<313,>=313];
> > does it mean I have to run 8 times of stat_analysis with each
threshold?
> > Thank you
> > Mandy
> >
> >
>
>
>
------------------------------------------------
Subject: threshold of Stat_analysis with CTS
From: John Halley Gotway
Time: Thu Sep 20 15:03:36 2018
Mandy,
OK, I see what's going on. When you run STAT-Analysis on the command
line,
you can only run one job at a time. In the command you sent, you're
trying
to run 2 of them on the command line. That doesn't work. You can use
STAT-Analysis to run multiple jobs, but you would need to define them
in a
config file.
So first run this job:
/home-gs/users/SZQXG152/METOOLS/met/bin/stat_analysis \
-lookin
/home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801*/0600/point_stat/02
\
-job filter -line_type CTC -fcst_var APCP -fcst_lev A1 -interp_mthd
DW_MEAN -fcst_thresh 'ge0.1&&le1.5' -dump_row
/home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
And then look in this output file:
/home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
How many lines of output do you see, 800 or 4000?
Look at the FCST_THRESH column. Does it only contain data for
'ge0.1&&le1.5'?
Thanks,
John
On Thu, Sep 20, 2018 at 2:24 PM Zhang, Mandy via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036 >
>
> Hi John,
>
> As you said, after I ran job filter with "-job filter -fcst_var
APCP
> -fcst_lev A1 -interp_mthd DW_MEAN -fcst_thresh
> '>=0.1&&<=1.5' -line_type CTC \", it shows 800 total column,
>
> and my question now is:
>
> if i only do stat_analysis on specified threshold, do i need to run
it
> with 2 steps: 1. job filter the data and get the data of filter.stat
2.
> -job aggregate_stat with output of job_filter (filter.stat).
>
> I try to write job filter and job aggregate_stat in one script as:
>
> #!/bin/sh
>
> echo
> echo "*** Running STAT-Analysis ***"
> /home-gs/users/SZQXG152/METOOLS/met/bin/stat_analysis \
> -lookin
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801*/0600/point_stat/02
> \
> -job filter -line_type CTC -fcst_var APCP -fcst_lev A1
-interp_mthd
> DW_MEAN -fcst_thresh 'ge0.1&&le1.5' -dump_row
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
> \
> -job aggregate_stat -line_type CTC -out_line_type CTS -fcst_var
APCP
> -fcst_lev A1 -interp_mthd DW_MEAN -fcst_thresh 'ge0.1&&le1.5'
-dump_row
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_aggregate_stat_'ge0.1&&le1.5'.stat
> -out
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/CTCAPCP_A1_1mon_filter.txt
> \
> -v 2
> ##############
>
> but the error is :
>
> *** Running STAT-Analysis ***
> DEBUG 2: Processing 9 STAT files.
> DEBUG 2: STAT Lines read = 66672
> DEBUG 2: STAT Lines retained = 20
> DEBUG 2:
> DEBUG 2: Processing Job 1: -job aggregate_stat -fcst_var APCP
-fcst_var
> APCP -fcst_lev A1 -fcst_lev A1 -interp_mthd DW_MEAN -interp_mthd
DW_MEAN
> -fcst_thresh >=0.1&&<=1.5 -fcst_thresh >=0.1&&<=1.5 -line_type CTC
> -line_type CTC -dump_row
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/01/job_aggregate_stat_ge0.1&&le1.5.stat
> -out_line_type CTS -out_alpha 0.05000
> DEBUG 1: Creating dump row output file
> "/home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/01/job_aggregate_stat_ge0.1&&le1.5.stat"
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744072668357467
> ERROR :
> ERROR : do_job_aggr_stat() -> this function may only be called when
the
> "-line_type" and "-out_line_type" options have been used exactly
once: -job
> aggregate_stat -fcst_var APCP -fcst_var APCP -fcst_lev A1 -fcst_lev
A1
> -interp_mthd DW_MEAN -interp_mthd DW_MEAN -fcst_thresh >=0.1&&<=1.5
> -fcst_thresh >=0.1&&<=1.5 -line_type CTC -line_type CTC -dump_row
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/01/job_aggregate_stat_ge0.1&&le1.5.stat
> -out_line_type CTS -out_alpha 0.05000
> ERROR :
> ERROR :
> ERROR : main() -> encountered an error value of 1. Calling
clean_up()
> and usage() before exiting.
> ERROR :
>
> #############
>
> Do you know what's the issue?
>
> Thank you!
>
> Mandy
>
>
> ________________________________
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, September 20, 2018 11:34:09 AM
> To: Zhang, Mandy
> Subject: Re: [rt.rap.ucar.edu #87036] threshold of Stat_analysis
with CTS
>
> Mandy,
>
>
> Thanks for sending your sample data. I see the discrepancy you
describe,
> but there's a good explanation for it. With the commands listed
below, I
> grabbed the columns for the FCST_THRESH, TOTAL, and GSS. You
pointed out
> that with the "-by" option you got a GSS value of -0.00047107 but
with
> "-fcst_thresh" you got "0.0034811".
>
>
> But play close attention to the TOTAL column. In the first case,
TOTAL is
> 800 but in the second case it's 4000. That indicates to me that the
these
> jobs were not run on the same set of .stat data.
>
>
> FYI, you may find using the "-dump_row" job command option to be
helpful
> (e.g. -dump_row data_for_agg_by_fcst_thresh.stat). The "-dump_row"
option
> tells stat_analysis to write all the input files that were used for
that
> job to the specified output file. That enables you to confirm that
you ran
> the job on the exact set of data you intended.
>
>
>
> > egrep -v JOB_LIST CTSAPCP_A1_1mon.txt | awk '{print $2,$3,$47}'
>
> FCST_THRESH TOTAL GSS
>
> >=0.1&&<=1.5 800 -0.00047107
>
> >=1.6&&<=6.9 800 NA
>
> >=15.0&&<=39.9 800 NA
>
> >=40.0&&<=49.9 800 NA
>
> >=50.0 800 NA
>
> >=7.0&&<=14.9 800 NA
>
>
> > egrep -v JOB_LIST CTSAPCP_A1_1mon_ge0.1le1.5.txt | awk '{print
> $1,$2,$46}'
>
> COL_NAME: TOTAL GSS
>
> CTS: 4000 0.0034811
>
>
> So the real question is why are you getting 800 vs 4000 in the TOTAL
> column? Perhaps setting "-fcst_thresh '>=0.1&&<=1.5'" isn't
filtering the
> data the way we expect. You could run a "filter" job on your data
to test
> this...
>
>
> -job filter -fcst_var APCP -fcst_lev A1 -interp_mthd DW_MEAN
-fcst_thresh
> '>=0.1&&<=1.5' -line_type CTC \
>
> -dump_row filter.stat
>
>
> Then look to see if stat-analysis was successful in filtering out
the lines
> you want.
>
>
> Hope this helps.
>
>
> Thanks,
> John
>
> On Wed, Sep 19, 2018 at 3:24 PM Zhang, Mandy via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036 >
> >
> > Zhang, Mandy has shared OneDrive for Business files with you. To
view
> > them, click the links below.
> >
> >
> > <
> >
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
> > >
> > [https://r1.res.office365.com/owa/prem/images/dc-txt_20.png]<
> >
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
> > >
> >
> > CTSAPCP_A1_1mon_ge0.1le1.5.txt<
> >
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
> > >
> >
> > <
> >
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
> > >
> > [https://r1.res.office365.com/owa/prem/images/dc-txt_20.png]<
> >
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
> > >
> >
> > CTSAPCP_A1_1mon.txt<
> >
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
> > >
> >
> >
> >
> >
> > Hi John,
> >
> > I calculate the precipitation by using option "-by FCST_THRESH"
and
> > compared to the result with option "-fcst_thresh ", they show
different
> > value of GSS. the value of "-by FCST_THRESH" is negative and seems
> > unreasonable.
> >
> > please find my attachment.
> >
> > Thank you very much.
> >
> > Mandy
> >
> > ________________________________
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Wednesday, September 19, 2018 12:30:21 PM
> > To: Zhang, Mandy
> > Subject: Re: [rt.rap.ucar.edu #87036] threshold of Stat_analysis
with
> CTS
> >
> > Mandy,
> >
> > Instead of specifying "-fcst_thresh '<271'" to select a single
threshold,
> > please try using this setting instead: "-by FCST_THRESH".
> >
> > That'll run the same job but separately for each unique value
found in
> the
> > FCST_THRESH column. And you can specify multiple columns too...
> > -by FCST_THRESH,FCST_LEAD
> >
> > Is that what you're looking for?
> >
> > Thanks,
> > John
> >
> > On Wed, Sep 19, 2018 at 10:15 AM Zhang, Mandy via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > Wed Sep 19 10:14:40 2018: Request 87036 was acted upon.
> > > Transaction: Ticket created by Man.Zhang-1 at ou.edu
> > > Queue: met_help
> > > Subject: threshold of Stat_analysis with CTS
> > > Owner: Nobody
> > > Requestors: Man.Zhang-1 at ou.edu
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036
> >
> > >
> > >
> > > Hi John,
> > >
> > > As we talked before, when we run Stat_Analysis with CTS, we
need to
> > > set threshold, if the variables like temperature has the
threshold as
> > > [<271,>=
> > >
> >
>
271&&<275,>=275&&<283,>=283&&<295,>=295&&<308,>=308&&<310,>=310&&<313,>=313];
> > > does it mean I have to run 8 times of stat_analysis with each
> threshold?
> > > Thank you
> > > Mandy
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: threshold of Stat_analysis with CTS
From: Zhang, Mandy
Time: Thu Sep 20 15:36:11 2018
Zhang, Mandy has shared a OneDrive for Business file with you. To view
it, click the link below.
<https://sooners-my.sharepoint.com/:u:/g/personal/man_zhang-
1_ou_edu/EawqFRkeyX5Ep4STupok7g0BnM7yNvkNUtlxa3R-BxpMnQ>
[https://r1.res.office365.com/owa/prem/images/dc-
generic_20.png]<https://sooners-
my.sharepoint.com/:u:/g/personal/man_zhang-
1_ou_edu/EawqFRkeyX5Ep4STupok7g0BnM7yNvkNUtlxa3R-BxpMnQ>
job_filter_stat_ge0.1le1.5<https://sooners-
my.sharepoint.com/:u:/g/personal/man_zhang-
1_ou_edu/EawqFRkeyX5Ep4STupok7g0BnM7yNvkNUtlxa3R-BxpMnQ>
Hi John,
I see, please find my attachment, it looks like "thresh" doesn't work.
my command is :
*******************
#!/bin/sh
echo
echo "*** Running STAT-Analysis ***"
/home-gs/users/SZQXG152/METOOLS/met/bin/stat_analysis \
-lookin /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801*/0600/point_stat/02
\
-job filter -line_type CTC -fcst_var APCP -fcst_lev A1 -interp_mthd
DW_MEAN -fcst_thresh 'ge0.1&&le1.5' -dump_row /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
\
-v 2
***
Thank you!
Mandy
________________________________
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Thursday, September 20, 2018 4:03:36 PM
To: Zhang, Mandy
Subject: Re: [rt.rap.ucar.edu #87036] threshold of Stat_analysis with
CTS
Mandy,
OK, I see what's going on. When you run STAT-Analysis on the command
line,
you can only run one job at a time. In the command you sent, you're
trying
to run 2 of them on the command line. That doesn't work. You can use
STAT-Analysis to run multiple jobs, but you would need to define them
in a
config file.
So first run this job:
/home-gs/users/SZQXG152/METOOLS/met/bin/stat_analysis \
-lookin
/home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801*/0600/point_stat/02
\
-job filter -line_type CTC -fcst_var APCP -fcst_lev A1 -interp_mthd
DW_MEAN -fcst_thresh 'ge0.1&&le1.5' -dump_row
/home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
And then look in this output file:
/home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
How many lines of output do you see, 800 or 4000?
Look at the FCST_THRESH column. Does it only contain data for
'ge0.1&&le1.5'?
Thanks,
John
On Thu, Sep 20, 2018 at 2:24 PM Zhang, Mandy via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036 >
>
> Hi John,
>
> As you said, after I ran job filter with "-job filter -fcst_var
APCP
> -fcst_lev A1 -interp_mthd DW_MEAN -fcst_thresh
> '>=0.1&&<=1.5' -line_type CTC \", it shows 800 total column,
>
> and my question now is:
>
> if i only do stat_analysis on specified threshold, do i need to run
it
> with 2 steps: 1. job filter the data and get the data of filter.stat
2.
> -job aggregate_stat with output of job_filter (filter.stat).
>
> I try to write job filter and job aggregate_stat in one script as:
>
> #!/bin/sh
>
> echo
> echo "*** Running STAT-Analysis ***"
> /home-gs/users/SZQXG152/METOOLS/met/bin/stat_analysis \
> -lookin
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801*/0600/point_stat/02
> \
> -job filter -line_type CTC -fcst_var APCP -fcst_lev A1
-interp_mthd
> DW_MEAN -fcst_thresh 'ge0.1&&le1.5' -dump_row
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
> \
> -job aggregate_stat -line_type CTC -out_line_type CTS -fcst_var
APCP
> -fcst_lev A1 -interp_mthd DW_MEAN -fcst_thresh 'ge0.1&&le1.5'
-dump_row
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_aggregate_stat_'ge0.1&&le1.5'.stat
> -out
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/CTCAPCP_A1_1mon_filter.txt
> \
> -v 2
> ##############
>
> but the error is :
>
> *** Running STAT-Analysis ***
> DEBUG 2: Processing 9 STAT files.
> DEBUG 2: STAT Lines read = 66672
> DEBUG 2: STAT Lines retained = 20
> DEBUG 2:
> DEBUG 2: Processing Job 1: -job aggregate_stat -fcst_var APCP
-fcst_var
> APCP -fcst_lev A1 -fcst_lev A1 -interp_mthd DW_MEAN -interp_mthd
DW_MEAN
> -fcst_thresh >=0.1&&<=1.5 -fcst_thresh >=0.1&&<=1.5 -line_type CTC
> -line_type CTC -dump_row
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/01/job_aggregate_stat_ge0.1&&le1.5.stat
> -out_line_type CTS -out_alpha 0.05000
> DEBUG 1: Creating dump row output file
> "/home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/01/job_aggregate_stat_ge0.1&&le1.5.stat"
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744072668357467
> ERROR :
> ERROR : do_job_aggr_stat() -> this function may only be called when
the
> "-line_type" and "-out_line_type" options have been used exactly
once: -job
> aggregate_stat -fcst_var APCP -fcst_var APCP -fcst_lev A1 -fcst_lev
A1
> -interp_mthd DW_MEAN -interp_mthd DW_MEAN -fcst_thresh >=0.1&&<=1.5
> -fcst_thresh >=0.1&&<=1.5 -line_type CTC -line_type CTC -dump_row
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/01/job_aggregate_stat_ge0.1&&le1.5.stat
> -out_line_type CTS -out_alpha 0.05000
> ERROR :
> ERROR :
> ERROR : main() -> encountered an error value of 1. Calling
clean_up()
> and usage() before exiting.
> ERROR :
>
> #############
>
> Do you know what's the issue?
>
> Thank you!
>
> Mandy
>
>
> ________________________________
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, September 20, 2018 11:34:09 AM
> To: Zhang, Mandy
> Subject: Re: [rt.rap.ucar.edu #87036] threshold of Stat_analysis
with CTS
>
> Mandy,
>
>
> Thanks for sending your sample data. I see the discrepancy you
describe,
> but there's a good explanation for it. With the commands listed
below, I
> grabbed the columns for the FCST_THRESH, TOTAL, and GSS. You
pointed out
> that with the "-by" option you got a GSS value of -0.00047107 but
with
> "-fcst_thresh" you got "0.0034811".
>
>
> But play close attention to the TOTAL column. In the first case,
TOTAL is
> 800 but in the second case it's 4000. That indicates to me that the
these
> jobs were not run on the same set of .stat data.
>
>
> FYI, you may find using the "-dump_row" job command option to be
helpful
> (e.g. -dump_row data_for_agg_by_fcst_thresh.stat). The "-dump_row"
option
> tells stat_analysis to write all the input files that were used for
that
> job to the specified output file. That enables you to confirm that
you ran
> the job on the exact set of data you intended.
>
>
>
> > egrep -v JOB_LIST CTSAPCP_A1_1mon.txt | awk '{print $2,$3,$47}'
>
> FCST_THRESH TOTAL GSS
>
> >=0.1&&<=1.5 800 -0.00047107
>
> >=1.6&&<=6.9 800 NA
>
> >=15.0&&<=39.9 800 NA
>
> >=40.0&&<=49.9 800 NA
>
> >=50.0 800 NA
>
> >=7.0&&<=14.9 800 NA
>
>
> > egrep -v JOB_LIST CTSAPCP_A1_1mon_ge0.1le1.5.txt | awk '{print
> $1,$2,$46}'
>
> COL_NAME: TOTAL GSS
>
> CTS: 4000 0.0034811
>
>
> So the real question is why are you getting 800 vs 4000 in the TOTAL
> column? Perhaps setting "-fcst_thresh '>=0.1&&<=1.5'" isn't
filtering the
> data the way we expect. You could run a "filter" job on your data
to test
> this...
>
>
> -job filter -fcst_var APCP -fcst_lev A1 -interp_mthd DW_MEAN
-fcst_thresh
> '>=0.1&&<=1.5' -line_type CTC \
>
> -dump_row filter.stat
>
>
> Then look to see if stat-analysis was successful in filtering out
the lines
> you want.
>
>
> Hope this helps.
>
>
> Thanks,
> John
>
> On Wed, Sep 19, 2018 at 3:24 PM Zhang, Mandy via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036 >
> >
> > Zhang, Mandy has shared OneDrive for Business files with you. To
view
> > them, click the links below.
> >
> >
> > <
> >
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
> > >
> > [https://r1.res.office365.com/owa/prem/images/dc-txt_20.png]<
> >
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
> > >
> >
> > CTSAPCP_A1_1mon_ge0.1le1.5.txt<
> >
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
> > >
> >
> > <
> >
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
> > >
> > [https://r1.res.office365.com/owa/prem/images/dc-txt_20.png]<
> >
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
> > >
> >
> > CTSAPCP_A1_1mon.txt<
> >
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
> > >
> >
> >
> >
> >
> > Hi John,
> >
> > I calculate the precipitation by using option "-by FCST_THRESH"
and
> > compared to the result with option "-fcst_thresh ", they show
different
> > value of GSS. the value of "-by FCST_THRESH" is negative and seems
> > unreasonable.
> >
> > please find my attachment.
> >
> > Thank you very much.
> >
> > Mandy
> >
> > ________________________________
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Wednesday, September 19, 2018 12:30:21 PM
> > To: Zhang, Mandy
> > Subject: Re: [rt.rap.ucar.edu #87036] threshold of Stat_analysis
with
> CTS
> >
> > Mandy,
> >
> > Instead of specifying "-fcst_thresh '<271'" to select a single
threshold,
> > please try using this setting instead: "-by FCST_THRESH".
> >
> > That'll run the same job but separately for each unique value
found in
> the
> > FCST_THRESH column. And you can specify multiple columns too...
> > -by FCST_THRESH,FCST_LEAD
> >
> > Is that what you're looking for?
> >
> > Thanks,
> > John
> >
> > On Wed, Sep 19, 2018 at 10:15 AM Zhang, Mandy via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > Wed Sep 19 10:14:40 2018: Request 87036 was acted upon.
> > > Transaction: Ticket created by Man.Zhang-1 at ou.edu
> > > Queue: met_help
> > > Subject: threshold of Stat_analysis with CTS
> > > Owner: Nobody
> > > Requestors: Man.Zhang-1 at ou.edu
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036
> >
> > >
> > >
> > > Hi John,
> > >
> > > As we talked before, when we run Stat_Analysis with CTS, we
need to
> > > set threshold, if the variables like temperature has the
threshold as
> > > [<271,>=
> > >
> >
>
271&&<275,>=275&&<283,>=283&&<295,>=295&&<308,>=308&&<310,>=310&&<313,>=313];
> > > does it mean I have to run 8 times of stat_analysis with each
> threshold?
> > > Thank you
> > > Mandy
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: threshold of Stat_analysis with CTS
From: John Halley Gotway
Time: Fri Sep 21 14:29:46 2018
Mandy,
Thanks for sending the sample data. I ran it through the debugger and
confirmed that you've uncovered a bug. The good news is that using
the
"-by FCST_THRESH" option will produce accurate results.
But specifying a specific threshold (.e.g. -fcst_thresh 'ge0.1&&le1.5'
has
no effect. Specifically, the problem is in this function:
bool SingleThresh::operator==(const SingleThresh &st) const
In the source code file named "threshold.cc".
Thresholds which include && or || are stored as a "Complex" type. And
the
logic in the function is not sufficient. It returns true as long as
both
thresholds being compared are Complex (i.e. include && or ||).
We'll need to come up with more robust logic for checking for the
equality
of complex threshold types.
Thanks,
John
On Thu, Sep 20, 2018 at 3:36 PM Zhang, Mandy via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036 >
>
> Zhang, Mandy has shared a OneDrive for Business file with you. To
view it,
> click the link below.
>
>
> <
> https://sooners-my.sharepoint.com/:u:/g/personal/man_zhang-
1_ou_edu/EawqFRkeyX5Ep4STupok7g0BnM7yNvkNUtlxa3R-BxpMnQ
> >
> [https://r1.res.office365.com/owa/prem/images/dc-generic_20.png]<
> https://sooners-my.sharepoint.com/:u:/g/personal/man_zhang-
1_ou_edu/EawqFRkeyX5Ep4STupok7g0BnM7yNvkNUtlxa3R-BxpMnQ
> >
>
> job_filter_stat_ge0.1le1.5<
> https://sooners-my.sharepoint.com/:u:/g/personal/man_zhang-
1_ou_edu/EawqFRkeyX5Ep4STupok7g0BnM7yNvkNUtlxa3R-BxpMnQ
> >
>
>
>
>
> Hi John,
>
> I see, please find my attachment, it looks like "thresh" doesn't
work.
>
> my command is :
>
> *******************
>
> #!/bin/sh
>
> echo
> echo "*** Running STAT-Analysis ***"
> /home-gs/users/SZQXG152/METOOLS/met/bin/stat_analysis \
> -lookin
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801*/0600/point_stat/02
> \
> -job filter -line_type CTC -fcst_var APCP -fcst_lev A1
-interp_mthd
> DW_MEAN -fcst_thresh 'ge0.1&&le1.5' -dump_row
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
> \
>
> -v 2
> ***
>
> Thank you!
>
> Mandy
>
> ________________________________
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, September 20, 2018 4:03:36 PM
> To: Zhang, Mandy
> Subject: Re: [rt.rap.ucar.edu #87036] threshold of Stat_analysis
with CTS
>
> Mandy,
>
> OK, I see what's going on. When you run STAT-Analysis on the
command line,
> you can only run one job at a time. In the command you sent, you're
trying
> to run 2 of them on the command line. That doesn't work. You can
use
> STAT-Analysis to run multiple jobs, but you would need to define
them in a
> config file.
>
> So first run this job:
> /home-gs/users/SZQXG152/METOOLS/met/bin/stat_analysis \
> -lookin
>
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801*/0600/point_stat/02
> \
> -job filter -line_type CTC -fcst_var APCP -fcst_lev A1
-interp_mthd
> DW_MEAN -fcst_thresh 'ge0.1&&le1.5' -dump_row
>
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
>
> And then look in this output file:
>
>
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
>
> How many lines of output do you see, 800 or 4000?
> Look at the FCST_THRESH column. Does it only contain data for
> 'ge0.1&&le1.5'?
>
> Thanks,
> John
>
> On Thu, Sep 20, 2018 at 2:24 PM Zhang, Mandy via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036 >
> >
> > Hi John,
> >
> > As you said, after I ran job filter with "-job filter
-fcst_var APCP
> > -fcst_lev A1 -interp_mthd DW_MEAN -fcst_thresh
> > '>=0.1&&<=1.5' -line_type CTC \", it shows 800 total column,
> >
> > and my question now is:
> >
> > if i only do stat_analysis on specified threshold, do i need to
run it
> > with 2 steps: 1. job filter the data and get the data of
filter.stat 2.
> > -job aggregate_stat with output of job_filter (filter.stat).
> >
> > I try to write job filter and job aggregate_stat in one script as:
> >
> > #!/bin/sh
> >
> > echo
> > echo "*** Running STAT-Analysis ***"
> > /home-gs/users/SZQXG152/METOOLS/met/bin/stat_analysis \
> > -lookin
> >
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801*/0600/point_stat/02
> > \
> > -job filter -line_type CTC -fcst_var APCP -fcst_lev A1
-interp_mthd
> > DW_MEAN -fcst_thresh 'ge0.1&&le1.5' -dump_row
> >
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
> > \
> > -job aggregate_stat -line_type CTC -out_line_type CTS
-fcst_var APCP
> > -fcst_lev A1 -interp_mthd DW_MEAN -fcst_thresh 'ge0.1&&le1.5'
-dump_row
> >
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_aggregate_stat_'ge0.1&&le1.5'.stat
> > -out
> >
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/CTCAPCP_A1_1mon_filter.txt
> > \
> > -v 2
> > ##############
> >
> > but the error is :
> >
> > *** Running STAT-Analysis ***
> > DEBUG 2: Processing 9 STAT files.
> > DEBUG 2: STAT Lines read = 66672
> > DEBUG 2: STAT Lines retained = 20
> > DEBUG 2:
> > DEBUG 2: Processing Job 1: -job aggregate_stat -fcst_var APCP
-fcst_var
> > APCP -fcst_lev A1 -fcst_lev A1 -interp_mthd DW_MEAN -interp_mthd
DW_MEAN
> > -fcst_thresh >=0.1&&<=1.5 -fcst_thresh >=0.1&&<=1.5 -line_type CTC
> > -line_type CTC -dump_row
> >
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/01/job_aggregate_stat_ge0.1&&le1.5.stat
> > -out_line_type CTS -out_alpha 0.05000
> > DEBUG 1: Creating dump row output file
> >
> "/home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/01/job_aggregate_stat_ge0.1&&le1.5.stat"
> > GSL_RNG_TYPE=mt19937
> > GSL_RNG_SEED=18446744072668357467
> > ERROR :
> > ERROR : do_job_aggr_stat() -> this function may only be called
when the
> > "-line_type" and "-out_line_type" options have been used exactly
once:
> -job
> > aggregate_stat -fcst_var APCP -fcst_var APCP -fcst_lev A1
-fcst_lev A1
> > -interp_mthd DW_MEAN -interp_mthd DW_MEAN -fcst_thresh
>=0.1&&<=1.5
> > -fcst_thresh >=0.1&&<=1.5 -line_type CTC -line_type CTC -dump_row
> >
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/01/job_aggregate_stat_ge0.1&&le1.5.stat
> > -out_line_type CTS -out_alpha 0.05000
> > ERROR :
> > ERROR :
> > ERROR : main() -> encountered an error value of 1. Calling
clean_up()
> > and usage() before exiting.
> > ERROR :
> >
> > #############
> >
> > Do you know what's the issue?
> >
> > Thank you!
> >
> > Mandy
> >
> >
> > ________________________________
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Thursday, September 20, 2018 11:34:09 AM
> > To: Zhang, Mandy
> > Subject: Re: [rt.rap.ucar.edu #87036] threshold of Stat_analysis
with
> CTS
> >
> > Mandy,
> >
> >
> > Thanks for sending your sample data. I see the discrepancy you
describe,
> > but there's a good explanation for it. With the commands listed
below, I
> > grabbed the columns for the FCST_THRESH, TOTAL, and GSS. You
pointed out
> > that with the "-by" option you got a GSS value of -0.00047107 but
with
> > "-fcst_thresh" you got "0.0034811".
> >
> >
> > But play close attention to the TOTAL column. In the first case,
TOTAL
> is
> > 800 but in the second case it's 4000. That indicates to me that
the
> these
> > jobs were not run on the same set of .stat data.
> >
> >
> > FYI, you may find using the "-dump_row" job command option to be
helpful
> > (e.g. -dump_row data_for_agg_by_fcst_thresh.stat). The "-
dump_row"
> option
> > tells stat_analysis to write all the input files that were used
for that
> > job to the specified output file. That enables you to confirm
that you
> ran
> > the job on the exact set of data you intended.
> >
> >
> >
> > > egrep -v JOB_LIST CTSAPCP_A1_1mon.txt | awk '{print $2,$3,$47}'
> >
> > FCST_THRESH TOTAL GSS
> >
> > >=0.1&&<=1.5 800 -0.00047107
> >
> > >=1.6&&<=6.9 800 NA
> >
> > >=15.0&&<=39.9 800 NA
> >
> > >=40.0&&<=49.9 800 NA
> >
> > >=50.0 800 NA
> >
> > >=7.0&&<=14.9 800 NA
> >
> >
> > > egrep -v JOB_LIST CTSAPCP_A1_1mon_ge0.1le1.5.txt | awk '{print
> > $1,$2,$46}'
> >
> > COL_NAME: TOTAL GSS
> >
> > CTS: 4000 0.0034811
> >
> >
> > So the real question is why are you getting 800 vs 4000 in the
TOTAL
> > column? Perhaps setting "-fcst_thresh '>=0.1&&<=1.5'" isn't
filtering
> the
> > data the way we expect. You could run a "filter" job on your data
to
> test
> > this...
> >
> >
> > -job filter -fcst_var APCP -fcst_lev A1 -interp_mthd DW_MEAN
-fcst_thresh
> > '>=0.1&&<=1.5' -line_type CTC \
> >
> > -dump_row filter.stat
> >
> >
> > Then look to see if stat-analysis was successful in filtering out
the
> lines
> > you want.
> >
> >
> > Hope this helps.
> >
> >
> > Thanks,
> > John
> >
> > On Wed, Sep 19, 2018 at 3:24 PM Zhang, Mandy via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036 >
> > >
> > > Zhang, Mandy has shared OneDrive for Business files with you. To
view
> > > them, click the links below.
> > >
> > >
> > > <
> > >
> >
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
> > > >
> > > [https://r1.res.office365.com/owa/prem/images/dc-txt_20.png]<
> > >
> >
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
> > > >
> > >
> > > CTSAPCP_A1_1mon_ge0.1le1.5.txt<
> > >
> >
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
> > > >
> > >
> > > <
> > >
> >
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
> > > >
> > > [https://r1.res.office365.com/owa/prem/images/dc-txt_20.png]<
> > >
> >
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
> > > >
> > >
> > > CTSAPCP_A1_1mon.txt<
> > >
> >
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
> > > >
> > >
> > >
> > >
> > >
> > > Hi John,
> > >
> > > I calculate the precipitation by using option "-by
FCST_THRESH" and
> > > compared to the result with option "-fcst_thresh ", they show
different
> > > value of GSS. the value of "-by FCST_THRESH" is negative and
seems
> > > unreasonable.
> > >
> > > please find my attachment.
> > >
> > > Thank you very much.
> > >
> > > Mandy
> > >
> > > ________________________________
> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> > > Sent: Wednesday, September 19, 2018 12:30:21 PM
> > > To: Zhang, Mandy
> > > Subject: Re: [rt.rap.ucar.edu #87036] threshold of Stat_analysis
with
> > CTS
> > >
> > > Mandy,
> > >
> > > Instead of specifying "-fcst_thresh '<271'" to select a single
> threshold,
> > > please try using this setting instead: "-by FCST_THRESH".
> > >
> > > That'll run the same job but separately for each unique value
found in
> > the
> > > FCST_THRESH column. And you can specify multiple columns too...
> > > -by FCST_THRESH,FCST_LEAD
> > >
> > > Is that what you're looking for?
> > >
> > > Thanks,
> > > John
> > >
> > > On Wed, Sep 19, 2018 at 10:15 AM Zhang, Mandy via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > Wed Sep 19 10:14:40 2018: Request 87036 was acted upon.
> > > > Transaction: Ticket created by Man.Zhang-1 at ou.edu
> > > > Queue: met_help
> > > > Subject: threshold of Stat_analysis with CTS
> > > > Owner: Nobody
> > > > Requestors: Man.Zhang-1 at ou.edu
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036
> > >
> > > >
> > > >
> > > > Hi John,
> > > >
> > > > As we talked before, when we run Stat_Analysis with CTS,
we need
> to
> > > > set threshold, if the variables like temperature has the
threshold as
> > > > [<271,>=
> > > >
> > >
> >
>
271&&<275,>=275&&<283,>=283&&<295,>=295&&<308,>=308&&<310,>=310&&<313,>=313];
> > > > does it mean I have to run 8 times of stat_analysis with each
> > threshold?
> > > > Thank you
> > > > Mandy
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: threshold of Stat_analysis with CTS
From: John Halley Gotway
Time: Fri Sep 21 14:58:24 2018
Mandy,
OK, I just committed a bugfix to the development version of MET that
will
be included in the upcoming met-8.0 release.
When checking for equality of thresholds, for "Complex" types (i.e.
with &&
or ||), check for equality of their string representations.
Converting
them to strings is a bit slower... so for the "Simple" types, we'll
use the
old, faster logic.
I tested and confirmed that this change enables the -fcst_thresh
option in
stat_analysis to work as you'd expect on your data.
Would you also like this fix posted as a bugfix for met-7.0 or is
using the
upcoming 8.0 release sufficient?
Thanks,
John
On Fri, Sep 21, 2018 at 2:29 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Mandy,
>
> Thanks for sending the sample data. I ran it through the debugger
and
> confirmed that you've uncovered a bug. The good news is that using
the
> "-by FCST_THRESH" option will produce accurate results.
>
> But specifying a specific threshold (.e.g. -fcst_thresh
'ge0.1&&le1.5'
> has no effect. Specifically, the problem is in this function:
> bool SingleThresh::operator==(const SingleThresh &st) const
>
> In the source code file named "threshold.cc".
>
> Thresholds which include && or || are stored as a "Complex" type.
And the
> logic in the function is not sufficient. It returns true as long as
both
> thresholds being compared are Complex (i.e. include && or ||).
>
> We'll need to come up with more robust logic for checking for the
equality
> of complex threshold types.
>
> Thanks,
> John
>
> On Thu, Sep 20, 2018 at 3:36 PM Zhang, Mandy via RT
<met_help at ucar.edu>
> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036 >
>>
>> Zhang, Mandy has shared a OneDrive for Business file with you. To
view
>> it, click the link below.
>>
>>
>> <
>> https://sooners-my.sharepoint.com/:u:/g/personal/man_zhang-
1_ou_edu/EawqFRkeyX5Ep4STupok7g0BnM7yNvkNUtlxa3R-BxpMnQ
>> >
>> [https://r1.res.office365.com/owa/prem/images/dc-generic_20.png]<
>> https://sooners-my.sharepoint.com/:u:/g/personal/man_zhang-
1_ou_edu/EawqFRkeyX5Ep4STupok7g0BnM7yNvkNUtlxa3R-BxpMnQ
>> >
>>
>> job_filter_stat_ge0.1le1.5<
>> https://sooners-my.sharepoint.com/:u:/g/personal/man_zhang-
1_ou_edu/EawqFRkeyX5Ep4STupok7g0BnM7yNvkNUtlxa3R-BxpMnQ
>> >
>>
>>
>>
>>
>> Hi John,
>>
>> I see, please find my attachment, it looks like "thresh" doesn't
work.
>>
>> my command is :
>>
>> *******************
>>
>> #!/bin/sh
>>
>> echo
>> echo "*** Running STAT-Analysis ***"
>> /home-gs/users/SZQXG152/METOOLS/met/bin/stat_analysis \
>> -lookin
>> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801*/0600/point_stat/02
>> \
>> -job filter -line_type CTC -fcst_var APCP -fcst_lev A1
-interp_mthd
>> DW_MEAN -fcst_thresh 'ge0.1&&le1.5' -dump_row
>> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
>> \
>>
>> -v 2
>> ***
>>
>> Thank you!
>>
>> Mandy
>>
>> ________________________________
>> From: John Halley Gotway via RT <met_help at ucar.edu>
>> Sent: Thursday, September 20, 2018 4:03:36 PM
>> To: Zhang, Mandy
>> Subject: Re: [rt.rap.ucar.edu #87036] threshold of Stat_analysis
with CTS
>>
>> Mandy,
>>
>> OK, I see what's going on. When you run STAT-Analysis on the
command
>> line,
>> you can only run one job at a time. In the command you sent,
you're
>> trying
>> to run 2 of them on the command line. That doesn't work. You can
use
>> STAT-Analysis to run multiple jobs, but you would need to define
them in a
>> config file.
>>
>> So first run this job:
>> /home-gs/users/SZQXG152/METOOLS/met/bin/stat_analysis \
>> -lookin
>>
>> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801*/0600/point_stat/02
>> \
>> -job filter -line_type CTC -fcst_var APCP -fcst_lev A1
-interp_mthd
>> DW_MEAN -fcst_thresh 'ge0.1&&le1.5' -dump_row
>>
>> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
>>
>> And then look in this output file:
>>
>>
>> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
>>
>> How many lines of output do you see, 800 or 4000?
>> Look at the FCST_THRESH column. Does it only contain data for
>> 'ge0.1&&le1.5'?
>>
>> Thanks,
>> John
>>
>> On Thu, Sep 20, 2018 at 2:24 PM Zhang, Mandy via RT
<met_help at ucar.edu>
>> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036 >
>> >
>> > Hi John,
>> >
>> > As you said, after I ran job filter with "-job filter
-fcst_var APCP
>> > -fcst_lev A1 -interp_mthd DW_MEAN -fcst_thresh
>> > '>=0.1&&<=1.5' -line_type CTC \", it shows 800 total column,
>> >
>> > and my question now is:
>> >
>> > if i only do stat_analysis on specified threshold, do i need to
run it
>> > with 2 steps: 1. job filter the data and get the data of
filter.stat 2.
>> > -job aggregate_stat with output of job_filter (filter.stat).
>> >
>> > I try to write job filter and job aggregate_stat in one script
as:
>> >
>> > #!/bin/sh
>> >
>> > echo
>> > echo "*** Running STAT-Analysis ***"
>> > /home-gs/users/SZQXG152/METOOLS/met/bin/stat_analysis \
>> > -lookin
>> >
>> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801*/0600/point_stat/02
>> > \
>> > -job filter -line_type CTC -fcst_var APCP -fcst_lev A1
-interp_mthd
>> > DW_MEAN -fcst_thresh 'ge0.1&&le1.5' -dump_row
>> >
>> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
>> > \
>> > -job aggregate_stat -line_type CTC -out_line_type CTS
-fcst_var APCP
>> > -fcst_lev A1 -interp_mthd DW_MEAN -fcst_thresh 'ge0.1&&le1.5'
-dump_row
>> >
>> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_aggregate_stat_'ge0.1&&le1.5'.stat
>> > -out
>> >
>> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/CTCAPCP_A1_1mon_filter.txt
>> > \
>> > -v 2
>> > ##############
>> >
>> > but the error is :
>> >
>> > *** Running STAT-Analysis ***
>> > DEBUG 2: Processing 9 STAT files.
>> > DEBUG 2: STAT Lines read = 66672
>> > DEBUG 2: STAT Lines retained = 20
>> > DEBUG 2:
>> > DEBUG 2: Processing Job 1: -job aggregate_stat -fcst_var APCP
-fcst_var
>> > APCP -fcst_lev A1 -fcst_lev A1 -interp_mthd DW_MEAN -interp_mthd
DW_MEAN
>> > -fcst_thresh >=0.1&&<=1.5 -fcst_thresh >=0.1&&<=1.5 -line_type
CTC
>> > -line_type CTC -dump_row
>> >
>> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/01/job_aggregate_stat_ge0.1&&le1.5.stat
>> > -out_line_type CTS -out_alpha 0.05000
>> > DEBUG 1: Creating dump row output file
>> >
>> "/home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/01/job_aggregate_stat_ge0.1&&le1.5.stat"
>> > GSL_RNG_TYPE=mt19937
>> > GSL_RNG_SEED=18446744072668357467
>> > ERROR :
>> > ERROR : do_job_aggr_stat() -> this function may only be called
when the
>> > "-line_type" and "-out_line_type" options have been used exactly
once:
>> -job
>> > aggregate_stat -fcst_var APCP -fcst_var APCP -fcst_lev A1
-fcst_lev A1
>> > -interp_mthd DW_MEAN -interp_mthd DW_MEAN -fcst_thresh
>=0.1&&<=1.5
>> > -fcst_thresh >=0.1&&<=1.5 -line_type CTC -line_type CTC -dump_row
>> >
>> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/01/job_aggregate_stat_ge0.1&&le1.5.stat
>> > -out_line_type CTS -out_alpha 0.05000
>> > ERROR :
>> > ERROR :
>> > ERROR : main() -> encountered an error value of 1. Calling
clean_up()
>> > and usage() before exiting.
>> > ERROR :
>> >
>> > #############
>> >
>> > Do you know what's the issue?
>> >
>> > Thank you!
>> >
>> > Mandy
>> >
>> >
>> > ________________________________
>> > From: John Halley Gotway via RT <met_help at ucar.edu>
>> > Sent: Thursday, September 20, 2018 11:34:09 AM
>> > To: Zhang, Mandy
>> > Subject: Re: [rt.rap.ucar.edu #87036] threshold of Stat_analysis
with
>> CTS
>> >
>> > Mandy,
>> >
>> >
>> > Thanks for sending your sample data. I see the discrepancy you
>> describe,
>> > but there's a good explanation for it. With the commands listed
below,
>> I
>> > grabbed the columns for the FCST_THRESH, TOTAL, and GSS. You
pointed
>> out
>> > that with the "-by" option you got a GSS value of -0.00047107 but
with
>> > "-fcst_thresh" you got "0.0034811".
>> >
>> >
>> > But play close attention to the TOTAL column. In the first case,
TOTAL
>> is
>> > 800 but in the second case it's 4000. That indicates to me that
the
>> these
>> > jobs were not run on the same set of .stat data.
>> >
>> >
>> > FYI, you may find using the "-dump_row" job command option to be
helpful
>> > (e.g. -dump_row data_for_agg_by_fcst_thresh.stat). The "-
dump_row"
>> option
>> > tells stat_analysis to write all the input files that were used
for that
>> > job to the specified output file. That enables you to confirm
that you
>> ran
>> > the job on the exact set of data you intended.
>> >
>> >
>> >
>> > > egrep -v JOB_LIST CTSAPCP_A1_1mon.txt | awk '{print $2,$3,$47}'
>> >
>> > FCST_THRESH TOTAL GSS
>> >
>> > >=0.1&&<=1.5 800 -0.00047107
>> >
>> > >=1.6&&<=6.9 800 NA
>> >
>> > >=15.0&&<=39.9 800 NA
>> >
>> > >=40.0&&<=49.9 800 NA
>> >
>> > >=50.0 800 NA
>> >
>> > >=7.0&&<=14.9 800 NA
>> >
>> >
>> > > egrep -v JOB_LIST CTSAPCP_A1_1mon_ge0.1le1.5.txt | awk '{print
>> > $1,$2,$46}'
>> >
>> > COL_NAME: TOTAL GSS
>> >
>> > CTS: 4000 0.0034811
>> >
>> >
>> > So the real question is why are you getting 800 vs 4000 in the
TOTAL
>> > column? Perhaps setting "-fcst_thresh '>=0.1&&<=1.5'" isn't
filtering
>> the
>> > data the way we expect. You could run a "filter" job on your
data to
>> test
>> > this...
>> >
>> >
>> > -job filter -fcst_var APCP -fcst_lev A1 -interp_mthd DW_MEAN
>> -fcst_thresh
>> > '>=0.1&&<=1.5' -line_type CTC \
>> >
>> > -dump_row filter.stat
>> >
>> >
>> > Then look to see if stat-analysis was successful in filtering out
the
>> lines
>> > you want.
>> >
>> >
>> > Hope this helps.
>> >
>> >
>> > Thanks,
>> > John
>> >
>> > On Wed, Sep 19, 2018 at 3:24 PM Zhang, Mandy via RT
<met_help at ucar.edu>
>> > wrote:
>> >
>> > >
>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036 >
>> > >
>> > > Zhang, Mandy has shared OneDrive for Business files with you.
To view
>> > > them, click the links below.
>> > >
>> > >
>> > > <
>> > >
>> >
>> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
>> > > >
>> > > [https://r1.res.office365.com/owa/prem/images/dc-txt_20.png]<
>> > >
>> >
>> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
>> > > >
>> > >
>> > > CTSAPCP_A1_1mon_ge0.1le1.5.txt<
>> > >
>> >
>> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
>> > > >
>> > >
>> > > <
>> > >
>> >
>> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
>> > > >
>> > > [https://r1.res.office365.com/owa/prem/images/dc-txt_20.png]<
>> > >
>> >
>> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
>> > > >
>> > >
>> > > CTSAPCP_A1_1mon.txt<
>> > >
>> >
>> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
>> > > >
>> > >
>> > >
>> > >
>> > >
>> > > Hi John,
>> > >
>> > > I calculate the precipitation by using option "-by
FCST_THRESH" and
>> > > compared to the result with option "-fcst_thresh ", they show
>> different
>> > > value of GSS. the value of "-by FCST_THRESH" is negative and
seems
>> > > unreasonable.
>> > >
>> > > please find my attachment.
>> > >
>> > > Thank you very much.
>> > >
>> > > Mandy
>> > >
>> > > ________________________________
>> > > From: John Halley Gotway via RT <met_help at ucar.edu>
>> > > Sent: Wednesday, September 19, 2018 12:30:21 PM
>> > > To: Zhang, Mandy
>> > > Subject: Re: [rt.rap.ucar.edu #87036] threshold of
Stat_analysis with
>> > CTS
>> > >
>> > > Mandy,
>> > >
>> > > Instead of specifying "-fcst_thresh '<271'" to select a single
>> threshold,
>> > > please try using this setting instead: "-by FCST_THRESH".
>> > >
>> > > That'll run the same job but separately for each unique value
found in
>> > the
>> > > FCST_THRESH column. And you can specify multiple columns
too...
>> > > -by FCST_THRESH,FCST_LEAD
>> > >
>> > > Is that what you're looking for?
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > > On Wed, Sep 19, 2018 at 10:15 AM Zhang, Mandy via RT <
>> met_help at ucar.edu>
>> > > wrote:
>> > >
>> > > >
>> > > > Wed Sep 19 10:14:40 2018: Request 87036 was acted upon.
>> > > > Transaction: Ticket created by Man.Zhang-1 at ou.edu
>> > > > Queue: met_help
>> > > > Subject: threshold of Stat_analysis with CTS
>> > > > Owner: Nobody
>> > > > Requestors: Man.Zhang-1 at ou.edu
>> > > > Status: new
>> > > > Ticket <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036
>> > >
>> > > >
>> > > >
>> > > > Hi John,
>> > > >
>> > > > As we talked before, when we run Stat_Analysis with CTS,
we
>> need to
>> > > > set threshold, if the variables like temperature has the
threshold
>> as
>> > > > [<271,>=
>> > > >
>> > >
>> >
>>
271&&<275,>=275&&<283,>=283&&<295,>=295&&<308,>=308&&<310,>=310&&<313,>=313];
>> > > > does it mean I have to run 8 times of stat_analysis with each
>> > threshold?
>> > > > Thank you
>> > > > Mandy
>> > > >
>> > > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>>
>>
>>
------------------------------------------------
Subject: threshold of Stat_analysis with CTS
From: Zhang, Mandy
Time: Fri Sep 21 16:43:30 2018
Hi John,
Thank you for your thoughtful help, I would like you post it as
bugfix for met-7.0.
Have a good weekend!
Mandy
________________________________
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Friday, September 21, 2018 3:58:24 PM
To: Zhang, Mandy
Subject: Re: [rt.rap.ucar.edu #87036] threshold of Stat_analysis with
CTS
Mandy,
OK, I just committed a bugfix to the development version of MET that
will
be included in the upcoming met-8.0 release.
When checking for equality of thresholds, for "Complex" types (i.e.
with &&
or ||), check for equality of their string representations.
Converting
them to strings is a bit slower... so for the "Simple" types, we'll
use the
old, faster logic.
I tested and confirmed that this change enables the -fcst_thresh
option in
stat_analysis to work as you'd expect on your data.
Would you also like this fix posted as a bugfix for met-7.0 or is
using the
upcoming 8.0 release sufficient?
Thanks,
John
On Fri, Sep 21, 2018 at 2:29 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Mandy,
>
> Thanks for sending the sample data. I ran it through the debugger
and
> confirmed that you've uncovered a bug. The good news is that using
the
> "-by FCST_THRESH" option will produce accurate results.
>
> But specifying a specific threshold (.e.g. -fcst_thresh
'ge0.1&&le1.5'
> has no effect. Specifically, the problem is in this function:
> bool SingleThresh::operator==(const SingleThresh &st) const
>
> In the source code file named "threshold.cc".
>
> Thresholds which include && or || are stored as a "Complex" type.
And the
> logic in the function is not sufficient. It returns true as long as
both
> thresholds being compared are Complex (i.e. include && or ||).
>
> We'll need to come up with more robust logic for checking for the
equality
> of complex threshold types.
>
> Thanks,
> John
>
> On Thu, Sep 20, 2018 at 3:36 PM Zhang, Mandy via RT
<met_help at ucar.edu>
> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036 >
>>
>> Zhang, Mandy has shared a OneDrive for Business file with you. To
view
>> it, click the link below.
>>
>>
>> <
>> https://sooners-my.sharepoint.com/:u:/g/personal/man_zhang-
1_ou_edu/EawqFRkeyX5Ep4STupok7g0BnM7yNvkNUtlxa3R-BxpMnQ
>> >
>> [https://r1.res.office365.com/owa/prem/images/dc-generic_20.png]<
>> https://sooners-my.sharepoint.com/:u:/g/personal/man_zhang-
1_ou_edu/EawqFRkeyX5Ep4STupok7g0BnM7yNvkNUtlxa3R-BxpMnQ
>> >
>>
>> job_filter_stat_ge0.1le1.5<
>> https://sooners-my.sharepoint.com/:u:/g/personal/man_zhang-
1_ou_edu/EawqFRkeyX5Ep4STupok7g0BnM7yNvkNUtlxa3R-BxpMnQ
>> >
>>
>>
>>
>>
>> Hi John,
>>
>> I see, please find my attachment, it looks like "thresh" doesn't
work.
>>
>> my command is :
>>
>> *******************
>>
>> #!/bin/sh
>>
>> echo
>> echo "*** Running STAT-Analysis ***"
>> /home-gs/users/SZQXG152/METOOLS/met/bin/stat_analysis \
>> -lookin
>> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801*/0600/point_stat/02
>> \
>> -job filter -line_type CTC -fcst_var APCP -fcst_lev A1
-interp_mthd
>> DW_MEAN -fcst_thresh 'ge0.1&&le1.5' -dump_row
>> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
>> \
>>
>> -v 2
>> ***
>>
>> Thank you!
>>
>> Mandy
>>
>> ________________________________
>> From: John Halley Gotway via RT <met_help at ucar.edu>
>> Sent: Thursday, September 20, 2018 4:03:36 PM
>> To: Zhang, Mandy
>> Subject: Re: [rt.rap.ucar.edu #87036] threshold of Stat_analysis
with CTS
>>
>> Mandy,
>>
>> OK, I see what's going on. When you run STAT-Analysis on the
command
>> line,
>> you can only run one job at a time. In the command you sent,
you're
>> trying
>> to run 2 of them on the command line. That doesn't work. You can
use
>> STAT-Analysis to run multiple jobs, but you would need to define
them in a
>> config file.
>>
>> So first run this job:
>> /home-gs/users/SZQXG152/METOOLS/met/bin/stat_analysis \
>> -lookin
>>
>> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801*/0600/point_stat/02
>> \
>> -job filter -line_type CTC -fcst_var APCP -fcst_lev A1
-interp_mthd
>> DW_MEAN -fcst_thresh 'ge0.1&&le1.5' -dump_row
>>
>> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
>>
>> And then look in this output file:
>>
>>
>> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
>>
>> How many lines of output do you see, 800 or 4000?
>> Look at the FCST_THRESH column. Does it only contain data for
>> 'ge0.1&&le1.5'?
>>
>> Thanks,
>> John
>>
>> On Thu, Sep 20, 2018 at 2:24 PM Zhang, Mandy via RT
<met_help at ucar.edu>
>> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036 >
>> >
>> > Hi John,
>> >
>> > As you said, after I ran job filter with "-job filter
-fcst_var APCP
>> > -fcst_lev A1 -interp_mthd DW_MEAN -fcst_thresh
>> > '>=0.1&&<=1.5' -line_type CTC \", it shows 800 total column,
>> >
>> > and my question now is:
>> >
>> > if i only do stat_analysis on specified threshold, do i need to
run it
>> > with 2 steps: 1. job filter the data and get the data of
filter.stat 2.
>> > -job aggregate_stat with output of job_filter (filter.stat).
>> >
>> > I try to write job filter and job aggregate_stat in one script
as:
>> >
>> > #!/bin/sh
>> >
>> > echo
>> > echo "*** Running STAT-Analysis ***"
>> > /home-gs/users/SZQXG152/METOOLS/met/bin/stat_analysis \
>> > -lookin
>> >
>> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801*/0600/point_stat/02
>> > \
>> > -job filter -line_type CTC -fcst_var APCP -fcst_lev A1
-interp_mthd
>> > DW_MEAN -fcst_thresh 'ge0.1&&le1.5' -dump_row
>> >
>> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
>> > \
>> > -job aggregate_stat -line_type CTC -out_line_type CTS
-fcst_var APCP
>> > -fcst_lev A1 -interp_mthd DW_MEAN -fcst_thresh 'ge0.1&&le1.5'
-dump_row
>> >
>> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_aggregate_stat_'ge0.1&&le1.5'.stat
>> > -out
>> >
>> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/CTCAPCP_A1_1mon_filter.txt
>> > \
>> > -v 2
>> > ##############
>> >
>> > but the error is :
>> >
>> > *** Running STAT-Analysis ***
>> > DEBUG 2: Processing 9 STAT files.
>> > DEBUG 2: STAT Lines read = 66672
>> > DEBUG 2: STAT Lines retained = 20
>> > DEBUG 2:
>> > DEBUG 2: Processing Job 1: -job aggregate_stat -fcst_var APCP
-fcst_var
>> > APCP -fcst_lev A1 -fcst_lev A1 -interp_mthd DW_MEAN -interp_mthd
DW_MEAN
>> > -fcst_thresh >=0.1&&<=1.5 -fcst_thresh >=0.1&&<=1.5 -line_type
CTC
>> > -line_type CTC -dump_row
>> >
>> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/01/job_aggregate_stat_ge0.1&&le1.5.stat
>> > -out_line_type CTS -out_alpha 0.05000
>> > DEBUG 1: Creating dump row output file
>> >
>> "/home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/01/job_aggregate_stat_ge0.1&&le1.5.stat"
>> > GSL_RNG_TYPE=mt19937
>> > GSL_RNG_SEED=18446744072668357467
>> > ERROR :
>> > ERROR : do_job_aggr_stat() -> this function may only be called
when the
>> > "-line_type" and "-out_line_type" options have been used exactly
once:
>> -job
>> > aggregate_stat -fcst_var APCP -fcst_var APCP -fcst_lev A1
-fcst_lev A1
>> > -interp_mthd DW_MEAN -interp_mthd DW_MEAN -fcst_thresh
>=0.1&&<=1.5
>> > -fcst_thresh >=0.1&&<=1.5 -line_type CTC -line_type CTC -dump_row
>> >
>> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/01/job_aggregate_stat_ge0.1&&le1.5.stat
>> > -out_line_type CTS -out_alpha 0.05000
>> > ERROR :
>> > ERROR :
>> > ERROR : main() -> encountered an error value of 1. Calling
clean_up()
>> > and usage() before exiting.
>> > ERROR :
>> >
>> > #############
>> >
>> > Do you know what's the issue?
>> >
>> > Thank you!
>> >
>> > Mandy
>> >
>> >
>> > ________________________________
>> > From: John Halley Gotway via RT <met_help at ucar.edu>
>> > Sent: Thursday, September 20, 2018 11:34:09 AM
>> > To: Zhang, Mandy
>> > Subject: Re: [rt.rap.ucar.edu #87036] threshold of Stat_analysis
with
>> CTS
>> >
>> > Mandy,
>> >
>> >
>> > Thanks for sending your sample data. I see the discrepancy you
>> describe,
>> > but there's a good explanation for it. With the commands listed
below,
>> I
>> > grabbed the columns for the FCST_THRESH, TOTAL, and GSS. You
pointed
>> out
>> > that with the "-by" option you got a GSS value of -0.00047107 but
with
>> > "-fcst_thresh" you got "0.0034811".
>> >
>> >
>> > But play close attention to the TOTAL column. In the first case,
TOTAL
>> is
>> > 800 but in the second case it's 4000. That indicates to me that
the
>> these
>> > jobs were not run on the same set of .stat data.
>> >
>> >
>> > FYI, you may find using the "-dump_row" job command option to be
helpful
>> > (e.g. -dump_row data_for_agg_by_fcst_thresh.stat). The "-
dump_row"
>> option
>> > tells stat_analysis to write all the input files that were used
for that
>> > job to the specified output file. That enables you to confirm
that you
>> ran
>> > the job on the exact set of data you intended.
>> >
>> >
>> >
>> > > egrep -v JOB_LIST CTSAPCP_A1_1mon.txt | awk '{print $2,$3,$47}'
>> >
>> > FCST_THRESH TOTAL GSS
>> >
>> > >=0.1&&<=1.5 800 -0.00047107
>> >
>> > >=1.6&&<=6.9 800 NA
>> >
>> > >=15.0&&<=39.9 800 NA
>> >
>> > >=40.0&&<=49.9 800 NA
>> >
>> > >=50.0 800 NA
>> >
>> > >=7.0&&<=14.9 800 NA
>> >
>> >
>> > > egrep -v JOB_LIST CTSAPCP_A1_1mon_ge0.1le1.5.txt | awk '{print
>> > $1,$2,$46}'
>> >
>> > COL_NAME: TOTAL GSS
>> >
>> > CTS: 4000 0.0034811
>> >
>> >
>> > So the real question is why are you getting 800 vs 4000 in the
TOTAL
>> > column? Perhaps setting "-fcst_thresh '>=0.1&&<=1.5'" isn't
filtering
>> the
>> > data the way we expect. You could run a "filter" job on your
data to
>> test
>> > this...
>> >
>> >
>> > -job filter -fcst_var APCP -fcst_lev A1 -interp_mthd DW_MEAN
>> -fcst_thresh
>> > '>=0.1&&<=1.5' -line_type CTC \
>> >
>> > -dump_row filter.stat
>> >
>> >
>> > Then look to see if stat-analysis was successful in filtering out
the
>> lines
>> > you want.
>> >
>> >
>> > Hope this helps.
>> >
>> >
>> > Thanks,
>> > John
>> >
>> > On Wed, Sep 19, 2018 at 3:24 PM Zhang, Mandy via RT
<met_help at ucar.edu>
>> > wrote:
>> >
>> > >
>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036 >
>> > >
>> > > Zhang, Mandy has shared OneDrive for Business files with you.
To view
>> > > them, click the links below.
>> > >
>> > >
>> > > <
>> > >
>> >
>> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
>> > > >
>> > > [https://r1.res.office365.com/owa/prem/images/dc-txt_20.png]<
>> > >
>> >
>> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
>> > > >
>> > >
>> > > CTSAPCP_A1_1mon_ge0.1le1.5.txt<
>> > >
>> >
>> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
>> > > >
>> > >
>> > > <
>> > >
>> >
>> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
>> > > >
>> > > [https://r1.res.office365.com/owa/prem/images/dc-txt_20.png]<
>> > >
>> >
>> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
>> > > >
>> > >
>> > > CTSAPCP_A1_1mon.txt<
>> > >
>> >
>> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
>> > > >
>> > >
>> > >
>> > >
>> > >
>> > > Hi John,
>> > >
>> > > I calculate the precipitation by using option "-by
FCST_THRESH" and
>> > > compared to the result with option "-fcst_thresh ", they show
>> different
>> > > value of GSS. the value of "-by FCST_THRESH" is negative and
seems
>> > > unreasonable.
>> > >
>> > > please find my attachment.
>> > >
>> > > Thank you very much.
>> > >
>> > > Mandy
>> > >
>> > > ________________________________
>> > > From: John Halley Gotway via RT <met_help at ucar.edu>
>> > > Sent: Wednesday, September 19, 2018 12:30:21 PM
>> > > To: Zhang, Mandy
>> > > Subject: Re: [rt.rap.ucar.edu #87036] threshold of
Stat_analysis with
>> > CTS
>> > >
>> > > Mandy,
>> > >
>> > > Instead of specifying "-fcst_thresh '<271'" to select a single
>> threshold,
>> > > please try using this setting instead: "-by FCST_THRESH".
>> > >
>> > > That'll run the same job but separately for each unique value
found in
>> > the
>> > > FCST_THRESH column. And you can specify multiple columns
too...
>> > > -by FCST_THRESH,FCST_LEAD
>> > >
>> > > Is that what you're looking for?
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > > On Wed, Sep 19, 2018 at 10:15 AM Zhang, Mandy via RT <
>> met_help at ucar.edu>
>> > > wrote:
>> > >
>> > > >
>> > > > Wed Sep 19 10:14:40 2018: Request 87036 was acted upon.
>> > > > Transaction: Ticket created by Man.Zhang-1 at ou.edu
>> > > > Queue: met_help
>> > > > Subject: threshold of Stat_analysis with CTS
>> > > > Owner: Nobody
>> > > > Requestors: Man.Zhang-1 at ou.edu
>> > > > Status: new
>> > > > Ticket <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036
>> > >
>> > > >
>> > > >
>> > > > Hi John,
>> > > >
>> > > > As we talked before, when we run Stat_Analysis with CTS,
we
>> need to
>> > > > set threshold, if the variables like temperature has the
threshold
>> as
>> > > > [<271,>=
>> > > >
>> > >
>> >
>>
271&&<275,>=275&&<283,>=283&&<295,>=295&&<308,>=308&&<310,>=310&&<313,>=313];
>> > > > does it mean I have to run 8 times of stat_analysis with each
>> > threshold?
>> > > > Thank you
>> > > > Mandy
>> > > >
>> > > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>>
>>
>>
------------------------------------------------
Subject: threshold of Stat_analysis with CTS
From: John Halley Gotway
Time: Mon Sep 24 16:24:01 2018
Mandy,
I just posted the fix for this issue as a bugfix for met-7.0:
https://dtcenter.org/met/users/support/known_issues/METv7.0/index.php
Please let me know how it goes.
Thanks,
John
On Fri, Sep 21, 2018 at 4:43 PM Zhang, Mandy via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036 >
>
> Hi John,
>
> Thank you for your thoughtful help, I would like you post it as
> bugfix for met-7.0.
>
> Have a good weekend!
>
> Mandy
>
> ________________________________
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Friday, September 21, 2018 3:58:24 PM
> To: Zhang, Mandy
> Subject: Re: [rt.rap.ucar.edu #87036] threshold of Stat_analysis
with CTS
>
> Mandy,
>
> OK, I just committed a bugfix to the development version of MET that
will
> be included in the upcoming met-8.0 release.
>
> When checking for equality of thresholds, for "Complex" types (i.e.
with &&
> or ||), check for equality of their string representations.
Converting
> them to strings is a bit slower... so for the "Simple" types, we'll
use the
> old, faster logic.
>
> I tested and confirmed that this change enables the -fcst_thresh
option in
> stat_analysis to work as you'd expect on your data.
>
> Would you also like this fix posted as a bugfix for met-7.0 or is
using the
> upcoming 8.0 release sufficient?
>
> Thanks,
> John
>
> On Fri, Sep 21, 2018 at 2:29 PM John Halley Gotway <johnhg at ucar.edu>
> wrote:
>
> > Mandy,
> >
> > Thanks for sending the sample data. I ran it through the debugger
and
> > confirmed that you've uncovered a bug. The good news is that
using the
> > "-by FCST_THRESH" option will produce accurate results.
> >
> > But specifying a specific threshold (.e.g. -fcst_thresh
'ge0.1&&le1.5'
> > has no effect. Specifically, the problem is in this function:
> > bool SingleThresh::operator==(const SingleThresh &st) const
> >
> > In the source code file named "threshold.cc".
> >
> > Thresholds which include && or || are stored as a "Complex" type.
And
> the
> > logic in the function is not sufficient. It returns true as long
as both
> > thresholds being compared are Complex (i.e. include && or ||).
> >
> > We'll need to come up with more robust logic for checking for the
> equality
> > of complex threshold types.
> >
> > Thanks,
> > John
> >
> > On Thu, Sep 20, 2018 at 3:36 PM Zhang, Mandy via RT
<met_help at ucar.edu>
> > wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036 >
> >>
> >> Zhang, Mandy has shared a OneDrive for Business file with you. To
view
> >> it, click the link below.
> >>
> >>
> >> <
> >>
> https://sooners-my.sharepoint.com/:u:/g/personal/man_zhang-
1_ou_edu/EawqFRkeyX5Ep4STupok7g0BnM7yNvkNUtlxa3R-BxpMnQ
> >> >
> >> [https://r1.res.office365.com/owa/prem/images/dc-generic_20.png]<
> >>
> https://sooners-my.sharepoint.com/:u:/g/personal/man_zhang-
1_ou_edu/EawqFRkeyX5Ep4STupok7g0BnM7yNvkNUtlxa3R-BxpMnQ
> >> >
> >>
> >> job_filter_stat_ge0.1le1.5<
> >>
> https://sooners-my.sharepoint.com/:u:/g/personal/man_zhang-
1_ou_edu/EawqFRkeyX5Ep4STupok7g0BnM7yNvkNUtlxa3R-BxpMnQ
> >> >
> >>
> >>
> >>
> >>
> >> Hi John,
> >>
> >> I see, please find my attachment, it looks like "thresh" doesn't
work.
> >>
> >> my command is :
> >>
> >> *******************
> >>
> >> #!/bin/sh
> >>
> >> echo
> >> echo "*** Running STAT-Analysis ***"
> >> /home-gs/users/SZQXG152/METOOLS/met/bin/stat_analysis \
> >> -lookin
> >>
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801*/0600/point_stat/02
> >> \
> >> -job filter -line_type CTC -fcst_var APCP -fcst_lev A1
-interp_mthd
> >> DW_MEAN -fcst_thresh 'ge0.1&&le1.5' -dump_row
> >>
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
> >> \
> >>
> >> -v 2
> >> ***
> >>
> >> Thank you!
> >>
> >> Mandy
> >>
> >> ________________________________
> >> From: John Halley Gotway via RT <met_help at ucar.edu>
> >> Sent: Thursday, September 20, 2018 4:03:36 PM
> >> To: Zhang, Mandy
> >> Subject: Re: [rt.rap.ucar.edu #87036] threshold of Stat_analysis
with
> CTS
> >>
> >> Mandy,
> >>
> >> OK, I see what's going on. When you run STAT-Analysis on the
command
> >> line,
> >> you can only run one job at a time. In the command you sent,
you're
> >> trying
> >> to run 2 of them on the command line. That doesn't work. You
can use
> >> STAT-Analysis to run multiple jobs, but you would need to define
them
> in a
> >> config file.
> >>
> >> So first run this job:
> >> /home-gs/users/SZQXG152/METOOLS/met/bin/stat_analysis \
> >> -lookin
> >>
> >>
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801*/0600/point_stat/02
> >> \
> >> -job filter -line_type CTC -fcst_var APCP -fcst_lev A1
-interp_mthd
> >> DW_MEAN -fcst_thresh 'ge0.1&&le1.5' -dump_row
> >>
> >>
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
> >>
> >> And then look in this output file:
> >>
> >>
> >>
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
> >>
> >> How many lines of output do you see, 800 or 4000?
> >> Look at the FCST_THRESH column. Does it only contain data for
> >> 'ge0.1&&le1.5'?
> >>
> >> Thanks,
> >> John
> >>
> >> On Thu, Sep 20, 2018 at 2:24 PM Zhang, Mandy via RT
<met_help at ucar.edu>
> >> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036 >
> >> >
> >> > Hi John,
> >> >
> >> > As you said, after I ran job filter with "-job filter
-fcst_var
> APCP
> >> > -fcst_lev A1 -interp_mthd DW_MEAN -fcst_thresh
> >> > '>=0.1&&<=1.5' -line_type CTC \", it shows 800 total column,
> >> >
> >> > and my question now is:
> >> >
> >> > if i only do stat_analysis on specified threshold, do i need to
run it
> >> > with 2 steps: 1. job filter the data and get the data of
filter.stat
> 2.
> >> > -job aggregate_stat with output of job_filter (filter.stat).
> >> >
> >> > I try to write job filter and job aggregate_stat in one script
as:
> >> >
> >> > #!/bin/sh
> >> >
> >> > echo
> >> > echo "*** Running STAT-Analysis ***"
> >> > /home-gs/users/SZQXG152/METOOLS/met/bin/stat_analysis \
> >> > -lookin
> >> >
> >>
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801*/0600/point_stat/02
> >> > \
> >> > -job filter -line_type CTC -fcst_var APCP -fcst_lev A1
-interp_mthd
> >> > DW_MEAN -fcst_thresh 'ge0.1&&le1.5' -dump_row
> >> >
> >>
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_filter_stat_'ge0.1&&le1.5'.stat
> >> > \
> >> > -job aggregate_stat -line_type CTC -out_line_type CTS
-fcst_var
> APCP
> >> > -fcst_lev A1 -interp_mthd DW_MEAN -fcst_thresh 'ge0.1&&le1.5'
> -dump_row
> >> >
> >>
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/job_aggregate_stat_'ge0.1&&le1.5'.stat
> >> > -out
> >> >
> >>
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/02/CTCAPCP_A1_1mon_filter.txt
> >> > \
> >> > -v 2
> >> > ##############
> >> >
> >> > but the error is :
> >> >
> >> > *** Running STAT-Analysis ***
> >> > DEBUG 2: Processing 9 STAT files.
> >> > DEBUG 2: STAT Lines read = 66672
> >> > DEBUG 2: STAT Lines retained = 20
> >> > DEBUG 2:
> >> > DEBUG 2: Processing Job 1: -job aggregate_stat -fcst_var APCP
> -fcst_var
> >> > APCP -fcst_lev A1 -fcst_lev A1 -interp_mthd DW_MEAN
-interp_mthd
> DW_MEAN
> >> > -fcst_thresh >=0.1&&<=1.5 -fcst_thresh >=0.1&&<=1.5 -line_type
CTC
> >> > -line_type CTC -dump_row
> >> >
> >>
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/01/job_aggregate_stat_ge0.1&&le1.5.stat
> >> > -out_line_type CTS -out_alpha 0.05000
> >> > DEBUG 1: Creating dump row output file
> >> >
> >>
> "/home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/01/job_aggregate_stat_ge0.1&&le1.5.stat"
> >> > GSL_RNG_TYPE=mt19937
> >> > GSL_RNG_SEED=18446744072668357467
> >> > ERROR :
> >> > ERROR : do_job_aggr_stat() -> this function may only be called
when
> the
> >> > "-line_type" and "-out_line_type" options have been used
exactly once:
> >> -job
> >> > aggregate_stat -fcst_var APCP -fcst_var APCP -fcst_lev A1
-fcst_lev A1
> >> > -interp_mthd DW_MEAN -interp_mthd DW_MEAN -fcst_thresh
>=0.1&&<=1.5
> >> > -fcst_thresh >=0.1&&<=1.5 -line_type CTC -line_type CTC
-dump_row
> >> >
> >>
> /home-
gs/users/SZQXG152/metenm_test/ensm_stat_sfc/met/201801/mon/01/job_aggregate_stat_ge0.1&&le1.5.stat
> >> > -out_line_type CTS -out_alpha 0.05000
> >> > ERROR :
> >> > ERROR :
> >> > ERROR : main() -> encountered an error value of 1. Calling
> clean_up()
> >> > and usage() before exiting.
> >> > ERROR :
> >> >
> >> > #############
> >> >
> >> > Do you know what's the issue?
> >> >
> >> > Thank you!
> >> >
> >> > Mandy
> >> >
> >> >
> >> > ________________________________
> >> > From: John Halley Gotway via RT <met_help at ucar.edu>
> >> > Sent: Thursday, September 20, 2018 11:34:09 AM
> >> > To: Zhang, Mandy
> >> > Subject: Re: [rt.rap.ucar.edu #87036] threshold of
Stat_analysis with
> >> CTS
> >> >
> >> > Mandy,
> >> >
> >> >
> >> > Thanks for sending your sample data. I see the discrepancy you
> >> describe,
> >> > but there's a good explanation for it. With the commands
listed
> below,
> >> I
> >> > grabbed the columns for the FCST_THRESH, TOTAL, and GSS. You
pointed
> >> out
> >> > that with the "-by" option you got a GSS value of -0.00047107
but with
> >> > "-fcst_thresh" you got "0.0034811".
> >> >
> >> >
> >> > But play close attention to the TOTAL column. In the first
case,
> TOTAL
> >> is
> >> > 800 but in the second case it's 4000. That indicates to me
that the
> >> these
> >> > jobs were not run on the same set of .stat data.
> >> >
> >> >
> >> > FYI, you may find using the "-dump_row" job command option to
be
> helpful
> >> > (e.g. -dump_row data_for_agg_by_fcst_thresh.stat). The "-
dump_row"
> >> option
> >> > tells stat_analysis to write all the input files that were used
for
> that
> >> > job to the specified output file. That enables you to confirm
that
> you
> >> ran
> >> > the job on the exact set of data you intended.
> >> >
> >> >
> >> >
> >> > > egrep -v JOB_LIST CTSAPCP_A1_1mon.txt | awk '{print
$2,$3,$47}'
> >> >
> >> > FCST_THRESH TOTAL GSS
> >> >
> >> > >=0.1&&<=1.5 800 -0.00047107
> >> >
> >> > >=1.6&&<=6.9 800 NA
> >> >
> >> > >=15.0&&<=39.9 800 NA
> >> >
> >> > >=40.0&&<=49.9 800 NA
> >> >
> >> > >=50.0 800 NA
> >> >
> >> > >=7.0&&<=14.9 800 NA
> >> >
> >> >
> >> > > egrep -v JOB_LIST CTSAPCP_A1_1mon_ge0.1le1.5.txt | awk
'{print
> >> > $1,$2,$46}'
> >> >
> >> > COL_NAME: TOTAL GSS
> >> >
> >> > CTS: 4000 0.0034811
> >> >
> >> >
> >> > So the real question is why are you getting 800 vs 4000 in the
TOTAL
> >> > column? Perhaps setting "-fcst_thresh '>=0.1&&<=1.5'" isn't
filtering
> >> the
> >> > data the way we expect. You could run a "filter" job on your
data to
> >> test
> >> > this...
> >> >
> >> >
> >> > -job filter -fcst_var APCP -fcst_lev A1 -interp_mthd DW_MEAN
> >> -fcst_thresh
> >> > '>=0.1&&<=1.5' -line_type CTC \
> >> >
> >> > -dump_row filter.stat
> >> >
> >> >
> >> > Then look to see if stat-analysis was successful in filtering
out the
> >> lines
> >> > you want.
> >> >
> >> >
> >> > Hope this helps.
> >> >
> >> >
> >> > Thanks,
> >> > John
> >> >
> >> > On Wed, Sep 19, 2018 at 3:24 PM Zhang, Mandy via RT <
> met_help at ucar.edu>
> >> > wrote:
> >> >
> >> > >
> >> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036
>
> >> > >
> >> > > Zhang, Mandy has shared OneDrive for Business files with you.
To
> view
> >> > > them, click the links below.
> >> > >
> >> > >
> >> > > <
> >> > >
> >> >
> >>
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
> >> > > >
> >> > > [https://r1.res.office365.com/owa/prem/images/dc-txt_20.png]<
> >> > >
> >> >
> >>
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
> >> > > >
> >> > >
> >> > > CTSAPCP_A1_1mon_ge0.1le1.5.txt<
> >> > >
> >> >
> >>
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EbrWnZPVXBlJjACHseDHkNMBv-NieBn0PxEVKDPQHmXzCw
> >> > > >
> >> > >
> >> > > <
> >> > >
> >> >
> >>
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
> >> > > >
> >> > > [https://r1.res.office365.com/owa/prem/images/dc-txt_20.png]<
> >> > >
> >> >
> >>
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
> >> > > >
> >> > >
> >> > > CTSAPCP_A1_1mon.txt<
> >> > >
> >> >
> >>
> https://sooners-my.sharepoint.com/:t:/g/personal/man_zhang-
1_ou_edu/EaNGvWDAc5dIiSehnplw8ncBCd0m1LCt4EbiNJbZJXNjYg
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > Hi John,
> >> > >
> >> > > I calculate the precipitation by using option "-by
FCST_THRESH"
> and
> >> > > compared to the result with option "-fcst_thresh ", they show
> >> different
> >> > > value of GSS. the value of "-by FCST_THRESH" is negative and
seems
> >> > > unreasonable.
> >> > >
> >> > > please find my attachment.
> >> > >
> >> > > Thank you very much.
> >> > >
> >> > > Mandy
> >> > >
> >> > > ________________________________
> >> > > From: John Halley Gotway via RT <met_help at ucar.edu>
> >> > > Sent: Wednesday, September 19, 2018 12:30:21 PM
> >> > > To: Zhang, Mandy
> >> > > Subject: Re: [rt.rap.ucar.edu #87036] threshold of
Stat_analysis
> with
> >> > CTS
> >> > >
> >> > > Mandy,
> >> > >
> >> > > Instead of specifying "-fcst_thresh '<271'" to select a
single
> >> threshold,
> >> > > please try using this setting instead: "-by FCST_THRESH".
> >> > >
> >> > > That'll run the same job but separately for each unique value
found
> in
> >> > the
> >> > > FCST_THRESH column. And you can specify multiple columns
too...
> >> > > -by FCST_THRESH,FCST_LEAD
> >> > >
> >> > > Is that what you're looking for?
> >> > >
> >> > > Thanks,
> >> > > John
> >> > >
> >> > > On Wed, Sep 19, 2018 at 10:15 AM Zhang, Mandy via RT <
> >> met_help at ucar.edu>
> >> > > wrote:
> >> > >
> >> > > >
> >> > > > Wed Sep 19 10:14:40 2018: Request 87036 was acted upon.
> >> > > > Transaction: Ticket created by Man.Zhang-1 at ou.edu
> >> > > > Queue: met_help
> >> > > > Subject: threshold of Stat_analysis with CTS
> >> > > > Owner: Nobody
> >> > > > Requestors: Man.Zhang-1 at ou.edu
> >> > > > Status: new
> >> > > > Ticket <URL:
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87036
> >> > >
> >> > > >
> >> > > >
> >> > > > Hi John,
> >> > > >
> >> > > > As we talked before, when we run Stat_Analysis with
CTS, we
> >> need to
> >> > > > set threshold, if the variables like temperature has the
threshold
> >> as
> >> > > > [<271,>=
> >> > > >
> >> > >
> >> >
> >>
>
271&&<275,>=275&&<283,>=283&&<295,>=295&&<308,>=308&&<310,>=310&&<313,>=313];
> >> > > > does it mean I have to run 8 times of stat_analysis with
each
> >> > threshold?
> >> > > > Thank you
> >> > > > Mandy
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> >
> >> >
> >> >
> >>
> >>
> >>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list