[Met_help] [rt.rap.ucar.edu #68126] History for error from the stat tool.
John Halley Gotway via RT
met_help at ucar.edu
Fri Oct 3 12:30:41 MDT 2014
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi John.
I am using point stat and then the stat analysis tool for generating the aggregate stats of my analysis. I am verifying the point rainfall for various Rf thresholds (gt0.0, ge0.1,ge2.5,ge7.6,ge35.6,ge64.5)
Totally I have generated 33269 MPRs. (XL file attached) and also attaching individual ctc files.
Now following commands for aggregate-stat.
/oprn/model/wrf3/utils/met/METv3.0/bin/stat_analysis \
-job aggregate_stat -line_type MPR -out_line_type CTC -out_fcst_thresh ge0.1 -out_obs_thresh ge0.1 \
-out ~/wrf3/utils/met/METv3.0/geeta_test/output-agst/agst-full24-ge0.1-ctc.dat -v 4 \
-lookin statfiles24 \
#
/oprn/model/wrf3/utils/met/METv3.0/bin/stat_analysis \
-job aggregate_stat -line_type MPR -out_line_type CTS -out_fcst_thresh ge0.1 -out_obs_thresh ge0.1 \
-out ~/wrf3/utils/met/METv3.0/geeta_test/output-agst/agst-full24-ge0.1-cts.dat -v 4 \
-lookin statfiles24 \
#
#
/oprn/model/wrf3/utils/met/METv3.0/bin/stat_analysis \
-job aggregate_stat -line_type MPR -out_line_type CNT -out_fcst_thresh ge0.1 -out_obs_thresh ge0.1 \
-out ~/wrf3/utils/met/METv3.0/geeta_test/output-agst/agst-full24-ge0.1-cnt.dat -v 4 \
-lookin statfiles24 \
#
So I get three o/p files cts, CTC and cnt.
Now when I open ctc files for thresholds 2.5 say,
following is what I get.
OB_LIST: -job aggregate_stat -line_type MPR -out_line_type CTC -out_fcst_thresh >=2.500 -out_obs_thresh >=2.500
COL_NAME: TOTAL FY_OY FY_ON FN_OY FN_ON
CTC: 33544 9185 7287 3801 13271
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
"agst-full24-ge2.5-ctc.dat" 4 lines, 200 characters
So my question is HOW 33269 in changing to 33544????
I am sending my stat files and ctc o/p files.
pls look into that and let e what is hapening
geeta
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: error from the stat tool.
From: John Halley Gotway
Time: Wed Jul 09 12:33:51 2014
Geeta,
I suspect that there is a problem in how you're passing data to the
stat_analysis job. I took the .stat files you sent me and placed them
all
in a directory named "point_stat". I counted up the MPR lines present
in
the .stat files you sent:
> grep MPR point_stat/*.stat | wc -l
> 66538
There are 66538 of them.
Then I ran the following job:
> stat_analysis -lookin point_stat -job aggregate_stat -line_type MPR
-out_line_type CTC -out_fcst_thresh \>=2.500 -out_obs_thresh \>=2.500
COL_NAME: TOTAL FY_OY FY_ON FN_OY FN_ON
CTC: 66538 18214 14414 7528 26382
It used all 66538 MPR lines.
I noticed that some of your .stat files are named "point_stat_ge0.1*"
and
some are named "point_stat_ge0.0*". So I reran my job using only one
type
of those files:
> stat_analysis -lookin point_stat/point_stat_ge0.1*.stat -job
aggregate_stat -line_type MPR -out_line_type CTC -out_fcst_thresh
\>=2.500
-out_obs_thresh \>=2.500
COL_NAME: TOTAL FY_OY FY_ON FN_OY FN_ON
CTC: 33269 9107 7207 3764 13191
So I'm not sure how you're getting output with 33544 matched pairs.
But
you can easily figure this out. Just rerun your job using the "-
dump_row"
option. That will write all of the input lines used for that job to
an
output file. Then you can inspect that file and figure out where the
extra
lines are coming from. I suspect that you're accidentally passing too
much
data to stat_analysis.
Your jobs are telling it to look in the directory named "statfiles24"
(at
least I assume that's a directory). It will use every file in that
directory ending in .stat. Perhaps you've run other stat_analysis
jobs and
written extra ".stat" files into that directory and stat_analysis is
reading them back in. I'm just guessing here, but that would explain
it.
Hope that helps get you going in the right direction.
Thanks,
John
On Wed, Jul 9, 2014 at 1:41 AM, Geeta Geeta via RT <met_help at ucar.edu>
wrote:
>
> Wed Jul 09 01:41:27 2014: Request 68126 was acted upon.
> Transaction: Ticket created by geeta124 at hotmail.com
> Queue: met_help
> Subject: error from the stat tool.
> Owner: Nobody
> Requestors: geeta124 at hotmail.com
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68126 >
>
>
> Hi John.
>
> I am using point stat and then the stat analysis tool for generating
the
> aggregate stats of my analysis. I am verifying the point rainfall
for
> various Rf thresholds (gt0.0, ge0.1,ge2.5,ge7.6,ge35.6,ge64.5)
>
> Totally I have generated 33269 MPRs. (XL file attached) and also
attaching
> individual ctc files.
>
> Now following commands for aggregate-stat.
> /oprn/model/wrf3/utils/met/METv3.0/bin/stat_analysis \
> -job aggregate_stat -line_type MPR -out_line_type CTC
-out_fcst_thresh
> ge0.1 -out_obs_thresh ge0.1 \
> -out
> ~/wrf3/utils/met/METv3.0/geeta_test/output-agst/agst-full24-ge0.1-
ctc.dat
> -v 4 \
> -lookin statfiles24 \
> #
> /oprn/model/wrf3/utils/met/METv3.0/bin/stat_analysis \
> -job aggregate_stat -line_type MPR -out_line_type CTS
-out_fcst_thresh
> ge0.1 -out_obs_thresh ge0.1 \
> -out
> ~/wrf3/utils/met/METv3.0/geeta_test/output-agst/agst-full24-ge0.1-
cts.dat
> -v 4 \
> -lookin statfiles24 \
> #
> #
> /oprn/model/wrf3/utils/met/METv3.0/bin/stat_analysis \
> -job aggregate_stat -line_type MPR -out_line_type CNT
-out_fcst_thresh
> ge0.1 -out_obs_thresh ge0.1 \
> -out
> ~/wrf3/utils/met/METv3.0/geeta_test/output-agst/agst-full24-ge0.1-
cnt.dat
> -v 4 \
> -lookin statfiles24 \
> #
>
> So I get three o/p files cts, CTC and cnt.
>
> Now when I open ctc files for thresholds 2.5 say,
> following is what I get.
> OB_LIST: -job aggregate_stat -line_type MPR -out_line_type CTC
> -out_fcst_thresh >=2.500 -out_obs_thresh >=2.500
> COL_NAME: TOTAL FY_OY FY_ON FN_OY FN_ON
> CTC: 33544 9185 7287 3801 13271
>
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> ~
> "agst-full24-ge2.5-ctc.dat" 4 lines, 200 characters
>
>
> So my question is HOW 33269 in changing to 33544????
>
> I am sending my stat files and ctc o/p files.
>
>
>
>
> pls look into that and let e what is hapening
>
> geeta
>
>
------------------------------------------------
Subject: error from the stat tool.
From: Geeta Geeta
Time: Wed Jul 09 22:37:47 2014
thanks John,
I figured out. Actually there was a problem in which the thresholds
are defined.
Take the example of *.stat files of 2.5 say.
when we define a (FC>=threshold, OBs>=threshold). when we define b
(F>=threshold, Obs,Threhold).
It does double counting. But for each threshold exclusively, the MPR's
are 33269 for 24hrs and 33291 for 48 hrs.
I did the exercise of calculation in XL.
But I want to confirm if FBIAS is the BIAS score that we all know.
I do not know the concept of bootstrapping. Can u tell me in which
case will that be applicaple????
geeta
> Subject: Re: [rt.rap.ucar.edu #68126] error from the stat tool.
> From: met_help at ucar.edu
> To: geeta124 at hotmail.com
> Date: Wed, 9 Jul 2014 12:33:51 -0600
>
> Geeta,
>
> I suspect that there is a problem in how you're passing data to the
> stat_analysis job. I took the .stat files you sent me and placed
them all
> in a directory named "point_stat". I counted up the MPR lines
present in
> the .stat files you sent:
>
> > grep MPR point_stat/*.stat | wc -l
> > 66538
>
> There are 66538 of them.
>
> Then I ran the following job:
>
> > stat_analysis -lookin point_stat -job aggregate_stat -line_type
MPR
> -out_line_type CTC -out_fcst_thresh \>=2.500 -out_obs_thresh
\>=2.500
> COL_NAME: TOTAL FY_OY FY_ON FN_OY FN_ON
> CTC: 66538 18214 14414 7528 26382
>
> It used all 66538 MPR lines.
>
> I noticed that some of your .stat files are named
"point_stat_ge0.1*" and
> some are named "point_stat_ge0.0*". So I reran my job using only
one type
> of those files:
>
> > stat_analysis -lookin point_stat/point_stat_ge0.1*.stat -job
> aggregate_stat -line_type MPR -out_line_type CTC -out_fcst_thresh
\>=2.500
> -out_obs_thresh \>=2.500
> COL_NAME: TOTAL FY_OY FY_ON FN_OY FN_ON
> CTC: 33269 9107 7207 3764 13191
>
> So I'm not sure how you're getting output with 33544 matched pairs.
But
> you can easily figure this out. Just rerun your job using the "-
dump_row"
> option. That will write all of the input lines used for that job to
an
> output file. Then you can inspect that file and figure out where
the extra
> lines are coming from. I suspect that you're accidentally passing
too much
> data to stat_analysis.
>
> Your jobs are telling it to look in the directory named
"statfiles24" (at
> least I assume that's a directory). It will use every file in that
> directory ending in .stat. Perhaps you've run other stat_analysis
jobs and
> written extra ".stat" files into that directory and stat_analysis is
> reading them back in. I'm just guessing here, but that would
explain it.
>
> Hope that helps get you going in the right direction.
>
> Thanks,
> John
>
>
> On Wed, Jul 9, 2014 at 1:41 AM, Geeta Geeta via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Wed Jul 09 01:41:27 2014: Request 68126 was acted upon.
> > Transaction: Ticket created by geeta124 at hotmail.com
> > Queue: met_help
> > Subject: error from the stat tool.
> > Owner: Nobody
> > Requestors: geeta124 at hotmail.com
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68126 >
> >
> >
> > Hi John.
> >
> > I am using point stat and then the stat analysis tool for
generating the
> > aggregate stats of my analysis. I am verifying the point rainfall
for
> > various Rf thresholds (gt0.0, ge0.1,ge2.5,ge7.6,ge35.6,ge64.5)
> >
> > Totally I have generated 33269 MPRs. (XL file attached) and also
attaching
> > individual ctc files.
> >
> > Now following commands for aggregate-stat.
> > /oprn/model/wrf3/utils/met/METv3.0/bin/stat_analysis \
> > -job aggregate_stat -line_type MPR -out_line_type CTC
-out_fcst_thresh
> > ge0.1 -out_obs_thresh ge0.1 \
> > -out
> > ~/wrf3/utils/met/METv3.0/geeta_test/output-agst/agst-full24-ge0.1-
ctc.dat
> > -v 4 \
> > -lookin statfiles24 \
> > #
> > /oprn/model/wrf3/utils/met/METv3.0/bin/stat_analysis \
> > -job aggregate_stat -line_type MPR -out_line_type CTS
-out_fcst_thresh
> > ge0.1 -out_obs_thresh ge0.1 \
> > -out
> > ~/wrf3/utils/met/METv3.0/geeta_test/output-agst/agst-full24-ge0.1-
cts.dat
> > -v 4 \
> > -lookin statfiles24 \
> > #
> > #
> > /oprn/model/wrf3/utils/met/METv3.0/bin/stat_analysis \
> > -job aggregate_stat -line_type MPR -out_line_type CNT
-out_fcst_thresh
> > ge0.1 -out_obs_thresh ge0.1 \
> > -out
> > ~/wrf3/utils/met/METv3.0/geeta_test/output-agst/agst-full24-ge0.1-
cnt.dat
> > -v 4 \
> > -lookin statfiles24 \
> > #
> >
> > So I get three o/p files cts, CTC and cnt.
> >
> > Now when I open ctc files for thresholds 2.5 say,
> > following is what I get.
> > OB_LIST: -job aggregate_stat -line_type MPR -out_line_type
CTC
> > -out_fcst_thresh >=2.500 -out_obs_thresh >=2.500
> > COL_NAME: TOTAL FY_OY FY_ON FN_OY FN_ON
> > CTC: 33544 9185 7287 3801 13271
> >
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > ~
> > "agst-full24-ge2.5-ctc.dat" 4 lines, 200 characters
> >
> >
> > So my question is HOW 33269 in changing to 33544????
> >
> > I am sending my stat files and ctc o/p files.
> >
> >
> >
> >
> > pls look into that and let e what is hapening
> >
> > geeta
> >
> >
>
------------------------------------------------
Subject: error from the stat tool.
From: John Halley Gotway
Time: Thu Jul 10 09:44:02 2014
Geeta,
Great, I'm glad you were able to figure it out.
Regarding "bias" in MET, it can be a bit confusing. The MET output
contains output for 3 different ways of measuring bias.
Mostly commonly when people say "bias", they're referring to the
difference
between the average forecast value and the average observation value.
We
call that the mean error and it's contained in the ME column of the
continuous statistics line type (CNT). A perfect score for the mean
error
is 0.
Another form of bias is the "multiplicative bias", which is average
forecast value divided by the average observation value. That's
contained
in the MBIAS column of the continuous statistics line type (CNT). A
perfect score for the multiplicative bias is 1.
While the first two biases are for continuous statistics, the third is
a
categorical statistic. So you've chosen a threshold to define events
in
the forecast and observation fields. The frequency bias in the FBIAS
column of the categorical statistics line type (CTS) is the number of
forecast grid points at which the event occurred divided by the number
of
observation grid points at which the event occurred. A perfect score
is
1. Scores < 1 means the forecast under-predicts the event while > 1
means
the forecast over-predicts it.
Equations for all of the stats generated by MET can be found in
appendix C
of the MET User's Guide.
Hope that helps.
Thanks,
John
On Wed, Jul 9, 2014 at 10:37 PM, Geeta Geeta via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68126 >
>
> thanks John,
> I figured out. Actually there was a problem in which the thresholds
are
> defined.
> Take the example of *.stat files of 2.5 say.
>
> when we define a (FC>=threshold, OBs>=threshold). when we define b
> (F>=threshold, Obs,Threhold).
>
> It does double counting. But for each threshold exclusively, the
MPR's are
> 33269 for 24hrs and 33291 for 48 hrs.
>
> I did the exercise of calculation in XL.
>
> But I want to confirm if FBIAS is the BIAS score that we all know.
> I do not know the concept of bootstrapping. Can u tell me in which
case
> will that be applicaple????
>
> geeta
>
> > Subject: Re: [rt.rap.ucar.edu #68126] error from the stat tool.
> > From: met_help at ucar.edu
> > To: geeta124 at hotmail.com
> > Date: Wed, 9 Jul 2014 12:33:51 -0600
> >
> > Geeta,
> >
> > I suspect that there is a problem in how you're passing data to
the
> > stat_analysis job. I took the .stat files you sent me and placed
them
> all
> > in a directory named "point_stat". I counted up the MPR lines
present in
> > the .stat files you sent:
> >
> > > grep MPR point_stat/*.stat | wc -l
> > > 66538
> >
> > There are 66538 of them.
> >
> > Then I ran the following job:
> >
> > > stat_analysis -lookin point_stat -job aggregate_stat -line_type
MPR
> > -out_line_type CTC -out_fcst_thresh \>=2.500 -out_obs_thresh
\>=2.500
> > COL_NAME: TOTAL FY_OY FY_ON FN_OY FN_ON
> > CTC: 66538 18214 14414 7528 26382
> >
> > It used all 66538 MPR lines.
> >
> > I noticed that some of your .stat files are named
"point_stat_ge0.1*" and
> > some are named "point_stat_ge0.0*". So I reran my job using only
one
> type
> > of those files:
> >
> > > stat_analysis -lookin point_stat/point_stat_ge0.1*.stat -job
> > aggregate_stat -line_type MPR -out_line_type CTC -out_fcst_thresh
> \>=2.500
> > -out_obs_thresh \>=2.500
> > COL_NAME: TOTAL FY_OY FY_ON FN_OY FN_ON
> > CTC: 33269 9107 7207 3764 13191
> >
> > So I'm not sure how you're getting output with 33544 matched
pairs. But
> > you can easily figure this out. Just rerun your job using the
> "-dump_row"
> > option. That will write all of the input lines used for that job
to an
> > output file. Then you can inspect that file and figure out where
the
> extra
> > lines are coming from. I suspect that you're accidentally passing
too
> much
> > data to stat_analysis.
> >
> > Your jobs are telling it to look in the directory named
"statfiles24" (at
> > least I assume that's a directory). It will use every file in
that
> > directory ending in .stat. Perhaps you've run other stat_analysis
jobs
> and
> > written extra ".stat" files into that directory and stat_analysis
is
> > reading them back in. I'm just guessing here, but that would
explain it.
> >
> > Hope that helps get you going in the right direction.
> >
> > Thanks,
> > John
> >
> >
> > On Wed, Jul 9, 2014 at 1:41 AM, Geeta Geeta via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > Wed Jul 09 01:41:27 2014: Request 68126 was acted upon.
> > > Transaction: Ticket created by geeta124 at hotmail.com
> > > Queue: met_help
> > > Subject: error from the stat tool.
> > > Owner: Nobody
> > > Requestors: geeta124 at hotmail.com
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68126
> >
> > >
> > >
> > > Hi John.
> > >
> > > I am using point stat and then the stat analysis tool for
generating
> the
> > > aggregate stats of my analysis. I am verifying the point
rainfall for
> > > various Rf thresholds (gt0.0, ge0.1,ge2.5,ge7.6,ge35.6,ge64.5)
> > >
> > > Totally I have generated 33269 MPRs. (XL file attached) and also
> attaching
> > > individual ctc files.
> > >
> > > Now following commands for aggregate-stat.
> > > /oprn/model/wrf3/utils/met/METv3.0/bin/stat_analysis \
> > > -job aggregate_stat -line_type MPR -out_line_type CTC
> -out_fcst_thresh
> > > ge0.1 -out_obs_thresh ge0.1 \
> > > -out
> > >
> ~/wrf3/utils/met/METv3.0/geeta_test/output-agst/agst-full24-ge0.1-
ctc.dat
> > > -v 4 \
> > > -lookin statfiles24 \
> > > #
> > > /oprn/model/wrf3/utils/met/METv3.0/bin/stat_analysis \
> > > -job aggregate_stat -line_type MPR -out_line_type CTS
> -out_fcst_thresh
> > > ge0.1 -out_obs_thresh ge0.1 \
> > > -out
> > >
> ~/wrf3/utils/met/METv3.0/geeta_test/output-agst/agst-full24-ge0.1-
cts.dat
> > > -v 4 \
> > > -lookin statfiles24 \
> > > #
> > > #
> > > /oprn/model/wrf3/utils/met/METv3.0/bin/stat_analysis \
> > > -job aggregate_stat -line_type MPR -out_line_type CNT
> -out_fcst_thresh
> > > ge0.1 -out_obs_thresh ge0.1 \
> > > -out
> > >
> ~/wrf3/utils/met/METv3.0/geeta_test/output-agst/agst-full24-ge0.1-
cnt.dat
> > > -v 4 \
> > > -lookin statfiles24 \
> > > #
> > >
> > > So I get three o/p files cts, CTC and cnt.
> > >
> > > Now when I open ctc files for thresholds 2.5 say,
> > > following is what I get.
> > > OB_LIST: -job aggregate_stat -line_type MPR -out_line_type
CTC
> > > -out_fcst_thresh >=2.500 -out_obs_thresh >=2.500
> > > COL_NAME: TOTAL FY_OY FY_ON FN_OY FN_ON
> > > CTC: 33544 9185 7287 3801 13271
> > >
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > ~
> > > "agst-full24-ge2.5-ctc.dat" 4 lines, 200 characters
> > >
> > >
> > > So my question is HOW 33269 in changing to 33544????
> > >
> > > I am sending my stat files and ctc o/p files.
> > >
> > >
> > >
> > >
> > > pls look into that and let e what is hapening
> > >
> > > geeta
> > >
> > >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list