[Met_help] [rt.rap.ucar.edu #78218] History for multiple options cat_thresh
John Halley Gotway via RT
met_help at ucar.edu
Tue Oct 18 09:43:36 MDT 2016
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Dear,
how can I use multiple thresholds in cat_thresh? In MET users guide have
the example ">=5&&<=10". But when i use [ =>5&&<=10, >10&&<=15] I get the
error "when verifying using multi-category contingency tables, the
thresholds must be monotonically increasing and be of the same inequality
type (lt, le, gt, or ge)."
Thanks
Bianca Maske
--
Bianca Buss Maske
Bacharel em Meteorologia - UFPEL
Mestre em Meteorologia - INPE
Bolsista do projeto Climasul- SIMEPAR
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: multiple options cat_thresh
From: John Halley Gotway
Time: Mon Oct 10 10:12:35 2016
Hello Bianca,
I see that you're having some difficulty defining categorical
thresholds
for use in MET. If I had a clear understanding of what exactly you're
trying to accomplish, I could more easily help you set up the
configuration
file.
First, let me point out that you have a typo in the example sent,
"=>5"
should really be ">=5".
The next question is whether you're trying to compute multi-category
contingency table counts and statistics (i.e. the MCTC and MCTS line
types).
If not, then please disable their output by setting MCTC = NONE; and
MCTS =
NONE; in the configuration file. Once you've done that, you can
define any
sort of categorical thresholds you'd like.
If you do want MCTC and MCTS output, then your thresholds need to
follow
the format indicated by that error message... be monotonically
increasing
and all of the same type. For example, the following would work:
cat_thresh = [ >5, >10, >15 ];
When computing MCTC and MCTS output, MET uses N thresholds to define
N+1
categories. In the above example, those 3 thresholds define 4
categories
(less then 5, between 5 and 10, between 10 and 15, and greater than
15).
That results in a 4x4 contingency table.
It is only when MCTC or MCTS output are requested in the config file
that
MET enforces these threshold conventions.
Let's suppose you have MCTC and MCTS turned off and CTC turned on.
And you
choose the following:
cat_thresh = [ >=5&&<=10, >=10&&<=15 ];
>From MET's point of view, this defines two thresholds. For the first,
any
values between 5 and 10 are considered an "event" while values outside
that
range are "non-events". Similar for the second, but for values
between 10
and 15. These thresholds are processed totally independently. So
you'll
get a 2x2 contingency table (CTC) output line for each, where the
event is
falling within the defined range. Values above and below the range
are
non-events.
Perhaps that's a reasonable thing to do, but more commonly people use
a
simple threshold instead (i.e. >5). But if this type of thresholding
makes
sense for you analysis, feel free to use it. I can tell you that this
complex thresholding is more commonly used for the "cnt_thresh"
setting.
For example, setting cnt_thresh = [ >=5&&<=10 ]; causes MET to only
use
values falling within that range in the computation of continuous
statistics. And this is called conditional verification... what's the
RMSE
for forecast values falling between 5 and 10.
Hope that helps clarify what's going on.
Thanks,
John Halley Gotway
On Mon, Oct 10, 2016 at 8:02 AM, Bianca Maske via RT
<met_help at ucar.edu>
wrote:
>
> Mon Oct 10 08:02:32 2016: Request 78218 was acted upon.
> Transaction: Ticket created by bianca.maske at gmail.com
> Queue: met_help
> Subject: multiple options cat_thresh
> Owner: Nobody
> Requestors: bianca.maske at gmail.com
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78218 >
>
>
> Dear,
>
> how can I use multiple thresholds in cat_thresh? In MET users guide
have
> the example ">=5&&<=10". But when i use [ =>5&&<=10, >10&&<=15] I
get the
> error "when verifying using multi-category contingency tables, the
> thresholds must be monotonically increasing and be of the same
inequality
> type (lt, le, gt, or ge)."
>
> Thanks
>
> Bianca Maske
>
>
> --
> Bianca Buss Maske
> Bacharel em Meteorologia - UFPEL
> Mestre em Meteorologia - INPE
> Bolsista do projeto Climasul- SIMEPAR
>
>
------------------------------------------------
Subject: multiple options cat_thresh
From: Bianca Maske
Time: Mon Oct 10 13:54:17 2016
Thank you for your clarifications.
If I put off MCTC and MCTS I can use cat_thresh = [ >=5&&<=10,
>=10&&<=15
], but if I put off CTC and CTS and use cnt_thresh = [ >=5&&<=10,
>=10&&<=15 ] and cat_thresh = [ ] I don't have results.
The difference between MCTS and CTS is that MCTS get a result for
all threshold and CTS for each threshold? For example, in MCTS I have
one
value for Accuracy and in CTS I have one value for Accuracy for each
threshold.
Thank You.
2016-10-10 13:12 GMT-03:00 John Halley Gotway via RT
<met_help at ucar.edu>:
> Hello Bianca,
>
> I see that you're having some difficulty defining categorical
thresholds
> for use in MET. If I had a clear understanding of what exactly
you're
> trying to accomplish, I could more easily help you set up the
configuration
> file.
>
> First, let me point out that you have a typo in the example sent,
"=>5"
> should really be ">=5".
>
> The next question is whether you're trying to compute multi-category
> contingency table counts and statistics (i.e. the MCTC and MCTS line
> types).
>
> If not, then please disable their output by setting MCTC = NONE; and
MCTS =
> NONE; in the configuration file. Once you've done that, you can
define any
> sort of categorical thresholds you'd like.
>
> If you do want MCTC and MCTS output, then your thresholds need to
follow
> the format indicated by that error message... be monotonically
increasing
> and all of the same type. For example, the following would work:
> cat_thresh = [ >5, >10, >15 ];
>
> When computing MCTC and MCTS output, MET uses N thresholds to define
N+1
> categories. In the above example, those 3 thresholds define 4
categories
> (less then 5, between 5 and 10, between 10 and 15, and greater than
15).
> That results in a 4x4 contingency table.
>
> It is only when MCTC or MCTS output are requested in the config file
that
> MET enforces these threshold conventions.
>
> Let's suppose you have MCTC and MCTS turned off and CTC turned on.
And you
> choose the following:
> cat_thresh = [ >=5&&<=10, >=10&&<=15 ];
>
> From MET's point of view, this defines two thresholds. For the
first, any
> values between 5 and 10 are considered an "event" while values
outside that
> range are "non-events". Similar for the second, but for values
between 10
> and 15. These thresholds are processed totally independently. So
you'll
> get a 2x2 contingency table (CTC) output line for each, where the
event is
> falling within the defined range. Values above and below the range
are
> non-events.
>
> Perhaps that's a reasonable thing to do, but more commonly people
use a
> simple threshold instead (i.e. >5). But if this type of
thresholding makes
> sense for you analysis, feel free to use it. I can tell you that
this
> complex thresholding is more commonly used for the "cnt_thresh"
setting.
> For example, setting cnt_thresh = [ >=5&&<=10 ]; causes MET to only
use
> values falling within that range in the computation of continuous
> statistics. And this is called conditional verification... what's
the RMSE
> for forecast values falling between 5 and 10.
>
> Hope that helps clarify what's going on.
>
> Thanks,
> John Halley Gotway
>
>
> On Mon, Oct 10, 2016 at 8:02 AM, Bianca Maske via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Mon Oct 10 08:02:32 2016: Request 78218 was acted upon.
> > Transaction: Ticket created by bianca.maske at gmail.com
> > Queue: met_help
> > Subject: multiple options cat_thresh
> > Owner: Nobody
> > Requestors: bianca.maske at gmail.com
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78218 >
> >
> >
> > Dear,
> >
> > how can I use multiple thresholds in cat_thresh? In MET users
guide have
> > the example ">=5&&<=10". But when i use [ =>5&&<=10, >10&&<=15]
I get
> the
> > error "when verifying using multi-category contingency tables, the
> > thresholds must be monotonically increasing and be of the same
inequality
> > type (lt, le, gt, or ge)."
> >
> > Thanks
> >
> > Bianca Maske
> >
> >
> > --
> > Bianca Buss Maske
> > Bacharel em Meteorologia - UFPEL
> > Mestre em Meteorologia - INPE
> > Bolsista do projeto Climasul- SIMEPAR
> >
> >
>
>
--
Bianca Buss Maske
Bacharel em Meteorologia - UFPEL
Mestre em Meteorologia - INPE
Bolsista do projeto Climasul- SIMEPAR
------------------------------------------------
Subject: multiple options cat_thresh
From: John Halley Gotway
Time: Mon Oct 10 14:54:39 2016
Hello Bianca,
The "cat_thresh" config file settings affect the 2x2 contingency table
counts (CTC) and statistics (CTS) output line types, as well as the
NxN
multi-category contingency table counts (MCTC) and statistics (MCTS)
output
line types.
The "cnt_thresh" config file settings affect the continuous statistics
(CNT) and partial sums (SL1L2) output line types.
The output of each of these line types is controlled by the
"output_flag"
config file setting.
As for the difference between MCTS and CTS... yes, MCTS includes
statistics
defined using all of your thresholds. The CTS line type in only one
threshold at a time. Unfortunately, there are very few statistics (4
I
think) included in the MCTS output. There are many more statistics
defined
for 2x2 contingency tables. Yes, accuracy shows up in both the CTS
and
MCTS line types. In both, it's simply the percent correct (i.e. the
percent of cases falling of the diagonal of either a 2x2 or NxN
contingency
table).
Users often apply several thresholds to produce several CTS output
lines
and then include them all on one plot to compare how a model performs
at
higher threshold versus lower thresholds. For example, your model may
predict 1/2 precip very well but do very poorly for larger
accumulations.
Hope that helps.
Thanks,
John
On Mon, Oct 10, 2016 at 1:54 PM, Bianca Maske via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78218 >
>
> Thank you for your clarifications.
>
> If I put off MCTC and MCTS I can use cat_thresh = [ >=5&&<=10,
>=10&&<=15
> ], but if I put off CTC and CTS and use cnt_thresh = [ >=5&&<=10,
> >=10&&<=15 ] and cat_thresh = [ ] I don't have results.
>
> The difference between MCTS and CTS is that MCTS get a result for
> all threshold and CTS for each threshold? For example, in MCTS I
have one
> value for Accuracy and in CTS I have one value for Accuracy for each
> threshold.
>
> Thank You.
>
> 2016-10-10 13:12 GMT-03:00 John Halley Gotway via RT
<met_help at ucar.edu>:
>
> > Hello Bianca,
> >
> > I see that you're having some difficulty defining categorical
thresholds
> > for use in MET. If I had a clear understanding of what exactly
you're
> > trying to accomplish, I could more easily help you set up the
> configuration
> > file.
> >
> > First, let me point out that you have a typo in the example sent,
"=>5"
> > should really be ">=5".
> >
> > The next question is whether you're trying to compute multi-
category
> > contingency table counts and statistics (i.e. the MCTC and MCTS
line
> > types).
> >
> > If not, then please disable their output by setting MCTC = NONE;
and
> MCTS =
> > NONE; in the configuration file. Once you've done that, you can
define
> any
> > sort of categorical thresholds you'd like.
> >
> > If you do want MCTC and MCTS output, then your thresholds need to
follow
> > the format indicated by that error message... be monotonically
increasing
> > and all of the same type. For example, the following would work:
> > cat_thresh = [ >5, >10, >15 ];
> >
> > When computing MCTC and MCTS output, MET uses N thresholds to
define N+1
> > categories. In the above example, those 3 thresholds define 4
categories
> > (less then 5, between 5 and 10, between 10 and 15, and greater
than 15).
> > That results in a 4x4 contingency table.
> >
> > It is only when MCTC or MCTS output are requested in the config
file that
> > MET enforces these threshold conventions.
> >
> > Let's suppose you have MCTC and MCTS turned off and CTC turned on.
And
> you
> > choose the following:
> > cat_thresh = [ >=5&&<=10, >=10&&<=15 ];
> >
> > From MET's point of view, this defines two thresholds. For the
first,
> any
> > values between 5 and 10 are considered an "event" while values
outside
> that
> > range are "non-events". Similar for the second, but for values
between
> 10
> > and 15. These thresholds are processed totally independently. So
you'll
> > get a 2x2 contingency table (CTC) output line for each, where the
event
> is
> > falling within the defined range. Values above and below the
range are
> > non-events.
> >
> > Perhaps that's a reasonable thing to do, but more commonly people
use a
> > simple threshold instead (i.e. >5). But if this type of
thresholding
> makes
> > sense for you analysis, feel free to use it. I can tell you that
this
> > complex thresholding is more commonly used for the "cnt_thresh"
setting.
> > For example, setting cnt_thresh = [ >=5&&<=10 ]; causes MET to
only use
> > values falling within that range in the computation of continuous
> > statistics. And this is called conditional verification... what's
the
> RMSE
> > for forecast values falling between 5 and 10.
> >
> > Hope that helps clarify what's going on.
> >
> > Thanks,
> > John Halley Gotway
> >
> >
> > On Mon, Oct 10, 2016 at 8:02 AM, Bianca Maske via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > Mon Oct 10 08:02:32 2016: Request 78218 was acted upon.
> > > Transaction: Ticket created by bianca.maske at gmail.com
> > > Queue: met_help
> > > Subject: multiple options cat_thresh
> > > Owner: Nobody
> > > Requestors: bianca.maske at gmail.com
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=78218
> >
> > >
> > >
> > > Dear,
> > >
> > > how can I use multiple thresholds in cat_thresh? In MET users
guide
> have
> > > the example ">=5&&<=10". But when i use [ =>5&&<=10,
>10&&<=15] I get
> > the
> > > error "when verifying using multi-category contingency tables,
the
> > > thresholds must be monotonically increasing and be of the same
> inequality
> > > type (lt, le, gt, or ge)."
> > >
> > > Thanks
> > >
> > > Bianca Maske
> > >
> > >
> > > --
> > > Bianca Buss Maske
> > > Bacharel em Meteorologia - UFPEL
> > > Mestre em Meteorologia - INPE
> > > Bolsista do projeto Climasul- SIMEPAR
> > >
> > >
> >
> >
>
>
> --
> Bianca Buss Maske
> Bacharel em Meteorologia - UFPEL
> Mestre em Meteorologia - INPE
> Bolsista do projeto Climasul- SIMEPAR
>
>
------------------------------------------------
More information about the Met_help
mailing list