[Met_help] [rt.rap.ucar.edu #96942] History for grid_stat output
John Halley Gotway via RT
met_help at ucar.edu
Tue Oct 13 12:40:09 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello,
There are two things I want to confirm:
For attachment 1: it is part of the grid_stat output:
Why "Total number of matched pairs:" (column after LINE_TYPE) are the same
no matter which thresh_old is used? So this number is not related to the
case when which specific threshold is used? It is the "TOTAL", correct?
For attachment 2:
Does the number on the second Y-axis refer to the number of matching pairs
that meet the standard? Those number is much higher than what is shown in
the log file
Thank you.
Binyu
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: grid_stat output
From: John Halley Gotway
Time: Mon Oct 05 11:18:33 2020
Binyu,
I suspect you are getting confused by a categorical threshold and a
filtering threshold. For this reason, I retrospect I wish we'd chosen
a
different name for these two operations instead of calling both of
them
"thresholds".
You noticed that in the probabilistic output line types, the TOTAL
column
has a constant value of 6424. The TOTAL column is the number of
matched
pairs that was included in the analysis. All of the probabilistic
verification in MET is based on an Nx2 contingency table... and the
TOTAL
column is the same as the sum of all of the cells of that Nx2 table.
The
FCST_THRESH column indicates which probability bins were used to
define
that Nx2 table and the OBS_THRESH column indicates the verifying
threshold.
But in all cases, the same set of 6424 matched pairs were used in the
analysis.
What part of this process is causing confusion?
Thanks,
John
On Fri, Oct 2, 2020 at 2:09 PM binyu.wang at noaa.gov via RT
<met_help at ucar.edu>
wrote:
>
> Fri Oct 02 14:08:56 2020: Request 96942 was acted upon.
> Transaction: Ticket created by binyu.wang at noaa.gov
> Queue: met_help
> Subject: grid_stat output
> Owner: Nobody
> Requestors: binyu.wang at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96942 >
>
>
> Hello,
>
> There are two things I want to confirm:
> For attachment 1: it is part of the grid_stat output:
> Why "Total number of matched pairs:" (column after LINE_TYPE) are
the same
> no matter which thresh_old is used? So this number is not related
to the
> case when which specific threshold is used? It is the "TOTAL",
correct?
>
> For attachment 2:
> Does the number on the second Y-axis refer to the number of matching
pairs
> that meet the standard? Those number is much higher than what is
shown in
> the log file
> Thank you.
>
> Binyu
>
>
------------------------------------------------
Subject: grid_stat output
From: binyu.wang at noaa.gov
Time: Mon Oct 05 12:10:59 2020
To me, "TOTAL" only refers to the pairs that meet the standard. So if
the
threshold is different, the matched pairs that meet the standard
should be
different.
e.g OBS_THRESH > 0.1 should have more pairs (TOTAL) compared to
OBS_THRESH
> 1, unless there is no any pair when the OBS is less than 1 but
larger
than 0.1.
Binyu
On Mon, Oct 5, 2020 at 1:18 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> I suspect you are getting confused by a categorical threshold and a
> filtering threshold. For this reason, I retrospect I wish we'd
chosen a
> different name for these two operations instead of calling both of
them
> "thresholds".
>
> You noticed that in the probabilistic output line types, the TOTAL
column
> has a constant value of 6424. The TOTAL column is the number of
matched
> pairs that was included in the analysis. All of the probabilistic
> verification in MET is based on an Nx2 contingency table... and the
TOTAL
> column is the same as the sum of all of the cells of that Nx2 table.
The
> FCST_THRESH column indicates which probability bins were used to
define
> that Nx2 table and the OBS_THRESH column indicates the verifying
threshold.
>
> But in all cases, the same set of 6424 matched pairs were used in
the
> analysis.
>
> What part of this process is causing confusion?
>
> Thanks,
> John
>
> On Fri, Oct 2, 2020 at 2:09 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > Fri Oct 02 14:08:56 2020: Request 96942 was acted upon.
> > Transaction: Ticket created by binyu.wang at noaa.gov
> > Queue: met_help
> > Subject: grid_stat output
> > Owner: Nobody
> > Requestors: binyu.wang at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96942 >
> >
> >
> > Hello,
> >
> > There are two things I want to confirm:
> > For attachment 1: it is part of the grid_stat output:
> > Why "Total number of matched pairs:" (column after LINE_TYPE) are
the
> same
> > no matter which thresh_old is used? So this number is not related
to the
> > case when which specific threshold is used? It is the "TOTAL",
correct?
> >
> > For attachment 2:
> > Does the number on the second Y-axis refer to the number of
matching
> pairs
> > that meet the standard? Those number is much higher than what is
shown in
> > the log file
> > Thank you.
> >
> > Binyu
> >
> >
>
>
------------------------------------------------
Subject: grid_stat output
From: John Halley Gotway
Time: Mon Oct 05 13:57:26 2020
Binyu,
What you are requesting is called "conditional verification" where you
define some filtering threshold for the values and only include pairs
which
meet that condition in the verification.
In Point-Stat and Grid-Stat, conditional verification is provided
for...
- continuous statistics using the cnt_thresh and cnt_logic config file
options
- wind vector statistics using the wind_thresh and wind_logic config
file
options
However there is currently no mechanism by which to do conditional
verification for...
- categorical statistics (in the CTC, CTS, MCTC, and MCTS line types)
- probabilistic statistics (in the PCT, PSTD, PJC, PRC, and ECLV line
types)
The "cat_thresh" option defines the categories used for categorical
and
probabilistic verification. It does not define filtering criteria to
determine which points should be included.
You can of course apply the censor_thresh and censor_val options in
whatever way you'd like. That could be a way of achieving the goal you
described, but conditional verification of categorical stats is not
directly supported in these tools.
John
On Mon, Oct 5, 2020 at 12:11 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96942 >
>
> To me, "TOTAL" only refers to the pairs that meet the standard. So
if the
> threshold is different, the matched pairs that meet the standard
should be
> different.
> e.g OBS_THRESH > 0.1 should have more pairs (TOTAL) compared to
OBS_THRESH
> > 1, unless there is no any pair when the OBS is less than 1 but
larger
> than 0.1.
>
> Binyu
>
> On Mon, Oct 5, 2020 at 1:18 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > I suspect you are getting confused by a categorical threshold and
a
> > filtering threshold. For this reason, I retrospect I wish we'd
chosen a
> > different name for these two operations instead of calling both of
them
> > "thresholds".
> >
> > You noticed that in the probabilistic output line types, the TOTAL
column
> > has a constant value of 6424. The TOTAL column is the number of
matched
> > pairs that was included in the analysis. All of the probabilistic
> > verification in MET is based on an Nx2 contingency table... and
the TOTAL
> > column is the same as the sum of all of the cells of that Nx2
table. The
> > FCST_THRESH column indicates which probability bins were used to
define
> > that Nx2 table and the OBS_THRESH column indicates the verifying
> threshold.
> >
> > But in all cases, the same set of 6424 matched pairs were used in
the
> > analysis.
> >
> > What part of this process is causing confusion?
> >
> > Thanks,
> > John
> >
> > On Fri, Oct 2, 2020 at 2:09 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > >
> > > Fri Oct 02 14:08:56 2020: Request 96942 was acted upon.
> > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > Queue: met_help
> > > Subject: grid_stat output
> > > Owner: Nobody
> > > Requestors: binyu.wang at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96942
> >
> > >
> > >
> > > Hello,
> > >
> > > There are two things I want to confirm:
> > > For attachment 1: it is part of the grid_stat output:
> > > Why "Total number of matched pairs:" (column after LINE_TYPE)
are the
> > same
> > > no matter which thresh_old is used? So this number is not
related to
> the
> > > case when which specific threshold is used? It is the "TOTAL",
correct?
> > >
> > > For attachment 2:
> > > Does the number on the second Y-axis refer to the number of
matching
> > pairs
> > > that meet the standard? Those number is much higher than what is
shown
> in
> > > the log file
> > > Thank you.
> > >
> > > Binyu
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: grid_stat output
From: binyu.wang at noaa.gov
Time: Mon Oct 05 14:51:16 2020
Thank you for giving so much details. So "cat_thresh = [>=1]", "1" is
not a
filter, all the events (even value less than 1) are used in the
verification, this is why TOTAL is the same for all columns in the
attachment?
Binyu
On Mon, Oct 5, 2020 at 3:57 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> What you are requesting is called "conditional verification" where
you
> define some filtering threshold for the values and only include
pairs which
> meet that condition in the verification.
>
> In Point-Stat and Grid-Stat, conditional verification is provided
for...
> - continuous statistics using the cnt_thresh and cnt_logic config
file
> options
> - wind vector statistics using the wind_thresh and wind_logic config
file
> options
>
> However there is currently no mechanism by which to do conditional
> verification for...
> - categorical statistics (in the CTC, CTS, MCTC, and MCTS line
types)
> - probabilistic statistics (in the PCT, PSTD, PJC, PRC, and ECLV
line
> types)
>
> The "cat_thresh" option defines the categories used for categorical
and
> probabilistic verification. It does not define filtering criteria to
> determine which points should be included.
>
> You can of course apply the censor_thresh and censor_val options in
> whatever way you'd like. That could be a way of achieving the goal
you
> described, but conditional verification of categorical stats is not
> directly supported in these tools.
>
> John
>
> On Mon, Oct 5, 2020 at 12:11 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96942 >
> >
> > To me, "TOTAL" only refers to the pairs that meet the standard. So
if
> the
> > threshold is different, the matched pairs that meet the standard
should
> be
> > different.
> > e.g OBS_THRESH > 0.1 should have more pairs (TOTAL) compared to
> OBS_THRESH
> > > 1, unless there is no any pair when the OBS is less than 1 but
larger
> > than 0.1.
> >
> > Binyu
> >
> > On Mon, Oct 5, 2020 at 1:18 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Binyu,
> > >
> > > I suspect you are getting confused by a categorical threshold
and a
> > > filtering threshold. For this reason, I retrospect I wish we'd
chosen a
> > > different name for these two operations instead of calling both
of them
> > > "thresholds".
> > >
> > > You noticed that in the probabilistic output line types, the
TOTAL
> column
> > > has a constant value of 6424. The TOTAL column is the number of
matched
> > > pairs that was included in the analysis. All of the
probabilistic
> > > verification in MET is based on an Nx2 contingency table... and
the
> TOTAL
> > > column is the same as the sum of all of the cells of that Nx2
table.
> The
> > > FCST_THRESH column indicates which probability bins were used to
define
> > > that Nx2 table and the OBS_THRESH column indicates the verifying
> > threshold.
> > >
> > > But in all cases, the same set of 6424 matched pairs were used
in the
> > > analysis.
> > >
> > > What part of this process is causing confusion?
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Oct 2, 2020 at 2:09 PM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > Fri Oct 02 14:08:56 2020: Request 96942 was acted upon.
> > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > Queue: met_help
> > > > Subject: grid_stat output
> > > > Owner: Nobody
> > > > Requestors: binyu.wang at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96942
> > >
> > > >
> > > >
> > > > Hello,
> > > >
> > > > There are two things I want to confirm:
> > > > For attachment 1: it is part of the grid_stat output:
> > > > Why "Total number of matched pairs:" (column after LINE_TYPE)
are the
> > > same
> > > > no matter which thresh_old is used? So this number is not
related to
> > the
> > > > case when which specific threshold is used? It is the "TOTAL",
> correct?
> > > >
> > > > For attachment 2:
> > > > Does the number on the second Y-axis refer to the number of
matching
> > > pairs
> > > > that meet the standard? Those number is much higher than what
is
> shown
> > in
> > > > the log file
> > > > Thank you.
> > > >
> > > > Binyu
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: grid_stat output
From: John Halley Gotway
Time: Mon Oct 05 15:34:45 2020
Binyu,
Yes, that is correct. Probability forecasts are verified in MET using
an
Nx2 contingency table.
fcst_thresh defines the N probability bins.
obs_thresh is a single threshold which defines the event that was
being
forecast.
For example, if verifying a forecast of Probability of Precipitation,
you'd
likely set:
obs_thresh = [ >0 ];
So anytime we observed precip greater than 0, the event occurred.
John
On Mon, Oct 5, 2020 at 2:51 PM binyu.wang at noaa.gov via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96942 >
>
> Thank you for giving so much details. So "cat_thresh = [>=1]", "1"
is not a
> filter, all the events (even value less than 1) are used in the
> verification, this is why TOTAL is the same for all columns in the
> attachment?
>
> Binyu
>
>
>
> On Mon, Oct 5, 2020 at 3:57 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > What you are requesting is called "conditional verification" where
you
> > define some filtering threshold for the values and only include
pairs
> which
> > meet that condition in the verification.
> >
> > In Point-Stat and Grid-Stat, conditional verification is provided
for...
> > - continuous statistics using the cnt_thresh and cnt_logic config
file
> > options
> > - wind vector statistics using the wind_thresh and wind_logic
config file
> > options
> >
> > However there is currently no mechanism by which to do conditional
> > verification for...
> > - categorical statistics (in the CTC, CTS, MCTC, and MCTS line
types)
> > - probabilistic statistics (in the PCT, PSTD, PJC, PRC, and ECLV
line
> > types)
> >
> > The "cat_thresh" option defines the categories used for
categorical and
> > probabilistic verification. It does not define filtering criteria
to
> > determine which points should be included.
> >
> > You can of course apply the censor_thresh and censor_val options
in
> > whatever way you'd like. That could be a way of achieving the goal
you
> > described, but conditional verification of categorical stats is
not
> > directly supported in these tools.
> >
> > John
> >
> > On Mon, Oct 5, 2020 at 12:11 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96942 >
> > >
> > > To me, "TOTAL" only refers to the pairs that meet the standard.
So if
> > the
> > > threshold is different, the matched pairs that meet the standard
should
> > be
> > > different.
> > > e.g OBS_THRESH > 0.1 should have more pairs (TOTAL) compared to
> > OBS_THRESH
> > > > 1, unless there is no any pair when the OBS is less than 1 but
larger
> > > than 0.1.
> > >
> > > Binyu
> > >
> > > On Mon, Oct 5, 2020 at 1:18 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Binyu,
> > > >
> > > > I suspect you are getting confused by a categorical threshold
and a
> > > > filtering threshold. For this reason, I retrospect I wish we'd
> chosen a
> > > > different name for these two operations instead of calling
both of
> them
> > > > "thresholds".
> > > >
> > > > You noticed that in the probabilistic output line types, the
TOTAL
> > column
> > > > has a constant value of 6424. The TOTAL column is the number
of
> matched
> > > > pairs that was included in the analysis. All of the
probabilistic
> > > > verification in MET is based on an Nx2 contingency table...
and the
> > TOTAL
> > > > column is the same as the sum of all of the cells of that Nx2
table.
> > The
> > > > FCST_THRESH column indicates which probability bins were used
to
> define
> > > > that Nx2 table and the OBS_THRESH column indicates the
verifying
> > > threshold.
> > > >
> > > > But in all cases, the same set of 6424 matched pairs were used
in the
> > > > analysis.
> > > >
> > > > What part of this process is causing confusion?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Fri, Oct 2, 2020 at 2:09 PM binyu.wang at noaa.gov via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > Fri Oct 02 14:08:56 2020: Request 96942 was acted upon.
> > > > > Transaction: Ticket created by binyu.wang at noaa.gov
> > > > > Queue: met_help
> > > > > Subject: grid_stat output
> > > > > Owner: Nobody
> > > > > Requestors: binyu.wang at noaa.gov
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96942
> > > >
> > > > >
> > > > >
> > > > > Hello,
> > > > >
> > > > > There are two things I want to confirm:
> > > > > For attachment 1: it is part of the grid_stat output:
> > > > > Why "Total number of matched pairs:" (column after
LINE_TYPE) are
> the
> > > > same
> > > > > no matter which thresh_old is used? So this number is not
related
> to
> > > the
> > > > > case when which specific threshold is used? It is the
"TOTAL",
> > correct?
> > > > >
> > > > > For attachment 2:
> > > > > Does the number on the second Y-axis refer to the number of
> matching
> > > > pairs
> > > > > that meet the standard? Those number is much higher than
what is
> > shown
> > > in
> > > > > the log file
> > > > > Thank you.
> > > > >
> > > > > Binyu
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list