[Met_help] [rt.rap.ucar.edu #96348] History for CNT_THRESH clarification in Point_Stat
John Halley Gotway via RT
met_help at ucar.edu
Wed Aug 19 10:17:13 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi all,
I'm having an issue with Point_Stat, which has to be in my configuration file, and I'm having a hard time understanding what the issue is. Can't see the forest through the trees, I guess.
I'm trying to run an analysis (continuous) on wind gusts vs MADIS data, but only for forecast/obs over 16 m/s.
I've run the analysis, and point_stat finds matches, but gives me no tangible results. So, my assumption is that I have something wrong in my config file, but I honestly can't see what it is.
I've attached the config file and a snapshot of the ctc table, to give an example of the results. If you prefer me to add these to the ftp, or to include the actual data the analysis was run on, just let me know. I have a feeling they won't be needed.
For reference, this uses data from August 4th and includes areas of MidAtlantic coast-so it focuses on Tropical Storm Isaias where there would have been gusts both forecasted and observed over 16 m/s.
P.S. I'm really only worried about the GUST variable at this time...I will adjust the other wind variables once I figure out the problem.
Thanks!
-Tom
[https://firstenergycorp.com/content/dam/opcologos/emailsig/FE-logo.png]
Thomas Workoff
Sr Scientist
office: 330-436-1475 (850-1475)
tworkoff at firstenergycorp.com
341 White Pond Drive, Akron, OH 44320 | mailstop: A-WAC-C1 / AK-West Akron Campus
------------------------------------------------------------------------------
The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: CNT_THRESH clarification in Point_Stat
From: John Halley Gotway
Time: Wed Aug 19 09:31:20 2020
Thomas,
I see you have a question about the application of thresholds in the
Point-Stat tool. Judging by the title of your email "CNT_THRESH
clarification..." and the attachments you sent, containing the CTC
line
type output, I suspect there's just some confusion about how
thresholds
from the config file are applied in the computation of statistics. So
hopefully, this will be easy to clear up.
First, let me note that the string "NA" is actually supported as a
valid
threshold in MET configuration files. It just always evaluates to
true. The
"NA" threshold often serves as a good default value. For example, if
you
want to apply a real threshold to observation values and nothing to
forecast values, you'd set the fcst threshold to NA.
Here's a line from the Point-Stat config file you sent:
*cat_thresh = [ NA ];*
Since this appears at the highest level of context, it is applied to
both
the forecast and observation datasets. And it is the "cat_thresh"
threshold
which applies to the output contingency table counts and statistics in
the
CTC and CTS line types from Point-Stat. Since NA always evaluates to
true,
all the events in the CTC line type show up in the hit column (i.e.
FY_OY).
Switching instead to "cat_thresh = [ >16 ];" will produce the output
you
expect.
Stepping back a little, there are 3 main thresholds in the Point-Stat
config file:
cat_thresh, cnt_thresh, and wind_thresh
*cat_thresh* defines the "categorical thresholds" used to *define
events*
when computing CTC and CTS line types. If multiple thresholds are
provided
and MCTC or MCTS output is requested, then they are also used to
compute
those multi-category contingency table counts and statistics.
*cnt_thresh* defines the "continuous thresholds" used to *filter the
matched pairs* when computing the SL1L2, SAL1L2, and CNT line types.
This
is called conditional verification. For example, perhaps you want to
see
the RMSE statistics, but only for pairs where the observation value
exceeded 10. You could set the fcst cnt_thresh = NA; the obs
cnt_thresh =
>10; and the cnt_logic = INTERSECTION;
The cnt_logic defines how to combine the fcst and obs cnt_thresh
settings.
*wind_thresh* defines the "wind speed thresholds" used to *filter the
U/V
matched pairs* when computing the VL1L2, VAL1L2, and VCNT line types.
Similar to cnt_thresh, wind_thresh filters out only those matched
pairs
which meet the wind speed criteria and computes derives partial sums
and
statistics from them. Like cnt_logic, there is also a wind_logic
option to
define how to combine the fcst and obs wind speed thresholds.
I realize that this is a little confusing... cat_thresh defines events
over
all of the matched pairs found whereas cnt_thresh and wind_thresh
define
filters to subset the matched pairs.
The default settings for cnt_thresh and wind_thresh are simply "NA",
since
we figured the best default would be to simply use all available
matched
pairs. Since you'll probably want to apply different thresholds to
different fields, just define those thresholds with the dictionary for
each
field. For example:
{
name = "GUST";
level = [ "L0" ];
cat_thresh = [ >5, >10, >16 ];
},
{
name = "WIND";
level = [ "Z10" ];
cat_thresh = [ >2, >4 ];
},
In all cases, the FCST_THRESH and OBS_THRESH columns in the .stat
output
files indicate which thresholds were applied in the computation of
those
partial sums or statistics.
Hope that helps clarify.
Thanks,
John Halley Gotway
On Wed, Aug 19, 2020 at 8:57 AM Workoff, Thomas E via RT
<met_help at ucar.edu>
wrote:
>
> Wed Aug 19 08:57:15 2020: Request 96348 was acted upon.
> Transaction: Ticket created by tworkoff at firstenergycorp.com
> Queue: met_help
> Subject: CNT_THRESH clarification in Point_Stat
> Owner: Nobody
> Requestors: tworkoff at firstenergycorp.com
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96348 >
>
>
> Hi all,
>
> I'm having an issue with Point_Stat, which has to be in my
configuration
> file, and I'm having a hard time understanding what the issue is.
Can't
> see the forest through the trees, I guess.
>
> I'm trying to run an analysis (continuous) on wind gusts vs MADIS
data,
> but only for forecast/obs over 16 m/s.
>
> I've run the analysis, and point_stat finds matches, but gives me no
> tangible results. So, my assumption is that I have something wrong
in my
> config file, but I honestly can't see what it is.
>
> I've attached the config file and a snapshot of the ctc table, to
give an
> example of the results. If you prefer me to add these to the ftp,
or to
> include the actual data the analysis was run on, just let me know.
I have
> a feeling they won't be needed.
>
> For reference, this uses data from August 4th and includes areas of
> MidAtlantic coast-so it focuses on Tropical Storm Isaias where there
would
> have been gusts both forecasted and observed over 16 m/s.
>
> P.S. I'm really only worried about the GUST variable at this
time...I will
> adjust the other wind variables once I figure out the problem.
>
> Thanks!
> -Tom
>
>
>
>
> [https://firstenergycorp.com/content/dam/opcologos/emailsig/FE-
logo.png]
>
> Thomas Workoff
> Sr Scientist
> office: 330-436-1475 (850-1475)
> tworkoff at firstenergycorp.com
> 341 White Pond Drive, Akron, OH 44320 | mailstop: A-WAC-C1 / AK-West
Akron
> Campus
>
>
>
------------------------------------------------------------------------------
>
> The information contained in this message is intended only for the
> personal and confidential use of the recipient(s) named above. If
the
> reader of this message is not the intended recipient or an agent
> responsible for delivering it to the intended recipient, you are
hereby
> notified that you have received this document in error and that any
review,
> dissemination, distribution, or copying of this message is strictly
> prohibited. If you have received this communication in error, please
notify
> us immediately, and delete the original message.
>
>
------------------------------------------------
Subject: RE: [EXTERNAL] Re: [rt.rap.ucar.edu #96348] CNT_THRESH clarification in Point_Stat
From: Workoff, Thomas E
Time: Wed Aug 19 10:08:18 2020
John,
As always, thanks for your insight!
This helps greatly. I don't know why, but I though employing the
CNT_THRESH would simply apply to all the CAT_THRESH as well. My fault,
obviously.
I've simply stepped through and been explicit with my cnt and cat
thresholds, and things seem to working as planned now. You can close
any outstanding ticket.
Thank you for taking the time to set me straight!
Stay safe--
-Tom
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Wednesday, August 19, 2020 11:31 AM
To: Workoff, Thomas E <tworkoff at firstenergycorp.com>
Subject: [EXTERNAL] Re: [rt.rap.ucar.edu #96348] CNT_THRESH
clarification in Point_Stat
Thomas,
I see you have a question about the application of thresholds in the
Point-Stat tool. Judging by the title of your email "CNT_THRESH
clarification..." and the attachments you sent, containing the CTC
line type output, I suspect there's just some confusion about how
thresholds from the config file are applied in the computation of
statistics. So hopefully, this will be easy to clear up.
First, let me note that the string "NA" is actually supported as a
valid threshold in MET configuration files. It just always evaluates
to true. The "NA" threshold often serves as a good default value. For
example, if you want to apply a real threshold to observation values
and nothing to forecast values, you'd set the fcst threshold to NA.
Here's a line from the Point-Stat config file you sent:
*cat_thresh = [ NA ];*
Since this appears at the highest level of context, it is applied to
both the forecast and observation datasets. And it is the "cat_thresh"
threshold which applies to the output contingency table counts and
statistics in the CTC and CTS line types from Point-Stat. Since NA
always evaluates to true, all the events in the CTC line type show up
in the hit column (i.e. FY_OY).
Switching instead to "cat_thresh = [ >16 ];" will produce the output
you expect.
Stepping back a little, there are 3 main thresholds in the Point-Stat
config file:
cat_thresh, cnt_thresh, and wind_thresh
*cat_thresh* defines the "categorical thresholds" used to *define
events* when computing CTC and CTS line types. If multiple thresholds
are provided and MCTC or MCTS output is requested, then they are also
used to compute those multi-category contingency table counts and
statistics.
*cnt_thresh* defines the "continuous thresholds" used to *filter the
matched pairs* when computing the SL1L2, SAL1L2, and CNT line types.
This is called conditional verification. For example, perhaps you want
to see the RMSE statistics, but only for pairs where the observation
value exceeded 10. You could set the fcst cnt_thresh = NA; the obs
cnt_thresh =
>10; and the cnt_logic = INTERSECTION;
The cnt_logic defines how to combine the fcst and obs cnt_thresh
settings.
*wind_thresh* defines the "wind speed thresholds" used to *filter the
U/V matched pairs* when computing the VL1L2, VAL1L2, and VCNT line
types.
Similar to cnt_thresh, wind_thresh filters out only those matched
pairs which meet the wind speed criteria and computes derives partial
sums and statistics from them. Like cnt_logic, there is also a
wind_logic option to define how to combine the fcst and obs wind speed
thresholds.
I realize that this is a little confusing... cat_thresh defines events
over all of the matched pairs found whereas cnt_thresh and wind_thresh
define filters to subset the matched pairs.
The default settings for cnt_thresh and wind_thresh are simply "NA",
since we figured the best default would be to simply use all available
matched pairs. Since you'll probably want to apply different
thresholds to different fields, just define those thresholds with the
dictionary for each field. For example:
{
name = "GUST";
level = [ "L0" ];
cat_thresh = [ >5, >10, >16 ];
},
{
name = "WIND";
level = [ "Z10" ];
cat_thresh = [ >2, >4 ];
},
In all cases, the FCST_THRESH and OBS_THRESH columns in the .stat
output files indicate which thresholds were applied in the computation
of those partial sums or statistics.
Hope that helps clarify.
Thanks,
John Halley Gotway
On Wed, Aug 19, 2020 at 8:57 AM Workoff, Thomas E via RT
<met_help at ucar.edu>
wrote:
>
> Wed Aug 19 08:57:15 2020: Request 96348 was acted upon.
> Transaction: Ticket created by tworkoff at firstenergycorp.com
> Queue: met_help
> Subject: CNT_THRESH clarification in Point_Stat
> Owner: Nobody
> Requestors: tworkoff at firstenergycorp.com
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96348
> >
>
>
> Hi all,
>
> I'm having an issue with Point_Stat, which has to be in my
> configuration file, and I'm having a hard time understanding what
the
> issue is. Can't see the forest through the trees, I guess.
>
> I'm trying to run an analysis (continuous) on wind gusts vs MADIS
> data, but only for forecast/obs over 16 m/s.
>
> I've run the analysis, and point_stat finds matches, but gives me no
> tangible results. So, my assumption is that I have something wrong
in
> my config file, but I honestly can't see what it is.
>
> I've attached the config file and a snapshot of the ctc table, to
give
> an example of the results. If you prefer me to add these to the
ftp,
> or to include the actual data the analysis was run on, just let me
> know. I have a feeling they won't be needed.
>
> For reference, this uses data from August 4th and includes areas of
> MidAtlantic coast-so it focuses on Tropical Storm Isaias where there
> would have been gusts both forecasted and observed over 16 m/s.
>
> P.S. I'm really only worried about the GUST variable at this
time...I
> will adjust the other wind variables once I figure out the problem.
>
> Thanks!
> -Tom
>
>
>
>
> [https://firstenergycorp.com/content/dam/opcologos/emailsig/FE-
logo.pn
> g]
>
> Thomas Workoff
> Sr Scientist
> office: 330-436-1475 (850-1475)
> tworkoff at firstenergycorp.com
> 341 White Pond Drive, Akron, OH 44320 | mailstop: A-WAC-C1 / AK-West
> Akron Campus
>
>
>
----------------------------------------------------------------------
> --------
>
> The information contained in this message is intended only for the
> personal and confidential use of the recipient(s) named above. If
the
> reader of this message is not the intended recipient or an agent
> responsible for delivering it to the intended recipient, you are
> hereby notified that you have received this document in error and
that
> any review, dissemination, distribution, or copying of this message
is
> strictly prohibited. If you have received this communication in
error,
> please notify us immediately, and delete the original message.
>
>
------------------------------------------------------------------------------
The information contained in this message is intended only for the
personal and confidential use of the recipient(s) named above. If the
reader of this message is not the intended recipient or an agent
responsible for delivering it to the intended recipient, you are
hereby notified that you have received this document in error and that
any review, dissemination, distribution, or copying of this message is
strictly prohibited. If you have received this communication in error,
please notify us immediately, and delete the original message.
------------------------------------------------
More information about the Met_help
mailing list