[Met_help] [rt.rap.ucar.edu #95202] History for Point Stat -- verification by 'bins'?
John Halley Gotway via RT
met_help at ucar.edu
Fri May 8 17:11:27 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi all,
A bit of a weird strategy question here...and I apologize if this has been covered somewhere in the archives, and I just missed it!
I'm wondering if I can set up some verification by 'bins' (regarding wind speed, but that's not overly important at the moment).
Basically, I want to set up bins....for example 40-45 mph, 45-50mph, 50-55 mph....where if the forecast and observation land in the same 'bin', then its a hit. So, a 47 mph forecast that was observed at 49 mph would hit, while an observation of 43 mph would miss.
Is there someway to do this from MET? In a config file? I'm a bit leery of having to go in and postprocess all the forecasts (and ESPECIALLY leery of postprocessing the observations) to fit these bins. I guess this is a 'neighborhood' style of approach, but by value magnitude, and not spatially.
Any help and/or input would be greatly appreciated, and as always, thanks for your time!
-Tom
Tom Workoff
Senior Scientist
Environmental Energy Delivery
FirstEnergy
Office: 330-436-1475
tworkoff at firstenergycorp.com<mailto:tworkoff at firstenergycorp.com>
------------------------------------------------------------------------------
The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Point Stat -- verification by 'bins'?
From: David Fillmore
Time: Fri May 08 14:45:51 2020
Hi Tom-
This verification approach is already supported in Grid-Stat and
Point-Stat
using multi-category contingency tables and statistics, e.g. the MCTC
and
MCTS line types.
See the cat_thresh config entry to define your bin ranges.
Here is an example config file that you can look at
https://github.com/NCAR/MET/blob/master_v9.0/met/scripts/config/GridStatConfig_APCP_24
Let us know if this works ...
thanks,
David
On Fri, May 8, 2020 at 12:02 PM Workoff, Thomas E via RT
<met_help at ucar.edu>
wrote:
>
> Fri May 08 12:02:24 2020: Request 95202 was acted upon.
> Transaction: Ticket created by tworkoff at firstenergycorp.com
> Queue: met_help
> Subject: Point Stat -- verification by 'bins'?
> Owner: Nobody
> Requestors: tworkoff at firstenergycorp.com
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95202 >
>
>
> Hi all,
>
> A bit of a weird strategy question here...and I apologize if this
has been
> covered somewhere in the archives, and I just missed it!
>
> I'm wondering if I can set up some verification by 'bins' (regarding
wind
> speed, but that's not overly important at the moment).
>
> Basically, I want to set up bins....for example 40-45 mph, 45-50mph,
50-55
> mph....where if the forecast and observation land in the same 'bin',
then
> its a hit. So, a 47 mph forecast that was observed at 49 mph would
hit,
> while an observation of 43 mph would miss.
>
> Is there someway to do this from MET? In a config file? I'm a bit
leery
> of having to go in and postprocess all the forecasts (and ESPECIALLY
leery
> of postprocessing the observations) to fit these bins. I guess this
is a
> 'neighborhood' style of approach, but by value magnitude, and not
spatially.
>
> Any help and/or input would be greatly appreciated, and as always,
thanks
> for your time!
>
> -Tom
>
> Tom Workoff
> Senior Scientist
> Environmental Energy Delivery
> FirstEnergy
> Office: 330-436-1475
> tworkoff at firstenergycorp.com<mailto:tworkoff at firstenergycorp.com>
>
>
>
------------------------------------------------------------------------------
>
> The information contained in this message is intended only for the
> personal and confidential use of the recipient(s) named above. If
the
> reader of this message is not the intended recipient or an agent
> responsible for delivering it to the intended recipient, you are
hereby
> notified that you have received this document in error and that any
review,
> dissemination, distribution, or copying of this message is strictly
> prohibited. If you have received this communication in error, please
notify
> us immediately, and delete the original message.
>
>
------------------------------------------------
Subject: RE: [EXTERNAL] Re: [rt.rap.ucar.edu #95202] Point Stat -- verification by 'bins'?
From: Workoff, Thomas E
Time: Fri May 08 15:09:26 2020
Thanks, David.
I thought this was the case, but just wanted to confirm that this
method was treating all values in bins, and not simply calculating
metrics and having the results reported out in the different
'bins'....if that made sense (it probably didn't).
This clarification is very helpful.
Thanks for the quick response!
-----Original Message-----
From: David Fillmore via RT <met_help at ucar.edu>
Sent: Friday, May 8, 2020 4:46 PM
To: Workoff, Thomas E <tworkoff at firstenergycorp.com>
Subject: [EXTERNAL] Re: [rt.rap.ucar.edu #95202] Point Stat --
verification by 'bins'?
Hi Tom-
This verification approach is already supported in Grid-Stat and
Point-Stat using multi-category contingency tables and statistics,
e.g. the MCTC and MCTS line types.
See the cat_thresh config entry to define your bin ranges.
Here is an example config file that you can look at
https://github.com/NCAR/MET/blob/master_v9.0/met/scripts/config/GridStatConfig_APCP_24
Let us know if this works ...
thanks,
David
On Fri, May 8, 2020 at 12:02 PM Workoff, Thomas E via RT
<met_help at ucar.edu>
wrote:
>
> Fri May 08 12:02:24 2020: Request 95202 was acted upon.
> Transaction: Ticket created by tworkoff at firstenergycorp.com
> Queue: met_help
> Subject: Point Stat -- verification by 'bins'?
> Owner: Nobody
> Requestors: tworkoff at firstenergycorp.com
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95202
> >
>
>
> Hi all,
>
> A bit of a weird strategy question here...and I apologize if this
has
> been covered somewhere in the archives, and I just missed it!
>
> I'm wondering if I can set up some verification by 'bins' (regarding
> wind speed, but that's not overly important at the moment).
>
> Basically, I want to set up bins....for example 40-45 mph, 45-50mph,
> 50-55 mph....where if the forecast and observation land in the same
> 'bin', then its a hit. So, a 47 mph forecast that was observed at
49
> mph would hit, while an observation of 43 mph would miss.
>
> Is there someway to do this from MET? In a config file? I'm a bit
leery
> of having to go in and postprocess all the forecasts (and ESPECIALLY
> leery of postprocessing the observations) to fit these bins. I
guess
> this is a 'neighborhood' style of approach, but by value magnitude,
and not spatially.
>
> Any help and/or input would be greatly appreciated, and as always,
> thanks for your time!
>
> -Tom
>
> Tom Workoff
> Senior Scientist
> Environmental Energy Delivery
> FirstEnergy
> Office: 330-436-1475
> tworkoff at firstenergycorp.com<mailto:tworkoff at firstenergycorp.com>
>
>
>
----------------------------------------------------------------------
> --------
>
> The information contained in this message is intended only for the
> personal and confidential use of the recipient(s) named above. If
the
> reader of this message is not the intended recipient or an agent
> responsible for delivering it to the intended recipient, you are
> hereby notified that you have received this document in error and
that
> any review, dissemination, distribution, or copying of this message
is
> strictly prohibited. If you have received this communication in
error,
> please notify us immediately, and delete the original message.
>
>
------------------------------------------------------------------------------
The information contained in this message is intended only for the
personal and confidential use of the recipient(s) named above. If the
reader of this message is not the intended recipient or an agent
responsible for delivering it to the intended recipient, you are
hereby notified that you have received this document in error and that
any review, dissemination, distribution, or copying of this message is
strictly prohibited. If you have received this communication in error,
please notify us immediately, and delete the original message.
------------------------------------------------
Subject: Point Stat -- verification by 'bins'?
From: John Halley Gotway
Time: Fri May 08 16:21:57 2020
Thomas,
Just wanted to jump on with another comment on this ticket. The MCTC
and
MCTS lines contain exactly what you describe... applying thresholds
for a
multiple categories in a single contingency table.
The cnt_thresh is also available. It can be used to compute
continuous
statistics, but only within the range you specify. For example, If
you set
cnt_thresh = [ >5 ]; in the obs dictionary for wind speed, you could
compute the RMSE only when that condition is met. Thought you might
also
find that useful.
Thanks
John
On Fri, May 8, 2020 at 3:09 PM Workoff, Thomas E via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95202 >
>
> Thanks, David.
>
> I thought this was the case, but just wanted to confirm that this
method
> was treating all values in bins, and not simply calculating metrics
and
> having the results reported out in the different 'bins'....if that
made
> sense (it probably didn't).
>
> This clarification is very helpful.
>
> Thanks for the quick response!
>
> -----Original Message-----
> From: David Fillmore via RT <met_help at ucar.edu>
> Sent: Friday, May 8, 2020 4:46 PM
> To: Workoff, Thomas E <tworkoff at firstenergycorp.com>
> Subject: [EXTERNAL] Re: [rt.rap.ucar.edu #95202] Point Stat --
> verification by 'bins'?
>
> Hi Tom-
>
> This verification approach is already supported in Grid-Stat and
> Point-Stat using multi-category contingency tables and statistics,
e.g. the
> MCTC and MCTS line types.
> See the cat_thresh config entry to define your bin ranges.
> Here is an example config file that you can look at
>
>
https://github.com/NCAR/MET/blob/master_v9.0/met/scripts/config/GridStatConfig_APCP_24
>
>
> Let us know if this works ...
> thanks,
> David
>
>
> On Fri, May 8, 2020 at 12:02 PM Workoff, Thomas E via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > Fri May 08 12:02:24 2020: Request 95202 was acted upon.
> > Transaction: Ticket created by tworkoff at firstenergycorp.com
> > Queue: met_help
> > Subject: Point Stat -- verification by 'bins'?
> > Owner: Nobody
> > Requestors: tworkoff at firstenergycorp.com
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95202
> > >
> >
> >
> > Hi all,
> >
> > A bit of a weird strategy question here...and I apologize if this
has
> > been covered somewhere in the archives, and I just missed it!
> >
> > I'm wondering if I can set up some verification by 'bins'
(regarding
> > wind speed, but that's not overly important at the moment).
> >
> > Basically, I want to set up bins....for example 40-45 mph, 45-
50mph,
> > 50-55 mph....where if the forecast and observation land in the
same
> > 'bin', then its a hit. So, a 47 mph forecast that was observed at
49
> > mph would hit, while an observation of 43 mph would miss.
> >
> > Is there someway to do this from MET? In a config file? I'm a
bit leery
> > of having to go in and postprocess all the forecasts (and
ESPECIALLY
> > leery of postprocessing the observations) to fit these bins. I
guess
> > this is a 'neighborhood' style of approach, but by value
magnitude, and
> not spatially.
> >
> > Any help and/or input would be greatly appreciated, and as always,
> > thanks for your time!
> >
> > -Tom
> >
> > Tom Workoff
> > Senior Scientist
> > Environmental Energy Delivery
> > FirstEnergy
> > Office: 330-436-1475
> > tworkoff at firstenergycorp.com<mailto:tworkoff at firstenergycorp.com>
> >
> >
> >
----------------------------------------------------------------------
> > --------
> >
> > The information contained in this message is intended only for the
> > personal and confidential use of the recipient(s) named above. If
the
> > reader of this message is not the intended recipient or an agent
> > responsible for delivering it to the intended recipient, you are
> > hereby notified that you have received this document in error and
that
> > any review, dissemination, distribution, or copying of this
message is
> > strictly prohibited. If you have received this communication in
error,
> > please notify us immediately, and delete the original message.
> >
> >
>
>
>
------------------------------------------------------------------------------
>
> The information contained in this message is intended only for the
> personal and confidential use of the recipient(s) named above. If
the
> reader of this message is not the intended recipient or an agent
> responsible for delivering it to the intended recipient, you are
hereby
> notified that you have received this document in error and that any
review,
> dissemination, distribution, or copying of this message is strictly
> prohibited. If you have received this communication in error, please
notify
> us immediately, and delete the original message.
>
>
>
------------------------------------------------
Subject: RE: [EXTERNAL] Re: [rt.rap.ucar.edu #95202] Point Stat -- verification by 'bins'?
From: Workoff, Thomas E
Time: Fri May 08 16:43:18 2020
Thanks, John!
I have a much better understanding regarding the cnt and cat
thresholds now, and the difference between them. Honestly, they're
pretty self-explanatory, but for some reason I got spun around on the
difference between the two, and what was physically going on when
applying them.
I appreciate your help!
Have a great weekend, and stay safe!
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Friday, May 8, 2020 6:22 PM
To: Workoff, Thomas E <tworkoff at firstenergycorp.com>
Subject: Re: [EXTERNAL] Re: [rt.rap.ucar.edu #95202] Point Stat --
verification by 'bins'?
Thomas,
Just wanted to jump on with another comment on this ticket. The MCTC
and MCTS lines contain exactly what you describe... applying
thresholds for a multiple categories in a single contingency table.
The cnt_thresh is also available. It can be used to compute
continuous statistics, but only within the range you specify. For
example, If you set cnt_thresh = [ >5 ]; in the obs dictionary for
wind speed, you could compute the RMSE only when that condition is
met. Thought you might also find that useful.
Thanks
John
On Fri, May 8, 2020 at 3:09 PM Workoff, Thomas E via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95202 >
>
> Thanks, David.
>
> I thought this was the case, but just wanted to confirm that this
> method was treating all values in bins, and not simply calculating
> metrics and having the results reported out in the different
> 'bins'....if that made sense (it probably didn't).
>
> This clarification is very helpful.
>
> Thanks for the quick response!
>
> -----Original Message-----
> From: David Fillmore via RT <met_help at ucar.edu>
> Sent: Friday, May 8, 2020 4:46 PM
> To: Workoff, Thomas E <tworkoff at firstenergycorp.com>
> Subject: [EXTERNAL] Re: [rt.rap.ucar.edu #95202] Point Stat --
> verification by 'bins'?
>
> Hi Tom-
>
> This verification approach is already supported in Grid-Stat and
> Point-Stat using multi-category contingency tables and statistics,
> e.g. the MCTC and MCTS line types.
> See the cat_thresh config entry to define your bin ranges.
> Here is an example config file that you can look at
>
>
https://github.com/NCAR/MET/blob/master_v9.0/met/scripts/config/GridSt
> atConfig_APCP_24
>
>
> Let us know if this works ...
> thanks,
> David
>
>
> On Fri, May 8, 2020 at 12:02 PM Workoff, Thomas E via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > Fri May 08 12:02:24 2020: Request 95202 was acted upon.
> > Transaction: Ticket created by tworkoff at firstenergycorp.com
> > Queue: met_help
> > Subject: Point Stat -- verification by 'bins'?
> > Owner: Nobody
> > Requestors: tworkoff at firstenergycorp.com
> > Status: new
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95202
> > >
> >
> >
> > Hi all,
> >
> > A bit of a weird strategy question here...and I apologize if this
> > has been covered somewhere in the archives, and I just missed it!
> >
> > I'm wondering if I can set up some verification by 'bins'
(regarding
> > wind speed, but that's not overly important at the moment).
> >
> > Basically, I want to set up bins....for example 40-45 mph, 45-
50mph,
> > 50-55 mph....where if the forecast and observation land in the
same
> > 'bin', then its a hit. So, a 47 mph forecast that was observed at
> > 49 mph would hit, while an observation of 43 mph would miss.
> >
> > Is there someway to do this from MET? In a config file? I'm a
bit leery
> > of having to go in and postprocess all the forecasts (and
ESPECIALLY
> > leery of postprocessing the observations) to fit these bins. I
> > guess this is a 'neighborhood' style of approach, but by value
> > magnitude, and
> not spatially.
> >
> > Any help and/or input would be greatly appreciated, and as always,
> > thanks for your time!
> >
> > -Tom
> >
> > Tom Workoff
> > Senior Scientist
> > Environmental Energy Delivery
> > FirstEnergy
> > Office: 330-436-1475
> > tworkoff at firstenergycorp.com<mailto:tworkoff at firstenergycorp.com>
> >
> >
> >
--------------------------------------------------------------------
> > --
> > --------
> >
> > The information contained in this message is intended only for the
> > personal and confidential use of the recipient(s) named above. If
> > the reader of this message is not the intended recipient or an
agent
> > responsible for delivering it to the intended recipient, you are
> > hereby notified that you have received this document in error and
> > that any review, dissemination, distribution, or copying of this
> > message is strictly prohibited. If you have received this
> > communication in error, please notify us immediately, and delete
the original message.
> >
> >
>
>
>
----------------------------------------------------------------------
> --------
>
> The information contained in this message is intended only for the
> personal and confidential use of the recipient(s) named above. If
the
> reader of this message is not the intended recipient or an agent
> responsible for delivering it to the intended recipient, you are
> hereby notified that you have received this document in error and
that
> any review, dissemination, distribution, or copying of this message
is
> strictly prohibited. If you have received this communication in
error,
> please notify us immediately, and delete the original message.
>
>
>
------------------------------------------------------------------------------
The information contained in this message is intended only for the
personal and confidential use of the recipient(s) named above. If the
reader of this message is not the intended recipient or an agent
responsible for delivering it to the intended recipient, you are
hereby notified that you have received this document in error and that
any review, dissemination, distribution, or copying of this message is
strictly prohibited. If you have received this communication in error,
please notify us immediately, and delete the original message.
------------------------------------------------
More information about the Met_help
mailing list