[Met_help] [rt.rap.ucar.edu #97396] History for MET - distribution of forecast values for non-events
John Halley Gotway via RT
met_help at ucar.edu
Wed Feb 10 09:01:35 MST 2021
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Folks - I have a question about Likelihood definition on page 448 of MET_Users_Guide_v9; in it you say the following:
The distribution of forecast values for non-events can be derived from the POFD values computed by MET for the user-speci
ed thresholds.
My question is how to derive this distribution from the POFD values computed by MET. Thanks!
Efren A. Serra (Contractor)
Physicist
DeVine Consulting, Inc.
Naval Research Laboratory
Marine Meteorology Division
7 Grace Hopper Ave., STOP 2
Monterey, CA 93943
Code 7542
Mobile: 408-425-5027
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: MET - distribution of forecast values for non-events
From: John Halley Gotway
Time: Tue Nov 10 12:03:19 2020
Efren,
I see you have a question about the description of the Likelihood
statistic
in Appendix C of the MET User's Guide. Since MET version 9.0, we've
switched our documentation to using Sphinx. You can access it online
now:
https://dtcenter.github.io/MET/latest/Users_Guide/appendixC.html#likelihood
And you're asking about the reference to discrimination diagrams.
Here's a
description of them on another site:
https://www.cawcr.gov.au/projects/verification/
Search for "Discrimination Diagram" there.
So for each probability bin, we plot PODY and POFD. I was hoping to
send
you an example of making a discrimination diagram using METviewer, but
unfortunately, I don't think that's currently supported.
I looked at the probabilistic output line types from MET, which
include
PCT, PJC, PRC, and PSTD. I have some ideas about which of these stats
could
be used to construct a discrimination diagram but I'm not 100%
positive and
I don't want to lead you astray. For this reason, I'm going to re-
assign
this ticket to Eric, a statistician on our team. Hopefully, he'll be
able
to tell you which columns from MET output should be used to construct
a
discrimination diagram.
Thanks,
John
On Tue, Nov 10, 2020 at 8:22 AM efren.serra.ctr at nrlmry.navy.mil via RT
<
met_help at ucar.edu> wrote:
>
> Tue Nov 10 08:22:53 2020: Request 97396 was acted upon.
> Transaction: Ticket created by efren.serra.ctr at nrlmry.navy.mil
> Queue: met_help
> Subject: MET - distribution of forecast values for non-events
> Owner: Nobody
> Requestors: efren.serra.ctr at nrlmry.navy.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97396 >
>
>
> Folks - I have a question about Likelihood definition on page 448 of
> MET_Users_Guide_v9; in it you say the following:
> The distribution of forecast values for non-events can be derived
from the
> POFD values computed by MET for the user-speci ed thresholds.
>
> My question is how to derive this distribution from the POFD values
> computed by MET. Thanks!
>
> Efren A. Serra (Contractor)
> Physicist
>
> DeVine Consulting, Inc.
> Naval Research Laboratory
> Marine Meteorology Division
> 7 Grace Hopper Ave., STOP 2
> Monterey, CA 93943
> Code 7542
> Mobile: 408-425-5027
>
>
>
------------------------------------------------
Subject: MET - distribution of forecast values for non-events
From: John Halley Gotway
Time: Tue Nov 10 12:09:42 2020
Just realized that I'm not able to directly re-assign to Eric, so I'm
cc'ing him here.
Eric, can you please take a look at this message history? The question
is... exactly which columns from MET probabilistic output line types
should
be used to construct a discrimination diagram?
Thanks!
John
On Tue, Nov 10, 2020 at 12:03 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Efren,
>
> I see you have a question about the description of the Likelihood
> statistic in Appendix C of the MET User's Guide. Since MET version
9.0,
> we've switched our documentation to using Sphinx. You can access it
online
> now:
>
>
https://dtcenter.github.io/MET/latest/Users_Guide/appendixC.html#likelihood
>
> And you're asking about the reference to discrimination diagrams.
Here's a
> description of them on another site:
> https://www.cawcr.gov.au/projects/verification/
> Search for "Discrimination Diagram" there.
>
> So for each probability bin, we plot PODY and POFD. I was hoping to
send
> you an example of making a discrimination diagram using METviewer,
but
> unfortunately, I don't think that's currently supported.
>
> I looked at the probabilistic output line types from MET, which
include
> PCT, PJC, PRC, and PSTD. I have some ideas about which of these
stats could
> be used to construct a discrimination diagram but I'm not 100%
positive and
> I don't want to lead you astray. For this reason, I'm going to re-
assign
> this ticket to Eric, a statistician on our team. Hopefully, he'll be
able
> to tell you which columns from MET output should be used to
construct a
> discrimination diagram.
>
> Thanks,
> John
>
>
>
> On Tue, Nov 10, 2020 at 8:22 AM efren.serra.ctr at nrlmry.navy.mil via
RT <
> met_help at ucar.edu> wrote:
>
>>
>> Tue Nov 10 08:22:53 2020: Request 97396 was acted upon.
>> Transaction: Ticket created by efren.serra.ctr at nrlmry.navy.mil
>> Queue: met_help
>> Subject: MET - distribution of forecast values for non-events
>> Owner: Nobody
>> Requestors: efren.serra.ctr at nrlmry.navy.mil
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97396 >
>>
>>
>> Folks - I have a question about Likelihood definition on page 448
of
>> MET_Users_Guide_v9; in it you say the following:
>> The distribution of forecast values for non-events can be derived
from
>> the POFD values computed by MET for the user-speci ed thresholds.
>>
>> My question is how to derive this distribution from the POFD values
>> computed by MET. Thanks!
>>
>> Efren A. Serra (Contractor)
>> Physicist
>>
>> DeVine Consulting, Inc.
>> Naval Research Laboratory
>> Marine Meteorology Division
>> 7 Grace Hopper Ave., STOP 2
>> Monterey, CA 93943
>> Code 7542
>> Mobile: 408-425-5027
>>
>>
>>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #97396] MET - distribution of forecast values for non-events
From: efren.serra.ctr at nrlmry.navy.mil
Time: Tue Nov 10 13:19:42 2020
I look forward to Eric's reply. Plots of POFD vs PODY I use to create
a ROC diagram. I'm interested in the "discrimination" diagram now.
Thanks, John!
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Tuesday, November 10, 2020 11:03 AM
To: Serra, Mr. Efren, Contractor, Code 7531
<efren.serra.ctr at nrlmry.navy.mil>
Subject: Re: [rt.rap.ucar.edu #97396] MET - distribution of forecast
values for non-events
Efren,
I see you have a question about the description of the Likelihood
statistic in Appendix C of the MET User's Guide. Since MET version
9.0, we've switched our documentation to using Sphinx. You can access
it online now:
https://dtcenter.github.io/MET/latest/Users_Guide/appendixC.html#likelihood
And you're asking about the reference to discrimination diagrams.
Here's a description of them on another site:
https://www.cawcr.gov.au/projects/verification/
Search for "Discrimination Diagram" there.
So for each probability bin, we plot PODY and POFD. I was hoping to
send you an example of making a discrimination diagram using
METviewer, but unfortunately, I don't think that's currently
supported.
I looked at the probabilistic output line types from MET, which
include PCT, PJC, PRC, and PSTD. I have some ideas about which of
these stats could be used to construct a discrimination diagram but
I'm not 100% positive and I don't want to lead you astray. For this
reason, I'm going to re-assign this ticket to Eric, a statistician on
our team. Hopefully, he'll be able to tell you which columns from MET
output should be used to construct a discrimination diagram.
Thanks,
John
On Tue, Nov 10, 2020 at 8:22 AM efren.serra.ctr at nrlmry.navy.mil via RT
< met_help at ucar.edu> wrote:
>
> Tue Nov 10 08:22:53 2020: Request 97396 was acted upon.
> Transaction: Ticket created by efren.serra.ctr at nrlmry.navy.mil
> Queue: met_help
> Subject: MET - distribution of forecast values for non-events
> Owner: Nobody
> Requestors: efren.serra.ctr at nrlmry.navy.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97396
> >
>
>
> Folks - I have a question about Likelihood definition on page 448 of
> MET_Users_Guide_v9; in it you say the following:
> The distribution of forecast values for non-events can be derived
from
> the POFD values computed by MET for the user-speci ed thresholds.
>
> My question is how to derive this distribution from the POFD values
> computed by MET. Thanks!
>
> Efren A. Serra (Contractor)
> Physicist
>
> DeVine Consulting, Inc.
> Naval Research Laboratory
> Marine Meteorology Division
> 7 Grace Hopper Ave., STOP 2
> Monterey, CA 93943
> Code 7542
> Mobile: 408-425-5027
>
>
>
------------------------------------------------
Subject: MET - distribution of forecast values for non-events
From: John Halley Gotway
Time: Thu Nov 12 10:10:52 2020
Efren,
Here's the response Eric sent:
The discrimination diagram consists of a plot of p( y_i | o_j ) i =
1,,,k
and j = 1,2 (assuming the observations are dichotomous) [i.e., the
conditional frequencies of the forecast when the dichotomous
observation is
either zero or one] against the unconditional forecast probabilities
p( y_i
). It also should include a legend (or similar) giving the base rates
for
p( o_1 ) and p( o_2 ). So, I don't know what these are called in the
MET
probabilistic output line types but if you tell me what the possible
names
are perhaps I could figure that out.
Thanks,
John
On Tue, Nov 10, 2020 at 1:20 PM efren.serra.ctr at nrlmry.navy.mil via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97396 >
>
> I look forward to Eric's reply. Plots of POFD vs PODY I use to
create a
> ROC diagram. I'm interested in the "discrimination" diagram now.
Thanks,
> John!
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Tuesday, November 10, 2020 11:03 AM
> To: Serra, Mr. Efren, Contractor, Code 7531 <
> efren.serra.ctr at nrlmry.navy.mil>
> Subject: Re: [rt.rap.ucar.edu #97396] MET - distribution of forecast
> values for non-events
>
> Efren,
>
> I see you have a question about the description of the Likelihood
> statistic in Appendix C of the MET User's Guide. Since MET version
9.0,
> we've switched our documentation to using Sphinx. You can access it
online
> now:
>
>
https://dtcenter.github.io/MET/latest/Users_Guide/appendixC.html#likelihood
>
> And you're asking about the reference to discrimination diagrams.
Here's a
> description of them on another site:
> https://www.cawcr.gov.au/projects/verification/
> Search for "Discrimination Diagram" there.
>
> So for each probability bin, we plot PODY and POFD. I was hoping to
send
> you an example of making a discrimination diagram using METviewer,
but
> unfortunately, I don't think that's currently supported.
>
> I looked at the probabilistic output line types from MET, which
include
> PCT, PJC, PRC, and PSTD. I have some ideas about which of these
stats could
> be used to construct a discrimination diagram but I'm not 100%
positive and
> I don't want to lead you astray. For this reason, I'm going to re-
assign
> this ticket to Eric, a statistician on our team. Hopefully, he'll be
able
> to tell you which columns from MET output should be used to
construct a
> discrimination diagram.
>
> Thanks,
> John
>
>
>
> On Tue, Nov 10, 2020 at 8:22 AM efren.serra.ctr at nrlmry.navy.mil via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Nov 10 08:22:53 2020: Request 97396 was acted upon.
> > Transaction: Ticket created by efren.serra.ctr at nrlmry.navy.mil
> > Queue: met_help
> > Subject: MET - distribution of forecast values for non-events
> > Owner: Nobody
> > Requestors: efren.serra.ctr at nrlmry.navy.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97396
> > >
> >
> >
> > Folks - I have a question about Likelihood definition on page 448
of
> > MET_Users_Guide_v9; in it you say the following:
> > The distribution of forecast values for non-events can be derived
from
> > the POFD values computed by MET for the user-speci ed thresholds.
> >
> > My question is how to derive this distribution from the POFD
values
> > computed by MET. Thanks!
> >
> > Efren A. Serra (Contractor)
> > Physicist
> >
> > DeVine Consulting, Inc.
> > Naval Research Laboratory
> > Marine Meteorology Division
> > 7 Grace Hopper Ave., STOP 2
> > Monterey, CA 93943
> > Code 7542
> > Mobile: 408-425-5027
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #97396] MET - distribution of forecast values for non-events
From: efren.serra.ctr at nrlmry.navy.mil
Time: Thu Nov 12 13:22:15 2020
According to the MET documentation: probability(p_i|o_1) = ni1/n.1,
where ni1 = OY at pi and n.1 = Sum_i ni1. Hence, probability(p_i|o_0)
= ni0/n.0, where ni0 = ON at pi and n.0 = Sum_i ni0. Therefore, just
plotting OY vs ON at probabilitiy thresholds should take be it, isn't
it? Then the labeling he mentions, base rates, is just "ni1/ni." and
"ni0/ni." where ni. = Total or sum_i (ni1 + ni0) or just the sum of
the BASER_i columns in the pjc.txt files.
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Thursday, November 12, 2020 9:11 AM
To: Serra, Mr. Efren, Contractor, Code 7531
<efren.serra.ctr at nrlmry.navy.mil>
Subject: Re: [rt.rap.ucar.edu #97396] MET - distribution of forecast
values for non-events
Efren,
Here's the response Eric sent:
The discrimination diagram consists of a plot of p( y_i | o_j ) i =
1,,,k and j = 1,2 (assuming the observations are dichotomous) [i.e.,
the conditional frequencies of the forecast when the dichotomous
observation is either zero or one] against the unconditional forecast
probabilities p( y_i ). It also should include a legend (or similar)
giving the base rates for p( o_1 ) and p( o_2 ). So, I don't know
what these are called in the MET probabilistic output line types but
if you tell me what the possible names are perhaps I could figure that
out.
Thanks,
John
On Tue, Nov 10, 2020 at 1:20 PM efren.serra.ctr at nrlmry.navy.mil via RT
< met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97396 >
>
> I look forward to Eric's reply. Plots of POFD vs PODY I use to
create
> a ROC diagram. I'm interested in the "discrimination" diagram now.
> Thanks, John!
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Tuesday, November 10, 2020 11:03 AM
> To: Serra, Mr. Efren, Contractor, Code 7531 <
> efren.serra.ctr at nrlmry.navy.mil>
> Subject: Re: [rt.rap.ucar.edu #97396] MET - distribution of forecast
> values for non-events
>
> Efren,
>
> I see you have a question about the description of the Likelihood
> statistic in Appendix C of the MET User's Guide. Since MET version
> 9.0, we've switched our documentation to using Sphinx. You can
access
> it online
> now:
>
>
https://dtcenter.github.io/MET/latest/Users_Guide/appendixC.html#likel
> ihood
>
> And you're asking about the reference to discrimination diagrams.
> Here's a description of them on another site:
> https://www.cawcr.gov.au/projects/verification/
> Search for "Discrimination Diagram" there.
>
> So for each probability bin, we plot PODY and POFD. I was hoping to
> send you an example of making a discrimination diagram using
> METviewer, but unfortunately, I don't think that's currently
supported.
>
> I looked at the probabilistic output line types from MET, which
> include PCT, PJC, PRC, and PSTD. I have some ideas about which of
> these stats could be used to construct a discrimination diagram but
> I'm not 100% positive and I don't want to lead you astray. For this
> reason, I'm going to re-assign this ticket to Eric, a statistician
on
> our team. Hopefully, he'll be able to tell you which columns from
MET
> output should be used to construct a discrimination diagram.
>
> Thanks,
> John
>
>
>
> On Tue, Nov 10, 2020 at 8:22 AM efren.serra.ctr at nrlmry.navy.mil via
RT
> < met_help at ucar.edu> wrote:
>
> >
> > Tue Nov 10 08:22:53 2020: Request 97396 was acted upon.
> > Transaction: Ticket created by efren.serra.ctr at nrlmry.navy.mil
> > Queue: met_help
> > Subject: MET - distribution of forecast values for non-events
> > Owner: Nobody
> > Requestors: efren.serra.ctr at nrlmry.navy.mil
> > Status: new
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97396
> > >
> >
> >
> > Folks - I have a question about Likelihood definition on page 448
of
> > MET_Users_Guide_v9; in it you say the following:
> > The distribution of forecast values for non-events can be derived
> > from the POFD values computed by MET for the user-speci ed
thresholds.
> >
> > My question is how to derive this distribution from the POFD
values
> > computed by MET. Thanks!
> >
> > Efren A. Serra (Contractor)
> > Physicist
> >
> > DeVine Consulting, Inc.
> > Naval Research Laboratory
> > Marine Meteorology Division
> > 7 Grace Hopper Ave., STOP 2
> > Monterey, CA 93943
> > Code 7542
> > Mobile: 408-425-5027
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: MET - distribution of forecast values for non-events
From: John Halley Gotway
Time: Tue Nov 17 10:51:06 2020
Efren,
I realize I never responded. Yes, your logic makes sense to me. Did
that approach result in the expected resulting plot?
I realize that we should add the discrimination diagram as an option
through METviewer. So I'll forward this to the METviewer/METplotpy
developers.
Thanks,
John
------------------------------------------------
Subject: MET - distribution of forecast values for non-events
From: efren.serra.ctr at nrlmry.navy.mil
Time: Tue Nov 17 16:27:54 2020
John - Thanks to your support and contributions we have been making
steady progress. We are now creating correct reliability,
discrimination and ROC diagram plots.
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Tuesday, November 17, 2020 9:51 AM
To: Serra, Mr. Efren, Contractor, Code 7531
<efren.serra.ctr at nrlmry.navy.mil>
Subject: [rt.rap.ucar.edu #97396] MET - distribution of forecast
values for non-events
Efren,
I realize I never responded. Yes, your logic makes sense to me. Did
that approach result in the expected resulting plot?
I realize that we should add the discrimination diagram as an option
through METviewer. So I'll forward this to the METviewer/METplotpy
developers.
Thanks,
John
------------------------------------------------
Subject: MET - distribution of forecast values for non-events
From: Sampson, Mr. Buck
Time: Tue Nov 17 16:41:38 2020
Indeed. And I would be very happy to add John to our manuscript
if/when we
try to publish it. John, it is up to you. This is a follow-on
evaluation of
something we did a few years back, and MET/you have been very helpful.
You
also may have some additions. Just let us know,
Buck
-----Original Message-----
From: Serra, Mr. Efren, Contractor, Code 7531
<efren.serra.ctr at nrlmry.navy.mil>
Sent: Tuesday, November 17, 2020 3:28 PM
To: met_help at ucar.edu
Cc: Sampson, Mr. Buck <Buck.Sampson at nrlmry.navy.mil>
Subject: RE: [rt.rap.ucar.edu #97396] MET - distribution of forecast
values
for non-events
John - Thanks to your support and contributions we have been making
steady
progress. We are now creating correct reliability, discrimination and
ROC
diagram plots.
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Tuesday, November 17, 2020 9:51 AM
To: Serra, Mr. Efren, Contractor, Code 7531
<efren.serra.ctr at nrlmry.navy.mil>
Subject: [rt.rap.ucar.edu #97396] MET - distribution of forecast
values for
non-events
Efren,
I realize I never responded. Yes, your logic makes sense to me. Did
that
approach result in the expected resulting plot?
I realize that we should add the discrimination diagram as an option
through
METviewer. So I'll forward this to the METviewer/METplotpy developers.
Thanks,
John
------------------------------------------------
Subject: MET - distribution of forecast values for non-events
From: John Halley Gotway
Time: Tue Nov 17 16:46:30 2020
Buck and Efren,
I'm glad you all have found MET to be useful and hope we can continue
providing support to NRL in the future!
I'll go ahead and resolve this ticket. Thanks and have a nice
Thanksgiving.
John
On Tue, Nov 17, 2020 at 4:42 PM Sampson, Mr. Buck via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97396 >
>
> Indeed. And I would be very happy to add John to our manuscript
if/when
> we
> try to publish it. John, it is up to you. This is a follow-on
evaluation
> of
> something we did a few years back, and MET/you have been very
helpful.
> You
> also may have some additions. Just let us know,
>
> Buck
>
>
> -----Original Message-----
> From: Serra, Mr. Efren, Contractor, Code 7531
> <efren.serra.ctr at nrlmry.navy.mil>
> Sent: Tuesday, November 17, 2020 3:28 PM
> To: met_help at ucar.edu
> Cc: Sampson, Mr. Buck <Buck.Sampson at nrlmry.navy.mil>
> Subject: RE: [rt.rap.ucar.edu #97396] MET - distribution of forecast
> values
> for non-events
>
> John - Thanks to your support and contributions we have been making
steady
> progress. We are now creating correct reliability, discrimination
and ROC
> diagram plots.
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Tuesday, November 17, 2020 9:51 AM
> To: Serra, Mr. Efren, Contractor, Code 7531 <
> efren.serra.ctr at nrlmry.navy.mil>
> Subject: [rt.rap.ucar.edu #97396] MET - distribution of forecast
values
> for
> non-events
>
> Efren,
>
> I realize I never responded. Yes, your logic makes sense to me. Did
that
> approach result in the expected resulting plot?
>
> I realize that we should add the discrimination diagram as an option
> through
> METviewer. So I'll forward this to the METviewer/METplotpy
developers.
>
> Thanks,
> John
>
>
------------------------------------------------
More information about the Met_help
mailing list