[Met_help] [rt.rap.ucar.edu #92550] History for MET User's guide Table 7.13 error?
John Halley Gotway via RT
met_help at ucar.edu
Fri Oct 11 16:43:29 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi MET folks,
Just wanted to let you know that I think Table 7.13 on page 197 of the MET
8.1.1 user's guide is wrong. This table is for the ROC, but your
description for PODY_i and POFD_i seem to be the same as in Table 7.10,
which is correct for reliability (as in reliability diagrams). For ROC
statistics, the proper definition is looking to see if a value is greater
than a threshold (basically a contingency table definition). There should
be no need to look in both the ith and (i+1)th probability bins.
Thanks for looking into this, and please let me know if you have any
questions.
Best,
Craig
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: MET User's guide Table 7.13 error?
From: John Halley Gotway
Time: Thu Oct 10 18:40:56 2019
Hi Craig,
Thanks for letting us know about this issue. Am I correct in thinking
that
the problem is the "Description" of column numbers 28 and 29, for
PODY_i
and POFD_i?
They currently state:
PODY_i = Probability of detecting yes when forecast is between the ith
and
i+1th probability thresholds (repeated)
POFD_i = Probability of false detection when forecast is between the
ith
and i+1th probability thresholds (repeated)
And I suspect that the description should be changed to:
PODY_i = Probability of detecting yes when forecast is less than the
i+1th
probability thresholds (repeated)
POFD_i = Probability of false detection when forecast is less than the
i+1th probability thresholds (repeated)
MET verifies probabilities by constructing an Nx2 contingency table
(whose
counts are listed in the PCT line type). The points of the ROC curve
(PRC
line type) are created by converting that Nx2 contingency table into N
2x2
contingency tables. And for each of those, we compute PODY and POFD
to get
the points of the ROC curve. Does that sound right to you?
Please let me know if you find anything else that should be changed.
Thanks,
John Halley Gotway
On Wed, Oct 9, 2019 at 4:13 PM Craig Schwartz via RT
<met_help at ucar.edu>
wrote:
>
> Wed Oct 09 16:13:36 2019: Request 92550 was acted upon.
> Transaction: Ticket created by schwartz at ucar.edu
> Queue: met_help
> Subject: MET User's guide Table 7.13 error?
> Owner: Nobody
> Requestors: schwartz at ucar.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92550 >
>
>
> Hi MET folks,
>
> Just wanted to let you know that I think Table 7.13 on page 197 of
the MET
> 8.1.1 user's guide is wrong. This table is for the ROC, but your
> description for PODY_i and POFD_i seem to be the same as in Table
7.10,
> which is correct for reliability (as in reliability diagrams). For
ROC
> statistics, the proper definition is looking to see if a value is
greater
> than a threshold (basically a contingency table definition). There
should
> be no need to look in both the ith and (i+1)th probability bins.
>
> Thanks for looking into this, and please let me know if you have any
> questions.
>
> Best,
> Craig
>
>
------------------------------------------------
Subject: MET User's guide Table 7.13 error?
From: Julie Prestopnik
Time: Thu Oct 10 19:46:26 2019
Hi Craig.
Thank you for letting us know. I'll go ahead and assign this ticket
to
John Halley Gotway who is best equipped to verify this information and
confirm.
Julie
On Wed, Oct 9, 2019 at 4:13 PM Craig Schwartz via RT
<met_help at ucar.edu>
wrote:
>
> Wed Oct 09 16:13:36 2019: Request 92550 was acted upon.
> Transaction: Ticket created by schwartz at ucar.edu
> Queue: met_help
> Subject: MET User's guide Table 7.13 error?
> Owner: Nobody
> Requestors: schwartz at ucar.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92550 >
>
>
> Hi MET folks,
>
> Just wanted to let you know that I think Table 7.13 on page 197 of
the MET
> 8.1.1 user's guide is wrong. This table is for the ROC, but your
> description for PODY_i and POFD_i seem to be the same as in Table
7.10,
> which is correct for reliability (as in reliability diagrams). For
ROC
> statistics, the proper definition is looking to see if a value is
greater
> than a threshold (basically a contingency table definition). There
should
> be no need to look in both the ith and (i+1)th probability bins.
>
> Thanks for looking into this, and please let me know if you have any
> questions.
>
> Best,
> Craig
>
>
--
Julie Prestopnik
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: MET User's guide Table 7.13 error?
From: Craig Schwartz
Time: Fri Oct 11 16:30:36 2019
Hi John,
Yes, it was an issue with the "Description" column.
I believe it should be changed to:
PODY_i = Probability of detecting yes when forecast is greater than
the ith
probability thresholds (repeated)
POFD_i = Probability of false detection when forecast is greater than
the
ith probability thresholds (repeated)
It's possible, depending on the exact way MET figures out the
probabilistic
thresholds, that the above should be "ith + 1" instead of "ith", but
I'm
pretty sure it should be >= rather than < for the condition.
Completely agree that the PRC line type is derivable from the PCT line
type, and I believe a colleague in MMM verified that.
Thanks for having a look, and please let me know if you have any other
questions.
Best,
Craig
On Thu, Oct 10, 2019 at 6:41 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Hi Craig,
>
> Thanks for letting us know about this issue. Am I correct in
thinking that
> the problem is the "Description" of column numbers 28 and 29, for
PODY_i
> and POFD_i?
>
> They currently state:
> PODY_i = Probability of detecting yes when forecast is between the
ith and
> i+1th probability thresholds (repeated)
> POFD_i = Probability of false detection when forecast is between the
ith
> and i+1th probability thresholds (repeated)
>
> And I suspect that the description should be changed to:
> PODY_i = Probability of detecting yes when forecast is less than the
i+1th
> probability thresholds (repeated)
> POFD_i = Probability of false detection when forecast is less than
the
> i+1th probability thresholds (repeated)
>
> MET verifies probabilities by constructing an Nx2 contingency table
(whose
> counts are listed in the PCT line type). The points of the ROC
curve (PRC
> line type) are created by converting that Nx2 contingency table into
N 2x2
> contingency tables. And for each of those, we compute PODY and POFD
to get
> the points of the ROC curve. Does that sound right to you?
>
> Please let me know if you find anything else that should be changed.
>
> Thanks,
> John Halley Gotway
>
> On Wed, Oct 9, 2019 at 4:13 PM Craig Schwartz via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Wed Oct 09 16:13:36 2019: Request 92550 was acted upon.
> > Transaction: Ticket created by schwartz at ucar.edu
> > Queue: met_help
> > Subject: MET User's guide Table 7.13 error?
> > Owner: Nobody
> > Requestors: schwartz at ucar.edu
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92550 >
> >
> >
> > Hi MET folks,
> >
> > Just wanted to let you know that I think Table 7.13 on page 197 of
the
> MET
> > 8.1.1 user's guide is wrong. This table is for the ROC, but your
> > description for PODY_i and POFD_i seem to be the same as in Table
7.10,
> > which is correct for reliability (as in reliability diagrams).
For ROC
> > statistics, the proper definition is looking to see if a value is
greater
> > than a threshold (basically a contingency table definition).
There
> should
> > be no need to look in both the ith and (i+1)th probability bins.
> >
> > Thanks for looking into this, and please let me know if you have
any
> > questions.
> >
> > Best,
> > Craig
> >
> >
>
>
------------------------------------------------
Subject: MET User's guide Table 7.13 error?
From: John Halley Gotway
Time: Fri Oct 11 16:43:22 2019
Thanks Craig. I made those updates.
John
On Fri, Oct 11, 2019 at 4:31 PM Craig Schwartz via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92550 >
>
> Hi John,
>
> Yes, it was an issue with the "Description" column.
>
> I believe it should be changed to:
>
> PODY_i = Probability of detecting yes when forecast is greater than
the ith
> probability thresholds (repeated)
> POFD_i = Probability of false detection when forecast is greater
than the
> ith probability thresholds (repeated)
>
> It's possible, depending on the exact way MET figures out the
probabilistic
> thresholds, that the above should be "ith + 1" instead of "ith", but
I'm
> pretty sure it should be >= rather than < for the condition.
>
> Completely agree that the PRC line type is derivable from the PCT
line
> type, and I believe a colleague in MMM verified that.
>
> Thanks for having a look, and please let me know if you have any
other
> questions.
>
> Best,
> Craig
>
> On Thu, Oct 10, 2019 at 6:41 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Hi Craig,
> >
> > Thanks for letting us know about this issue. Am I correct in
thinking
> that
> > the problem is the "Description" of column numbers 28 and 29, for
PODY_i
> > and POFD_i?
> >
> > They currently state:
> > PODY_i = Probability of detecting yes when forecast is between the
ith
> and
> > i+1th probability thresholds (repeated)
> > POFD_i = Probability of false detection when forecast is between
the ith
> > and i+1th probability thresholds (repeated)
> >
> > And I suspect that the description should be changed to:
> > PODY_i = Probability of detecting yes when forecast is less than
the
> i+1th
> > probability thresholds (repeated)
> > POFD_i = Probability of false detection when forecast is less than
the
> > i+1th probability thresholds (repeated)
> >
> > MET verifies probabilities by constructing an Nx2 contingency
table
> (whose
> > counts are listed in the PCT line type). The points of the ROC
curve
> (PRC
> > line type) are created by converting that Nx2 contingency table
into N
> 2x2
> > contingency tables. And for each of those, we compute PODY and
POFD to
> get
> > the points of the ROC curve. Does that sound right to you?
> >
> > Please let me know if you find anything else that should be
changed.
> >
> > Thanks,
> > John Halley Gotway
> >
> > On Wed, Oct 9, 2019 at 4:13 PM Craig Schwartz via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > Wed Oct 09 16:13:36 2019: Request 92550 was acted upon.
> > > Transaction: Ticket created by schwartz at ucar.edu
> > > Queue: met_help
> > > Subject: MET User's guide Table 7.13 error?
> > > Owner: Nobody
> > > Requestors: schwartz at ucar.edu
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92550
> >
> > >
> > >
> > > Hi MET folks,
> > >
> > > Just wanted to let you know that I think Table 7.13 on page 197
of the
> > MET
> > > 8.1.1 user's guide is wrong. This table is for the ROC, but
your
> > > description for PODY_i and POFD_i seem to be the same as in
Table 7.10,
> > > which is correct for reliability (as in reliability diagrams).
For ROC
> > > statistics, the proper definition is looking to see if a value
is
> greater
> > > than a threshold (basically a contingency table definition).
There
> > should
> > > be no need to look in both the ith and (i+1)th probability bins.
> > >
> > > Thanks for looking into this, and please let me know if you have
any
> > > questions.
> > >
> > > Best,
> > > Craig
> > >
> > >
> >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list