[Met_help] [rt.rap.ucar.edu #92550] History for MET User's guide Table 7.13 error?

[John Halley Gotway via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hi MET folks,

Just wanted to let you know that I think Table 7.13 on page 197 of the MET
8.1.1 user's guide is wrong.  This table is for the ROC, but your
description for PODY_i and POFD_i seem to be the same as in Table 7.10,
which is correct for reliability (as in reliability diagrams).  For ROC
statistics, the proper definition is looking to see if a value is greater
than a threshold (basically a contingency table definition).  There should
be no need to look in both the ith and (i+1)th probability bins.

Thanks for looking into this, and please let me know if you have any
questions.

Best,
Craig


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: MET User's guide Table 7.13 error?
From: John Halley Gotway
Time: Thu Oct 10 18:40:56 2019

Hi Craig,

Thanks for letting us know about this issue.  Am I correct in thinking
that
the problem is the "Description" of column numbers 28 and 29, for
PODY_i
and POFD_i?

They currently state:
PODY_i = Probability of detecting yes when forecast is between the ith
and
i+1th probability thresholds (repeated)
POFD_i = Probability of false detection when forecast is between the
ith
and i+1th probability thresholds (repeated)

And I suspect that the description should be changed to:
PODY_i = Probability of detecting yes when forecast is less than the
i+1th
probability thresholds (repeated)
POFD_i = Probability of false detection when forecast is less than the
i+1th probability thresholds (repeated)

MET verifies probabilities by constructing an Nx2 contingency table
(whose
counts are listed in the PCT line type).  The points of the ROC curve
(PRC
line type) are created by converting that Nx2 contingency table into N
2x2
contingency tables.  And for each of those, we compute PODY and POFD
to get
the points of the ROC curve.  Does that sound right to you?

Please let me know if you find anything else that should be changed.

Thanks,
John Halley Gotway

On Wed, Oct 9, 2019 at 4:13 PM Craig Schwartz via RT
<met_help at ucar.edu>
wrote:

>
> Wed Oct 09 16:13:36 2019: Request 92550 was acted upon.
> Transaction: Ticket created by schwartz at ucar.edu
>        Queue: met_help
>      Subject: MET User's guide Table 7.13 error?
>        Owner: Nobody
>   Requestors: schwartz at ucar.edu
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92550 >
>
>
> Hi MET folks,
>
> Just wanted to let you know that I think Table 7.13 on page 197 of
the MET
> 8.1.1 user's guide is wrong.  This table is for the ROC, but your
> description for PODY_i and POFD_i seem to be the same as in Table
7.10,
> which is correct for reliability (as in reliability diagrams).  For
ROC
> statistics, the proper definition is looking to see if a value is
greater
> than a threshold (basically a contingency table definition).  There
should
> be no need to look in both the ith and (i+1)th probability bins.
>
> Thanks for looking into this, and please let me know if you have any
> questions.
>
> Best,
> Craig
>
>

------------------------------------------------
Subject: MET User's guide Table 7.13 error?
From: Julie Prestopnik
Time: Thu Oct 10 19:46:26 2019

Hi Craig.

Thank you for letting us know.  I'll go ahead and assign this ticket
to
John Halley Gotway who is best equipped to verify this information and
confirm.

Julie

On Wed, Oct 9, 2019 at 4:13 PM Craig Schwartz via RT
<met_help at ucar.edu>
wrote:

>
> Wed Oct 09 16:13:36 2019: Request 92550 was acted upon.
> Transaction: Ticket created by schwartz at ucar.edu
>        Queue: met_help
>      Subject: MET User's guide Table 7.13 error?
>        Owner: Nobody
>   Requestors: schwartz at ucar.edu
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92550 >
>
>
> Hi MET folks,
>
> Just wanted to let you know that I think Table 7.13 on page 197 of
the MET
> 8.1.1 user's guide is wrong.  This table is for the ROC, but your
> description for PODY_i and POFD_i seem to be the same as in Table
7.10,
> which is correct for reliability (as in reliability diagrams).  For
ROC
> statistics, the proper definition is looking to see if a value is
greater
> than a threshold (basically a contingency table definition).  There
should
> be no need to look in both the ith and (i+1)th probability bins.
>
> Thanks for looking into this, and please let me know if you have any
> questions.
>
> Best,
> Craig
>
>

--
Julie Prestopnik
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu

My working day may not be your working day.  Please do not feel
obliged to
reply to this email outside of your normal working hours.

------------------------------------------------