[Met_help] [rt.rap.ucar.edu #91964] History for brier score eqn
Randy Bullock via RT
met_help at ucar.edu
Wed Oct 16 11:58:33 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
I'm trying to resolve the differences in Brier score when I calculate it
myself and when I use grid_stat.
I think it has something to do with binning.
Going back to Appendix C.3 of the users guide, I see
The Brier score is the mean squared probability error. In MET, the Brier
Score (BS) is calculated from the nx2 contingency table via the following
equation:
BS = 1/T[n.1(p1 − 1)2 + n.0p0]
I think p1 represents the forecast probability when the event occurs and p0 is
the forecast probability when the event doesn't occur. Assuming that is
correct, I have 2 concerns
1. I don't think p1 is the same as p1 in Table C.2. That's a little
confusing. Moreover, p0 doesn't exist in Table C.2.
2. Shouldn't p0 be squared just as (p1-1)?
Understanding this might help me understand this other sentence in the
Users Guide:
This equation is equivalent when the midpoints of the binned probability
values are used as the pi ,
Then, maybe I can explain the differences in Brier score.
Dave
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: brier score eqn
From: John Halley Gotway
Time: Fri Sep 13 16:52:06 2019
Hi Dave,
It looks like you still haven't received a response from us on this
ticket! Tressa Fowler, who would normally answer these types of
questions,
recently retired.
But we did pass it along to Barb Brown and Eric Gilleland, and they
are now
both CC'ed on this ticket.
Here's the latest response from Barb, and I've attached the
attachments she
sent to this email. Looks like we need to update the MET
documentation for
the next release.
Thanks for pointing this out!
John
Hi all,
I think I figured this out, but it's a little difficult to show the
correct
equation in a text email. I created the attached Word/pdf document to
summarize the end result. (Perhaps the equation can be embedded in the
text?).
It looks like MET code is fine, but the documentation is incorrect.
The
formulation is different from what most people do which is why it took
me
awhile to figure it out; it's actually pretty clever but I had to re-
learn
it. Please let me know if you see anything wrong with my response.
Also let me know how you'd like to proceed in sending the response out
(since I don't have the original email from Dave to respond to). If
one of
you'd like to cc me on an interim response, perhaps I can follow the
process Tara suggested.
Barb
On Mon, Sep 9, 2019 at 10:23 AM David Ahijevych via RT
<met_help at ucar.edu>
wrote:
>
> Mon Sep 09 10:23:00 2019: Request 91964 was acted upon.
> Transaction: Ticket created by ahijevyc
> Queue: met_help
> Subject: brier score eqn
> Owner: Nobody
> Requestors: ahijevyc at ucar.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91964 >
>
>
> I'm trying to resolve the differences in Brier score when I
calculate it
> myself and when I use grid_stat.
> I think it has something to do with binning.
> Going back to Appendix C.3 of the users guide, I see
>
> The Brier score is the mean squared probability error. In MET, the
Brier
> Score (BS) is calculated from the nx2 contingency table via the
following
> equation:
> BS = 1/T[n.1(p1 − 1)2 + n.0p0]
>
>
> I think p1 represents the forecast probability when the event occurs
and
> p0 is
> the forecast probability when the event doesn't occur. Assuming
that is
> correct, I have 2 concerns
>
> 1. I don't think p1 is the same as p1 in Table C.2. That's a
little
> confusing. Moreover, p0 doesn't exist in Table C.2.
> 2. Shouldn't p0 be squared just as (p1-1)?
>
>
> Understanding this might help me understand this other sentence in
the
> Users Guide:
>
> This equation is equivalent when the midpoints of the binned
probability
> values are used as the pi ,
>
>
> Then, maybe I can explain the differences in Brier score.
>
> Dave
>
>
------------------------------------------------
Subject: brier score eqn
From: John Halley Gotway
Time: Fri Sep 13 16:56:08 2019
FYI, I wrote up a GitHub issue to make sure we correct this in the
met-9.0
documentation:
https://github.com/NCAR/MET/issues/1196
Thanks,
John
On Fri, Sep 13, 2019 at 4:51 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Hi Dave,
>
> It looks like you still haven't received a response from us on this
> ticket! Tressa Fowler, who would normally answer these types of
questions,
> recently retired.
>
> But we did pass it along to Barb Brown and Eric Gilleland, and they
are
> now both CC'ed on this ticket.
>
> Here's the latest response from Barb, and I've attached the
attachments
> she sent to this email. Looks like we need to update the MET
documentation
> for the next release.
>
> Thanks for pointing this out!
>
> John
>
>
> Hi all,
>
> I think I figured this out, but it's a little difficult to show the
> correct equation in a text email. I created the attached Word/pdf
document
> to summarize the end result. (Perhaps the equation can be embedded
in the
> text?).
>
> It looks like MET code is fine, but the documentation is incorrect.
The
> formulation is different from what most people do which is why it
took me
> awhile to figure it out; it's actually pretty clever but I had to
re-learn
> it. Please let me know if you see anything wrong with my response.
>
> Also let me know how you'd like to proceed in sending the response
out
> (since I don't have the original email from Dave to respond to). If
one of
> you'd like to cc me on an interim response, perhaps I can follow the
> process Tara suggested.
>
> Barb
>
> On Mon, Sep 9, 2019 at 10:23 AM David Ahijevych via RT
<met_help at ucar.edu>
> wrote:
>
>>
>> Mon Sep 09 10:23:00 2019: Request 91964 was acted upon.
>> Transaction: Ticket created by ahijevyc
>> Queue: met_help
>> Subject: brier score eqn
>> Owner: Nobody
>> Requestors: ahijevyc at ucar.edu
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91964 >
>>
>>
>> I'm trying to resolve the differences in Brier score when I
calculate it
>> myself and when I use grid_stat.
>> I think it has something to do with binning.
>> Going back to Appendix C.3 of the users guide, I see
>>
>> The Brier score is the mean squared probability error. In MET, the
Brier
>> Score (BS) is calculated from the nx2 contingency table via the
following
>> equation:
>> BS = 1/T[n.1(p1 − 1)2 + n.0p0]
>>
>>
>> I think p1 represents the forecast probability when the event
occurs and
>> p0 is
>> the forecast probability when the event doesn't occur. Assuming
that is
>> correct, I have 2 concerns
>>
>> 1. I don't think p1 is the same as p1 in Table C.2. That's a
little
>> confusing. Moreover, p0 doesn't exist in Table C.2.
>> 2. Shouldn't p0 be squared just as (p1-1)?
>>
>>
>> Understanding this might help me understand this other sentence in
the
>> Users Guide:
>>
>> This equation is equivalent when the midpoints of the binned
probability
>> values are used as the pi ,
>>
>>
>> Then, maybe I can explain the differences in Brier score.
>>
>> Dave
>>
>>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #91964] brier score eqn
From: Eric Gilleland
Time: Sun Sep 15 11:37:45 2019
Thanks, John. I guess Barb and I were under the impression that
replying all to that email would go to David, but no?
On Fri, Sep 13, 2019 at 4:56 PM John Halley Gotway via RT
<met_help at ucar.edu> wrote:
>
> FYI, I wrote up a GitHub issue to make sure we correct this in the
met-9.0
> documentation:
> https://github.com/NCAR/MET/issues/1196
>
> Thanks,
> John
>
> On Fri, Sep 13, 2019 at 4:51 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
>
> > Hi Dave,
> >
> > It looks like you still haven't received a response from us on
this
> > ticket! Tressa Fowler, who would normally answer these types of
questions,
> > recently retired.
> >
> > But we did pass it along to Barb Brown and Eric Gilleland, and
they are
> > now both CC'ed on this ticket.
> >
> > Here's the latest response from Barb, and I've attached the
attachments
> > she sent to this email. Looks like we need to update the MET
documentation
> > for the next release.
> >
> > Thanks for pointing this out!
> >
> > John
> >
> >
> > Hi all,
> >
> > I think I figured this out, but it's a little difficult to show
the
> > correct equation in a text email. I created the attached Word/pdf
document
> > to summarize the end result. (Perhaps the equation can be embedded
in the
> > text?).
> >
> > It looks like MET code is fine, but the documentation is
incorrect. The
> > formulation is different from what most people do which is why it
took me
> > awhile to figure it out; it's actually pretty clever but I had to
re-learn
> > it. Please let me know if you see anything wrong with my
response.
> >
> > Also let me know how you'd like to proceed in sending the response
out
> > (since I don't have the original email from Dave to respond to).
If one of
> > you'd like to cc me on an interim response, perhaps I can follow
the
> > process Tara suggested.
> >
> > Barb
> >
> > On Mon, Sep 9, 2019 at 10:23 AM David Ahijevych via RT
<met_help at ucar.edu>
> > wrote:
> >
> >>
> >> Mon Sep 09 10:23:00 2019: Request 91964 was acted upon.
> >> Transaction: Ticket created by ahijevyc
> >> Queue: met_help
> >> Subject: brier score eqn
> >> Owner: Nobody
> >> Requestors: ahijevyc at ucar.edu
> >> Status: new
> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91964 >
> >>
> >>
> >> I'm trying to resolve the differences in Brier score when I
calculate it
> >> myself and when I use grid_stat.
> >> I think it has something to do with binning.
> >> Going back to Appendix C.3 of the users guide, I see
> >>
> >> The Brier score is the mean squared probability error. In MET,
the Brier
> >> Score (BS) is calculated from the nx2 contingency table via the
following
> >> equation:
> >> BS = 1/T[n.1(p1 − 1)2 + n.0p0]
> >>
> >>
> >> I think p1 represents the forecast probability when the event
occurs and
> >> p0 is
> >> the forecast probability when the event doesn't occur. Assuming
that is
> >> correct, I have 2 concerns
> >>
> >> 1. I don't think p1 is the same as p1 in Table C.2. That's a
little
> >> confusing. Moreover, p0 doesn't exist in Table C.2.
> >> 2. Shouldn't p0 be squared just as (p1-1)?
> >>
> >>
> >> Understanding this might help me understand this other sentence
in the
> >> Users Guide:
> >>
> >> This equation is equivalent when the midpoints of the binned
probability
> >> values are used as the pi ,
> >>
> >>
> >> Then, maybe I can explain the differences in Brier score.
> >>
> >> Dave
> >>
> >>
>
------------------------------------------------
Subject: brier score eqn
From: David Ahijevych
Time: Mon Sep 16 14:12:33 2019
Hi Barb, Eric and John!
Thanks for the updated Brier score equation. It makes more sense.
I'll
let you know if I have any more questions.
Dave
On Sun, Sep 15, 2019 at 11:37 AM Eric Gilleland via RT
<met_help at ucar.edu>
wrote:
> Thanks, John. I guess Barb and I were under the impression that
> replying all to that email would go to David, but no?
>
> On Fri, Sep 13, 2019 at 4:56 PM John Halley Gotway via RT
> <met_help at ucar.edu> wrote:
> >
> > FYI, I wrote up a GitHub issue to make sure we correct this in the
> met-9.0
> > documentation:
> > https://github.com/NCAR/MET/issues/1196
> >
> > Thanks,
> > John
> >
> > On Fri, Sep 13, 2019 at 4:51 PM John Halley Gotway
<johnhg at ucar.edu>
> wrote:
> >
> > > Hi Dave,
> > >
> > > It looks like you still haven't received a response from us on
this
> > > ticket! Tressa Fowler, who would normally answer these types of
> questions,
> > > recently retired.
> > >
> > > But we did pass it along to Barb Brown and Eric Gilleland, and
they are
> > > now both CC'ed on this ticket.
> > >
> > > Here's the latest response from Barb, and I've attached the
attachments
> > > she sent to this email. Looks like we need to update the MET
> documentation
> > > for the next release.
> > >
> > > Thanks for pointing this out!
> > >
> > > John
> > >
> > >
> > > Hi all,
> > >
> > > I think I figured this out, but it's a little difficult to show
the
> > > correct equation in a text email. I created the attached
Word/pdf
> document
> > > to summarize the end result. (Perhaps the equation can be
embedded in
> the
> > > text?).
> > >
> > > It looks like MET code is fine, but the documentation is
incorrect.
> The
> > > formulation is different from what most people do which is why
it took
> me
> > > awhile to figure it out; it's actually pretty clever but I had
to
> re-learn
> > > it. Please let me know if you see anything wrong with my
response.
> > >
> > > Also let me know how you'd like to proceed in sending the
response out
> > > (since I don't have the original email from Dave to respond to).
If
> one of
> > > you'd like to cc me on an interim response, perhaps I can follow
the
> > > process Tara suggested.
> > >
> > > Barb
> > >
> > > On Mon, Sep 9, 2019 at 10:23 AM David Ahijevych via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > >>
> > >> Mon Sep 09 10:23:00 2019: Request 91964 was acted upon.
> > >> Transaction: Ticket created by ahijevyc
> > >> Queue: met_help
> > >> Subject: brier score eqn
> > >> Owner: Nobody
> > >> Requestors: ahijevyc at ucar.edu
> > >> Status: new
> > >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=91964
> >
> > >>
> > >>
> > >> I'm trying to resolve the differences in Brier score when I
calculate
> it
> > >> myself and when I use grid_stat.
> > >> I think it has something to do with binning.
> > >> Going back to Appendix C.3 of the users guide, I see
> > >>
> > >> The Brier score is the mean squared probability error. In MET,
the
> Brier
> > >> Score (BS) is calculated from the nx2 contingency table via the
> following
> > >> equation:
> > >> BS = 1/T[n.1(p1 − 1)2 + n.0p0]
> > >>
> > >>
> > >> I think p1 represents the forecast probability when the event
occurs
> and
> > >> p0 is
> > >> the forecast probability when the event doesn't occur.
Assuming that
> is
> > >> correct, I have 2 concerns
> > >>
> > >> 1. I don't think p1 is the same as p1 in Table C.2. That's a
little
> > >> confusing. Moreover, p0 doesn't exist in Table C.2.
> > >> 2. Shouldn't p0 be squared just as (p1-1)?
> > >>
> > >>
> > >> Understanding this might help me understand this other sentence
in the
> > >> Users Guide:
> > >>
> > >> This equation is equivalent when the midpoints of the binned
> probability
> > >> values are used as the pi ,
> > >>
> > >>
> > >> Then, maybe I can explain the differences in Brier score.
> > >>
> > >> Dave
> > >>
> > >>
> >
>
>
>
------------------------------------------------
More information about the Met_help
mailing list