[Met_help] [rt.rap.ucar.edu #96132] History for About the BSS of ensemble forecast

[John Halley Gotway via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

To whom maybe concerned，
     I want to use the MET8.1 to calculate the Brier Skill Score(BSS) of precipitation for my ensemble forecasts. Could I use the BS_cntl as the reference , How to set BS_reference as BS of control experiment of ensemble system ?
     Thank you very much!


Anwei


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: About the BSS of ensemble forecast
From: Minna Win
Time: Thu Aug 06 10:04:53 2020

Hello Anwei,

It looks like you have questions on setting up your BSS.  I'm
assigning
this issue to John Halley Gotway.  Please allow a few days for a full
response.

Regards,
Minna
---------------
Minna Win
National Center for Atmospheric Research
Developmental Testbed Center
Phone: 303-497-8423
Fax:   303-497-8401



On Thu, Aug 6, 2020 at 12:10 AM 赖 安伟 via RT <met_help at ucar.edu> wrote:

>
> Thu Aug 06 00:09:47 2020: Request 96132 was acted upon.
> Transaction: Ticket created by laianwei at hotmail.com
>        Queue: met_help
>      Subject: About the BSS of ensemble forecast
>        Owner: Nobody
>   Requestors: laianwei at hotmail.com
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96132 >
>
>
> To whom maybe concerned，
>      I want to use the MET8.1 to calculate the Brier Skill
Score(BSS) of
> precipitation for my ensemble forecasts. Could I use the BS_cntl as
the
> reference , How to set BS_reference as BS of control experiment of
ensemble
> system ?
>      Thank you very much!
>
>
> Anwei
>
>

------------------------------------------------
Subject: About the BSS of ensemble forecast
From: John Halley Gotway
Time: Fri Aug 07 11:03:33 2020

Anwei,

I see you have a question about the reference used when computing the
Brier
Skill Score statistic in MET. The BSS statistics is included in the
PSTD line type in MET when verifying probabilistic forecasts. It is
written
by the Point-Stat or Grid-Stat tools when verifying against point obs
or
gridded analyses, respectively. And there are actually two versions of
BSS
in that line:
  - BSS is computed relative to an external climatology.
  - BSS_SMPL is computed relative to the "sample climatology" of the
observations in the current run.

I assume you're asking about BSS, and defining the reference for it.
In
that PSTD line type, the BRIER column is the Brier Score of the
forecast
and the BRIERCL column is the Brier Score of the reference
climatology. And
then BSS is computed as 1 - BRIER / BRIERCL. So you'd like to know how
to
specify the data to be used when computing that BRIERCL column of
output.

In MET, when verifying a probabilistic forecast, the reference is
defined
using a climatological mean and standard deviation. And the input for
that
data is set in the Point-Stat and Grid-Stat configuration files using
the
"climo_mean" and "climo_stdev" dictionaries.

For each observation value, MET has the corresponding forecast
probability
value. It uses the climo mean and standard deviation (assuming
normality)
to compute the climatological probability of that observation value.
The
forecast BRIER score is computed for those forecast probabilities. And
the
climo BRIERCL score is computed for those climatological
probabilities.

So that's how you'd congifure/compute BSS within the MET tools.

Alternatively, if you're using the METviewer database and display
tool. You
could plot BRIER score for two different models and then derive a
skill
score (e.g. Brier Skill Score) by adding a "derived curve". The
derivation
logic in METviewer is simple: 1 - fcst/ref, where you define which
model is
the forecast and which is the reference. If you don't know about
METviewer,
then this won't make much sense.

Hope that helps.

Thanks,
John Halley Gotway

On Thu, Aug 6, 2020 at 10:04 AM Minna Win via RT <met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96132 >
>
> Hello Anwei,
>
> It looks like you have questions on setting up your BSS.  I'm
assigning
> this issue to John Halley Gotway.  Please allow a few days for a
full
> response.
>
> Regards,
> Minna
> ---------------
> Minna Win
> National Center for Atmospheric Research
> Developmental Testbed Center
> Phone: 303-497-8423
> Fax:   303-497-8401
>
>
>
> On Thu, Aug 6, 2020 at 12:10 AM 赖 安伟 via RT <met_help at ucar.edu>
wrote:
>
> >
> > Thu Aug 06 00:09:47 2020: Request 96132 was acted upon.
> > Transaction: Ticket created by laianwei at hotmail.com
> >        Queue: met_help
> >      Subject: About the BSS of ensemble forecast
> >        Owner: Nobody
> >   Requestors: laianwei at hotmail.com
> >       Status: new
> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96132 >
> >
> >
> > To whom maybe concerned，
> >      I want to use the MET8.1 to calculate the Brier Skill
Score(BSS) of
> > precipitation for my ensemble forecasts. Could I use the BS_cntl
as the
> > reference , How to set BS_reference as BS of control experiment of
> ensemble
> > system ?
> >      Thank you very much!
> >
> >
> > Anwei
> >
> >
>
>

------------------------------------------------