[Met_help] [rt.rap.ucar.edu #42151] History for Brier scores

Wed Nov 17 08:37:32 MST 2010

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Dear help team,

We are new to MET and feel a little lost. Unfortunately we could not attend a 
tutorial at NCAR so far, but we are eager to use the application.

We want to calculate Brier scores, uncertainty, reliability and resolution. 

Our data consists of an ensemble of precipitation forecasts and their 
corresponding gauges observations. We are interested in point verification 
against gauges. It is a statistical forecast method, not output from a model 
grid as WRF.

Our data consists thus of a table where in the first column we have the 
observation and the in rest of the columns the forecast from each ensemble 
member.
We want to verify the probability of exceeding the lower 30% and the 50% 
quantiles. Our reference forecast is the in-sample observed climatology for each 
station.

At the present stage it is not clear for us how to input this data into MET and 
what stages to follow. Could there be an example for us to look at or could you 
help us telling us steps to follow within the application?

Thanks in advance.

Dorita

----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Re: [rt.rap.ucar.edu #42151] Brier scores
From: John Halley Gotway
Time: Tue Nov 16 11:07:16 2010

Dorita,

MET is really designed to process gridded output from NWP models.  The
creation of the matched pair data (pairs of forecast and observation
values) is where MET does a great deal of it's work.  Based
on your description, it sounds like that task of creating matched
pairs is already done for your data.

I don't believe there would be a straight-forward way of ingesting
your data into MET.  MET computes brier scores, reliability,
resolution, and uncertainty when evaluating probabilistic forecasts.
However, the conversion of ensemble output into probabilities is the
user's responsibility.

Really, I'd suggest making use of R (http://www.r-project.org).  It's
a freely available statistical analysis and graphics tool.  You could
easily read your tabular data into R and make use of
existing packages and tools to evaluate it.

Hope that helps.

Thanks,
John Halley Gotway
met_help at ucar.edu

On 11/16/2010 04:24 AM, RAL HelpDesk {for Dorita Rostkier-Edelstein}
wrote:
>
> Tue Nov 16 04:24:38 2010: Request 42151 was acted upon.
> Transaction: Ticket created by drostkier at yahoo.com
>        Queue: met_help
>      Subject: Brier scores
>        Owner: Nobody
>   Requestors: drostkier at yahoo.com
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=42151 >
>
>
> Dear help team,
>
> We are new to MET and feel a little lost. Unfortunately we could not
attend a
> tutorial at NCAR so far, but we are eager to use the application.
>
> We want to calculate Brier scores, uncertainty, reliability and
resolution.
>
> Our data consists of an ensemble of precipitation forecasts and
their
> corresponding gauges observations. We are interested in point
verification
> against gauges. It is a statistical forecast method, not output from
a model
> grid as WRF.
>
> Our data consists thus of a table where in the first column we have
the
> observation and the in rest of the columns the forecast from each
ensemble
> member.
> We want to verify the probability of exceeding the lower 30% and the
50%
> quantiles. Our reference forecast is the in-sample observed
climatology for each
> station.
>
> At the present stage it is not clear for us how to input this data
into MET and
> what stages to follow. Could there be an example for us to look at
or could you
> help us telling us steps to follow within the application?
>
> Thanks in advance.
>
> Dorita

------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #42151] Brier scores
From: Tressa Fowler
Time: Tue Nov 16 11:14:46 2010

Hi Dorita,

I second John's recommendation to use the R software package. It is
free and available for download on the internet. To do forecast
verification, download the 'add on' verification package and the
associated documentation. Brier score along with decompositions are
included in that package.

Please let us know if you have further questions.

Regards,

Tressa

On Nov 16, 2010, at 11:07 AM, RAL HelpDesk {for John Halley Gotway}
wrote:

> Dorita,
>
> MET is really designed to process gridded output from NWP models.
> The creation of the matched pair data (pairs of forecast and
> observation values) is where MET does a great deal of it's work.
> Based
> on your description, it sounds like that task of creating matched
> pairs is already done for your data.
>
> I don't believe there would be a straight-forward way of ingesting
> your data into MET.  MET computes brier scores, reliability,
> resolution, and uncertainty when evaluating probabilistic forecasts.
> However, the conversion of ensemble output into probabilities is the
> user's responsibility.
>
> Really, I'd suggest making use of R (http://www.r-project.org).
> It's a freely available statistical analysis and graphics tool.  You
> could easily read your tabular data into R and make use of
> existing packages and tools to evaluate it.
>
> Hope that helps.
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu
>
>
> On 11/16/2010 04:24 AM, RAL HelpDesk {for Dorita Rostkier-Edelstein}
> wrote:
>>
>> Tue Nov 16 04:24:38 2010: Request 42151 was acted upon.
>> Transaction: Ticket created by drostkier at yahoo.com
>>       Queue: met_help
>>     Subject: Brier scores
>>       Owner: Nobody
>>  Requestors: drostkier at yahoo.com
>>      Status: new
>> Ticket <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?
>> id=42151 >
>>
>>
>> Dear help team,
>>
>> We are new to MET and feel a little lost. Unfortunately we could
>> not attend a
>> tutorial at NCAR so far, but we are eager to use the application.
>>
>> We want to calculate Brier scores, uncertainty, reliability and
>> resolution.
>>
>> Our data consists of an ensemble of precipitation forecasts and
their
>> corresponding gauges observations. We are interested in point
>> verification
>> against gauges. It is a statistical forecast method, not output
>> from a model
>> grid as WRF.
>>
>> Our data consists thus of a table where in the first column we have
>> the
>> observation and the in rest of the columns the forecast from each
>> ensemble
>> member.
>> We want to verify the probability of exceeding the lower 30% and
>> the 50%
>> quantiles. Our reference forecast is the in-sample observed
>> climatology for each
>> station.
>>
>> At the present stage it is not clear for us how to input this data
>> into MET and
>> what stages to follow. Could there be an example for us to look at
>> or could you
>> help us telling us steps to follow within the application?
>>
>> Thanks in advance.
>>
>> Dorita

------------------------------------------------
Subject: Brier scores
From: Dorita Rostkier-Edelstein
Time: Tue Nov 16 23:16:44 2010

Hi Tressa and John,

Thanks a lot for your quick reply. As a matter of fact I have used the
R
verification package by Matt a lot. I have a new associate scientist
working
with me and I thought that MET would be easier. He will use R too
then.

Dorita

________________________________
From: RAL HelpDesk {for Tressa Fowler} <met_help at ucar.edu>
To: drostkier at yahoo.com
Cc: pavel.kunin at gmail.com; rostkier at ucar.edu; tressa at ucar.edu
Sent: Tue, November 16, 2010 8:14:47 PM
Subject: Re: [rt.rap.ucar.edu #42151] Brier scores

Hi Dorita,

I second John's recommendation to use the R software package. It is
free and available for download on the internet. To do forecast
verification, download the 'add on' verification package and the
associated documentation. Brier score along with decompositions are
included in that package.

Please let us know if you have further questions.

Regards,

Tressa

On Nov 16, 2010, at 11:07 AM, RAL HelpDesk {for John Halley Gotway}
wrote:

> Dorita,
>
> MET is really designed to process gridded output from NWP models.
> The creation of the matched pair data (pairs of forecast and
> observation values) is where MET does a great deal of it's work.
> Based
> on your description, it sounds like that task of creating matched
> pairs is already done for your data.
>
> I don't believe there would be a straight-forward way of ingesting
> your data into MET.  MET computes brier scores, reliability,
> resolution, and uncertainty when evaluating probabilistic forecasts.
> However, the conversion of ensemble output into probabilities is the
> user's responsibility.
>
> Really, I'd suggest making use of R (http://www.r-project.org).
> It's a freely available statistical analysis and graphics tool.  You
> could easily read your tabular data into R and make use of
> existing packages and tools to evaluate it.
>
> Hope that helps.
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu
>
>
> On 11/16/2010 04:24 AM, RAL HelpDesk {for Dorita Rostkier-Edelstein}
> wrote:
>>
>> Tue Nov 16 04:24:38 2010: Request 42151 was acted upon.
>> Transaction: Ticket created by drostkier at yahoo.com
>>       Queue: met_help
>>     Subject: Brier scores
>>       Owner: Nobody
>>  Requestors: drostkier at yahoo.com
>>      Status: new
>> Ticket <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?
>> id=42151 >
>>
>>
>> Dear help team,
>>
>> We are new to MET and feel a little lost. Unfortunately we could
>> not attend a
>> tutorial at NCAR so far, but we are eager to use the application.
>>
>> We want to calculate Brier scores, uncertainty, reliability and
>> resolution.
>>
>> Our data consists of an ensemble of precipitation forecasts and
their
>> corresponding gauges observations. We are interested in point
>> verification
>> against gauges. It is a statistical forecast method, not output
>> from a model
>> grid as WRF.
>>
>> Our data consists thus of a table where in the first column we have
>> the
>> observation and the in rest of the columns the forecast from each
>> ensemble
>> member.
>> We want to verify the probability of exceeding the lower 30% and
>> the 50%
>> quantiles. Our reference forecast is the in-sample observed
>> climatology for each
>> station.
>>
>> At the present stage it is not clear for us how to input this data
>> into MET and
>> what stages to follow. Could there be an example for us to look at
>> or could you
>> help us telling us steps to follow within the application?
>>
>> Thanks in advance.
>>
>> Dorita

------------------------------------------------