[Met_help] [rt.rap.ucar.edu #72299] History for Can MET or MET-TC determine best weights for each member for a weighted ensemble mean?

[John Halley Gotway via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hello,

*Short version:*
Can MET or MET-TC determine the best weights for each member to create a
weighted ensemble mean?

*Long version (questions in bold):*
I run the GFDL hurricane ensemble for HFIP's Stream 1.5 activity. This
year, I am trying to come up with a weighted ensemble mean rather than
sticking with our traditional unweighted mean. I've searched the internet
for over a week trying to find out how this is done. Today I got the bright
idea to see if MET or MET-TC had some capability to generate appropriate
ensemble member weights based on a training data set. The closest thing I
could find was frequency of superior performance, which I have used earlier
this month with my groups's version of a verification program to come up
with a first version of a weighted mean. It worked pretty well, as I got
about a 5% improvement at most lead times over the unweighted mean.
However, I was informed that the FSP output from my group's verification
program is only really valid for one-to-one comparisons (i.e., not for a
12-member ensemble). *Is MET-TC's FSP a solid result for comparing more
than 2 models at the same time?* If so, I may be tempted to use it for a
simple ensemble mean weighting technique.

However, if I am to present my work at a conference or in a paper, I want
to be sure I use a robust method to determine different weights for my
ensemble members at each lead time, for each ocean basin (maybe even have
each member be bias-corrected too). *Can MET or MET-TC (past or current
test versions) produce regression coefficients (weights) for each ensemble
member, which would be based on a training period data set?*

*More info:*
Florida State University has a "Superensemble" that uses a weighted
ensemble mean. Here is a presentation showing how they created a weighted
mean:
http://www.ofcm.gov/ihc06/Presentations/03%20session3%20Modeling%20and%20Prediction/s3-12biswas.pdf
[See slides 3-6]

If MET does not have this capability yet, I think it would be a great
addition. I feel hurricane forecast guidance will be greatly improved if
ensemble modeling groups used a weighted (and bias-corrected) mean rather
than using a simple average of model forecasts.

Thanks for any guidance you can provide!
Matt
-- 
-------------------------------------------------------
Matthew Morin
Engility
High Technology Services Group
GFDL
Princeton University Forrestal Campus
201 Forrestal Road
Princeton, NJ 08540
phone: 609-452-5381
email: Matthew.Morin at noaa.gov
-------------------------------------------------------


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Can MET or MET-TC determine best weights for each member for a weighted ensemble mean?
From: John Halley Gotway
Time: Fri Jun 12 14:14:28 2015

Matt,

The simple answer to your question is no.  MET and MET-TC are
currently not
able to help you optimize a weighted ensemble-mean.  MET and MET-TC
are
only able to compute unweighted means.

The MET ensemble-stat tool is used to compute that unweighted mean for
gridded data.  The MET-TC tc_pairs tool is used to compute an
unweighted
consensus track and intensity forecast.

The obvious first step would be to enhance those tools to compute
weighted
means.  And I agree that that's a great idea.  In the coming years I
expect
that we'll be adding more support for ensemble post-processing... such
as
computing probability-matched means and, as you suggest, weighted
means.

However, the process of computing optimal weights using a training
dataset
is another challenge.  That'd take some thought and development.

I've cc'ed Tara Jensen, the scientist in charge of MET development, to
see
if she has any thoughts or suggestions on this.

Thanks,
John Halley Gotway
met_help at ucar.edu

On Fri, Jun 12, 2015 at 8:39 AM, Matthew Morin - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:

>
> Fri Jun 12 08:39:10 2015: Request 72299 was acted upon.
> Transaction: Ticket created by matthew.morin at noaa.gov
>        Queue: met_help
>      Subject: Can MET or MET-TC determine best weights for each
member for
> a weighted ensemble mean?
>        Owner: Nobody
>   Requestors: matthew.morin at noaa.gov
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=72299 >
>
>
> Hello,
>
> *Short version:*
> Can MET or MET-TC determine the best weights for each member to
create a
> weighted ensemble mean?
>
> *Long version (questions in bold):*
> I run the GFDL hurricane ensemble for HFIP's Stream 1.5 activity.
This
> year, I am trying to come up with a weighted ensemble mean rather
than
> sticking with our traditional unweighted mean. I've searched the
internet
> for over a week trying to find out how this is done. Today I got the
bright
> idea to see if MET or MET-TC had some capability to generate
appropriate
> ensemble member weights based on a training data set. The closest
thing I
> could find was frequency of superior performance, which I have used
earlier
> this month with my groups's version of a verification program to
come up
> with a first version of a weighted mean. It worked pretty well, as I
got
> about a 5% improvement at most lead times over the unweighted mean.
> However, I was informed that the FSP output from my group's
verification
> program is only really valid for one-to-one comparisons (i.e., not
for a
> 12-member ensemble). *Is MET-TC's FSP a solid result for comparing
more
> than 2 models at the same time?* If so, I may be tempted to use it
for a
> simple ensemble mean weighting technique.
>
> However, if I am to present my work at a conference or in a paper, I
want
> to be sure I use a robust method to determine different weights for
my
> ensemble members at each lead time, for each ocean basin (maybe even
have
> each member be bias-corrected too). *Can MET or MET-TC (past or
current
> test versions) produce regression coefficients (weights) for each
ensemble
> member, which would be based on a training period data set?*
>
> *More info:*
> Florida State University has a "Superensemble" that uses a weighted
> ensemble mean. Here is a presentation showing how they created a
weighted
> mean:
>
>
http://www.ofcm.gov/ihc06/Presentations/03%20session3%20Modeling%20and%20Prediction/s3-
12biswas.pdf
> [See slides 3-6]
>
> If MET does not have this capability yet, I think it would be a
great
> addition. I feel hurricane forecast guidance will be greatly
improved if
> ensemble modeling groups used a weighted (and bias-corrected) mean
rather
> than using a simple average of model forecasts.
>
> Thanks for any guidance you can provide!
> Matt
> --
> -------------------------------------------------------
> Matthew Morin
> Engility
> High Technology Services Group
> GFDL
> Princeton University Forrestal Campus
> 201 Forrestal Road
> Princeton, NJ 08540
> phone: 609-452-5381
> email: Matthew.Morin at noaa.gov
> -------------------------------------------------------
>
>

------------------------------------------------