[Met_help] [rt.rap.ucar.edu #75012] History for Error Bars on GO Index
John Halley Gotway via RT
met_help at ucar.edu
Tue Feb 23 10:47:02 MST 2016
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi all, I have a requirement to place error bars on our build of the WMO CBS Global Index. We can't use MET to build these because of the cosine filter required. Do you have error bar generation in METs GO index calculation? If so what method was used?
Thanks
Bob Craig
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Error Bars on GO Index
From: John Halley Gotway
Time: Mon Feb 08 10:05:37 2016
Hi Bob,
The short answer to your question is no, MET does not compute error
bars
for the GO Index. However, since it's a weighted average of RMSE-
based
skill score, it certainly seems doable.
When aggregating the GO Index over many cases, one could at least
apply
bootstrapping to that aggregation. And that would capture the daily
sampling uncertainty. Do you need CI's for daily Global Index values
(i.e.
for each initialization)... or CI's for some longer aggregation time
period
(i.e. monthly)?
Is MET's lack of the cosine filter all that's preventing you from
using MET
for this? Adding support for that sort of weighting seems like it'd
be
simple enough... especially since it's mandated by WMO. If that
really is
the primary impediment, I could talk to our project managers about how
we
might fund that development in the coming year.
Regarding GO Index CI's, I've copied our resident statistician on this
message to get her input.
Thanks,
John Halley Gotway
On Mon, Feb 8, 2016 at 8:52 AM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Mon Feb 08 08:52:03 2016: Request 75012 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: Error Bars on GO Index
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75012 >
>
>
> Hi all, I have a requirement to place error bars on our build of the
WMO
> CBS Global Index. We can't use MET to build these because of the
cosine
> filter required. Do you have error bar generation in METs GO index
> calculation? If so what method was used?
>
> Thanks
> Bob Craig
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #75012] Error Bars on GO Index
From: SITTEL, MATTHEW C CTR USAF AFWA 16 WS/WXN
Time: Mon Feb 08 10:13:45 2016
Hi all,
Bob asked me about this, and the concern I have in constructing CI's
is this: since the GO Index is a function of multiple variables
aggregated over a period of time, I think you'd want to draw
individual days and use all the RMSE values from that day, since there
may be correlation between multiple variables that comprise the index.
Otherwise you'd overestimate the variance if you didn't take the
correlation into account and picked each individual component randomly
from any day, such that day 1's temperature RMSE could be paired with
day 30's SLP RMSE. Would a consideration of correlation between each
pair of variables be appropriate in this case to determine the best
course of action each time it is run? I don't know offhand if there
is any correlation, and worst of all, I'm not allowed to research it
because of my contract... so I can only think about it and offer my
input that way.
Matt
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, February 08, 2016 11:06 AM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL; tressa at ucar.edu
Subject: Re: [rt.rap.ucar.edu #75012] Error Bars on GO Index
Hi Bob,
The short answer to your question is no, MET does not compute error
bars for the GO Index. However, since it's a weighted average of
RMSE-based skill score, it certainly seems doable.
When aggregating the GO Index over many cases, one could at least
apply bootstrapping to that aggregation. And that would capture the
daily sampling uncertainty. Do you need CI's for daily Global Index
values (i.e.
for each initialization)... or CI's for some longer aggregation time
period (i.e. monthly)?
Is MET's lack of the cosine filter all that's preventing you from
using MET for this? Adding support for that sort of weighting seems
like it'd be simple enough... especially since it's mandated by WMO.
If that really is the primary impediment, I could talk to our project
managers about how we might fund that development in the coming year.
Regarding GO Index CI's, I've copied our resident statistician on this
message to get her input.
Thanks,
John Halley Gotway
On Mon, Feb 8, 2016 at 8:52 AM, robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Mon Feb 08 08:52:03 2016: Request 75012 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: Error Bars on GO Index
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75012
> >
>
>
> Hi all, I have a requirement to place error bars on our build of the
> WMO CBS Global Index. We can't use MET to build these because of
the cosine
> filter required. Do you have error bar generation in METs GO index
> calculation? If so what method was used?
>
> Thanks
> Bob Craig
>
>
>
------------------------------------------------
Subject: Error Bars on GO Index
From: Tressa Fowler
Time: Mon Feb 08 10:36:53 2016
HI guys,
I will think about this a bit and reply in more detail later, but my
first
reaction is that Matt is right. Keeping the components of the index
together is a good idea.
Tress
On Mon, Feb 8, 2016 at 10:13 AM, SITTEL, MATTHEW C CTR USAF AFWA 16
WS/WXN
via RT <met_help at ucar.edu> wrote:
> Hi all,
>
> Bob asked me about this, and the concern I have in constructing CI's
is
> this: since the GO Index is a function of multiple variables
aggregated
> over a period of time, I think you'd want to draw individual days
and use
> all the RMSE values from that day, since there may be correlation
between
> multiple variables that comprise the index. Otherwise you'd
overestimate
> the variance if you didn't take the correlation into account and
picked
> each individual component randomly from any day, such that day 1's
> temperature RMSE could be paired with day 30's SLP RMSE. Would a
> consideration of correlation between each pair of variables be
appropriate
> in this case to determine the best course of action each time it is
run? I
> don't know offhand if there is any correlation, and worst of all,
I'm not
> allowed to research it because of my contract... so I can only think
about
> it and offer my input that way.
>
> Matt
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Monday, February 08, 2016 11:06 AM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> Cc: SITTEL, MATTHEW C CTR USAF AFMC AFLCMC/HBAW-OL; tressa at ucar.edu
> Subject: Re: [rt.rap.ucar.edu #75012] Error Bars on GO Index
>
> Hi Bob,
>
> The short answer to your question is no, MET does not compute error
bars
> for the GO Index. However, since it's a weighted average of RMSE-
based
> skill score, it certainly seems doable.
>
> When aggregating the GO Index over many cases, one could at least
apply
> bootstrapping to that aggregation. And that would capture the daily
> sampling uncertainty. Do you need CI's for daily Global Index
values (i.e.
> for each initialization)... or CI's for some longer aggregation time
> period (i.e. monthly)?
>
> Is MET's lack of the cosine filter all that's preventing you from
using
> MET for this? Adding support for that sort of weighting seems like
it'd be
> simple enough... especially since it's mandated by WMO. If that
really is
> the primary impediment, I could talk to our project managers about
how we
> might fund that development in the coming year.
>
> Regarding GO Index CI's, I've copied our resident statistician on
this
> message to get her input.
>
> Thanks,
> John Halley Gotway
>
> On Mon, Feb 8, 2016 at 8:52 AM, robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Feb 08 08:52:03 2016: Request 75012 was acted upon.
> > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > Queue: met_help
> > Subject: Error Bars on GO Index
> > Owner: Nobody
> > Requestors: robert.craig.2 at us.af.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75012
> > >
> >
> >
> > Hi all, I have a requirement to place error bars on our build of
the
> > WMO CBS Global Index. We can't use MET to build these because of
the
> cosine
> > filter required. Do you have error bar generation in METs GO
index
> > calculation? If so what method was used?
> >
> > Thanks
> > Bob Craig
> >
> >
> >
>
>
>
>
--
Things may come to those who wait, but only the things left by those
who
hustle. - Abraham Lincoln
------------------------------------------------
Subject: Error Bars on GO Index
From: Tressa Fowler
Time: Thu Feb 18 12:54:06 2016
I think the best answer here would be to investigate how correlated
error values are among the different fields on the same day. Since
that's not possible, I think it's ok to make a few educated guesses.
Here are mine:
The GO index is a sum of RMSE values, which means we can probably
appeal to the normal approximation for CI. We may underestimate the
true variance somewhat if the individual RMSE components are
correlated.
We can also do bootstrapping. If all errors from individual forecasts
are sampled together, then correlation between errors should be
accounted for.
If we do both and compare them, then we can see how much difference
there is in the resulting CIs. When they are close, I usually just
report the normal. If they are pretty different, I might be inclined
to go with the bootstrap.
What do you guys think?
Tressa
------------------------------------------------
More information about the Met_help
mailing list