[Met_help] [rt.rap.ucar.edu #51986] History for 95% CI question (UNCLASSIFIED)

[Tressa Fowler via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Classification: UNCLASSIFIED
Caveats: NONE

I just conducted a brief survey of some websites to find out the statistical
significance of overlapping versus non-overlapping CIs and I find
contradictory information.

So, I would like your assessment of a typical situation where this issue has
come up.

My plots show RMSE, MAE and ME and their 95% CIs (bootstrap) as calculated
by MET Point-Stat and Stat-Analysis.

On a given plot, I compare the error statistics and CIs which were generated
from two different forecast models. An example is the error statistic for
the 2m temperature forecast for each model. 

In some cases the value of the error statistic with the CI for one model
overlaps the error statistic with the CI for the other model.

In other cases there is no overlap.

My interpretation about what I could conclude from these two situations was
as follows:

1. If there is no overlap, then the difference between the error statistics
is statistically significant at the 95% confidence level.

2. If there is overlap, then the difference between the error statistics is
not statistically significant at the 95% confidence level.

What may I safely conclude about both of these situations statistically?

Thanks.

R/
John


Mr John W. Raby, Meteorologist
U.S. Army Research Laboratory
White Sands Missile Range, NM 88002
(575) 678-2004 DSN 258-2004
FAX (575) 678-1230 DSN 258-1230
Email: john.w.raby2.civ at mail.mil



Classification: UNCLASSIFIED
Caveats: NONE




----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: 95% CI question (UNCLASSIFIED)
From: Tressa Fowler
Time: Fri Dec 16 10:51:34 2011

Hi John,

Yes, if there is no overlap, then the statistic is significantly
different. If there is overlap, it may or may not be. Variances are
difficult to estimate, and are often overinflated. In these cases,
there may be a difference you just don't see, but would if you used a
paired analysis or some other test.

Tressa

On Mon Dec 12 14:02:17 2011, john.w.raby2.civ at mail.mil wrote:
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> I just conducted a brief survey of some websites to find out the
statistical
> significance of overlapping versus non-overlapping CIs and I find
> contradictory information.
>
> So, I would like your assessment of a typical situation where this
issue has
> come up.
>
> My plots show RMSE, MAE and ME and their 95% CIs (bootstrap) as
calculated
> by MET Point-Stat and Stat-Analysis.
>
> On a given plot, I compare the error statistics and CIs which were
generated
> from two different forecast models. An example is the error
statistic for
> the 2m temperature forecast for each model.
>
> In some cases the value of the error statistic with the CI for one
model
> overlaps the error statistic with the CI for the other model.
>
> In other cases there is no overlap.
>
> My interpretation about what I could conclude from these two
situations was
> as follows:
>
> 1. If there is no overlap, then the difference between the error
statistics
> is statistically significant at the 95% confidence level.
>
> 2. If there is overlap, then the difference between the error
statistics is
> not statistically significant at the 95% confidence level.
>
> What may I safely conclude about both of these situations
statistically?
>
> Thanks.
>
> R/
> John
>
>
> Mr John W. Raby, Meteorologist
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> (575) 678-2004 DSN 258-2004
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>



------------------------------------------------
Subject: 95% CI question (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Fri Dec 16 10:56:21 2011

Classification: UNCLASSIFIED
Caveats: NONE

Tressa -

Thanks for your response. This help us cast our results/conclusions in
the
right way here.

The plot certainly thickens as I learn more about applying statistics
to
forecast verification.

R/
John


-----Original Message-----
From: Tressa Fowler via RT [mailto:met_help at ucar.edu]
Sent: Friday, December 16, 2011 10:52 AM
To: Raby, John W USA CIV (US)
Subject: [rt.rap.ucar.edu #51986] 95% CI question (UNCLASSIFIED)

Hi John,

Yes, if there is no overlap, then the statistic is significantly
different. If
there is overlap, it may or may not be. Variances are difficult to
estimate,
and are often overinflated. In these cases, there may be a difference
you just
don't see, but would if you used a paired analysis or some other test.

Tressa

On Mon Dec 12 14:02:17 2011, john.w.raby2.civ at mail.mil wrote:
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> I just conducted a brief survey of some websites to find out the
> statistical significance of overlapping versus non-overlapping CIs
and
> I find contradictory information.
>
> So, I would like your assessment of a typical situation where this
> issue has come up.
>
> My plots show RMSE, MAE and ME and their 95% CIs (bootstrap) as
> calculated by MET Point-Stat and Stat-Analysis.
>
> On a given plot, I compare the error statistics and CIs which were
> generated from two different forecast models. An example is the
error
> statistic for the 2m temperature forecast for each model.
>
> In some cases the value of the error statistic with the CI for one
> model overlaps the error statistic with the CI for the other model.
>
> In other cases there is no overlap.
>
> My interpretation about what I could conclude from these two
> situations was as follows:
>
> 1. If there is no overlap, then the difference between the error
> statistics is statistically significant at the 95% confidence level.
>
> 2. If there is overlap, then the difference between the error
> statistics is not statistically significant at the 95% confidence
level.
>
> What may I safely conclude about both of these situations
statistically?
>
> Thanks.
>
> R/
> John
>
>
> Mr John W. Raby, Meteorologist
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> (575) 678-2004 DSN 258-2004
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>




Classification: UNCLASSIFIED
Caveats: NONE



------------------------------------------------