[Met_help] [rt.rap.ucar.edu #71419] History for CI for MET-TC tc_stat

Mon Mar 6 12:23:13 MST 2017

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hi there,

In the output from MET-TC (tc_stat), I'm wondering how the MEAN_NCL and
MEAN_NCU columns are computed. I'm assuming these are the lower and upper
confidence intervals, is this correct? If so, how is this computed? I've
attached a sample output file for reference.

Thanks,
Andy

-- 
Andrew Penny
Hurricane Model Diagnostician
SRG / National Hurricane Center
11691 SW 17th Street
Miami, FL 33165-2149
phone: 305.229.4457
email: andrew.penny at noaa.gov

----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: CI for MET-TC tc_stat
From: John Halley Gotway
Time: Mon Apr 13 15:23:52 2015

Andy,

You are correct, NCL and NCL stand for normal confidence interval
lower and
upper limits.

In the same data you sent, the first line not containing missing data
is a
mean value of 10.81587 with a normal CI of (9.88275, 11.749) computed
over
315 values.  The standard deviation of the 315 values is 8.4498.

The CI is computed by the function named "compute_mean_stdev()"
defined in:
   src/libcode/vx_statistics/met_stats.cc

The CI equation is basically this:
  mean +/- z * stdev / sqrt(n)

Where "z" is the critical value as determined by the alpha value
(default
alpha of 0.05 means a 95% CI).  For a 2-tailed 95% CI, z = 1.96.
Using
your data values, that's:
   10.81587 +/- 1.96 * 8.4498 / 17.7482 = 10.81587 +/- 0.93314 =
(9.8827,
11.749)

Hope that helps.

Thanks,
John

On Mon, Apr 13, 2015 at 8:02 AM, Andrew Penny - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:

>
> Mon Apr 13 08:02:57 2015: Request 71419 was acted upon.
> Transaction: Ticket created by andrew.penny at noaa.gov
>        Queue: met_help
>      Subject: CI for MET-TC tc_stat
>        Owner: Nobody
>   Requestors: andrew.penny at noaa.gov
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71419 >
>
>
> Hi there,
>
> In the output from MET-TC (tc_stat), I'm wondering how the MEAN_NCL
and
> MEAN_NCU columns are computed. I'm assuming these are the lower and
upper
> confidence intervals, is this correct? If so, how is this computed?
I've
> attached a sample output file for reference.
>
> Thanks,
> Andy
>
> --
> Andrew Penny
> Hurricane Model Diagnostician
> SRG / National Hurricane Center
> 11691 SW 17th Street
> Miami, FL 33165-2149
> phone: 305.229.4457
> email: andrew.penny at noaa.gov
>
>

------------------------------------------------
Subject: CI for MET-TC tc_stat
From: Andrew Penny - NOAA Affiliate
Time: Tue Apr 14 05:16:15 2015

Thanks for the explanation, John. Very much appreciated!

Take care,
Andy

On Mon, Apr 13, 2015 at 5:23 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:

> Andy,
>
> You are correct, NCL and NCL stand for normal confidence interval
lower and
> upper limits.
>
> In the same data you sent, the first line not containing missing
data is a
> mean value of 10.81587 with a normal CI of (9.88275, 11.749)
computed over
> 315 values.  The standard deviation of the 315 values is 8.4498.
>
> The CI is computed by the function named "compute_mean_stdev()"
defined in:
>    src/libcode/vx_statistics/met_stats.cc
>
> The CI equation is basically this:
>   mean +/- z * stdev / sqrt(n)
>
> Where "z" is the critical value as determined by the alpha value
(default
> alpha of 0.05 means a 95% CI).  For a 2-tailed 95% CI, z = 1.96.
Using
> your data values, that's:
>    10.81587 +/- 1.96 * 8.4498 / 17.7482 = 10.81587 +/- 0.93314 =
(9.8827,
> 11.749)
>
> Hope that helps.
>
> Thanks,
> John
>
>
>
>
>
> On Mon, Apr 13, 2015 at 8:02 AM, Andrew Penny - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Apr 13 08:02:57 2015: Request 71419 was acted upon.
> > Transaction: Ticket created by andrew.penny at noaa.gov
> >        Queue: met_help
> >      Subject: CI for MET-TC tc_stat
> >        Owner: Nobody
> >   Requestors: andrew.penny at noaa.gov
> >       Status: new
> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71419 >
> >
> >
> > Hi there,
> >
> > In the output from MET-TC (tc_stat), I'm wondering how the
MEAN_NCL and
> > MEAN_NCU columns are computed. I'm assuming these are the lower
and upper
> > confidence intervals, is this correct? If so, how is this
computed? I've
> > attached a sample output file for reference.
> >
> > Thanks,
> > Andy
> >
> > --
> > Andrew Penny
> > Hurricane Model Diagnostician
> > SRG / National Hurricane Center
> > 11691 SW 17th Street
> > Miami, FL 33165-2149
> > phone: 305.229.4457
> > email: andrew.penny at noaa.gov
> >
> >
>
>

--
Andrew Penny
Hurricane Model Diagnostician
SRG / National Hurricane Center
11691 SW 17th Street
Miami, FL 33165-2149
phone: 305.229.4457
email: andrew.penny at noaa.gov

------------------------------------------------
Subject: CI for MET-TC tc_stat
From: Andrew Penny - NOAA Affiliate
Time: Fri Mar 03 09:20:34 2017

Hi John,

As a follow up to our discussion a long time ago regarding the
confidence
intervals output from tc_stat, what would be the most appropriate (or
correct) way to test for significance with respect to track and
intensity
errors between two models? Is there a way to do this within MET-TC?
I've
been told that the errors can be considered significantly different
when
the CIs don't overlap, or when the mean from one sample is outside the
bounds of the CIs of the other sample, but I'm not sure either of
these are
correct.

Wouldn't it be more appropriate to look at the differences in the
errors
for each forecast (computing both the mean and variance of the
differences
for the entire sample) and see whether the resulting CIs encompass
zero
(not significantly different)?

Thanks,
Andy

On Mon, Apr 13, 2015 at 9:23 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:

> Andy,
>
> You are correct, NCL and NCL stand for normal confidence interval
lower and
> upper limits.
>
> In the same data you sent, the first line not containing missing
data is a
> mean value of 10.81587 with a normal CI of (9.88275, 11.749)
computed over
> 315 values.  The standard deviation of the 315 values is 8.4498.
>
> The CI is computed by the function named "compute_mean_stdev()"
defined in:
>    src/libcode/vx_statistics/met_stats.cc
>
> The CI equation is basically this:
>   mean +/- z * stdev / sqrt(n)
>
> Where "z" is the critical value as determined by the alpha value
(default
> alpha of 0.05 means a 95% CI).  For a 2-tailed 95% CI, z = 1.96.
Using
> your data values, that's:
>    10.81587 +/- 1.96 * 8.4498 / 17.7482 = 10.81587 +/- 0.93314 =
(9.8827,
> 11.749)
>
> Hope that helps.
>
> Thanks,
> John
>
>
>
>
>
> On Mon, Apr 13, 2015 at 8:02 AM, Andrew Penny - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Apr 13 08:02:57 2015: Request 71419 was acted upon.
> > Transaction: Ticket created by andrew.penny at noaa.gov
> >        Queue: met_help
> >      Subject: CI for MET-TC tc_stat
> >        Owner: Nobody
> >   Requestors: andrew.penny at noaa.gov
> >       Status: new
> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71419 >
> >
> >
> > Hi there,
> >
> > In the output from MET-TC (tc_stat), I'm wondering how the
MEAN_NCL and
> > MEAN_NCU columns are computed. I'm assuming these are the lower
and upper
> > confidence intervals, is this correct? If so, how is this
computed? I've
> > attached a sample output file for reference.
> >
> > Thanks,
> > Andy
> >
> > --
> > Andrew Penny
> > Hurricane Model Diagnostician
> > SRG / National Hurricane Center
> > 11691 SW 17th Street
> > Miami, FL 33165-2149
> > phone: 305.229.4457
> > email: andrew.penny at noaa.gov
> >
> >
>
>

--
Andrew Penny
Hurricane Model Diagnostician
SRG / National Hurricane Center
11691 SW 17th Street
Miami, FL 33165-2149
phone: 305.229.4457
email: andrew.penny at noaa.gov

------------------------------------------------
Subject: CI for MET-TC tc_stat
From: John Halley Gotway
Time: Fri Mar 03 09:45:38 2017

Andy,

In the evaluation projects that we've done in the NCAR/DTC, the
statisticians have advised two options at various points.  And they
are
both options that you mentioned in your email.

(1) If the confidence intervals for two lines do not overlap at all,
then
you can interpret their difference as significant.
(2) If the confidence interval of the pair-wise differences do not
include
zero, then the differences are significant at the alpha value you've
chosen
(i.e. 95% or 99% CI).

And the statisticians advise that (2) is a more powerful test than
(1).
Some differences which are NOT significant by (1) will be shown to be
significant by (2).

In general in the DTC, we've computed pair-wise differences and
checked to
see if the 95 or 99% confidence intervals about the mean includes 0 or
not.  And I have not heard about the 3rd option you mentioned...
checking
if the mean of one distribution falls outside the CI.

Hope that helps clarify.

Out of curiosity, are you using the plot_tcmpr.R Rscript that's
included in
the MET release to plot the output of tc-pairs?

Thanks,
John

On Fri, Mar 3, 2017 at 9:20 AM, Andrew Penny - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71419 >
>
> Hi John,
>
> As a follow up to our discussion a long time ago regarding the
confidence
> intervals output from tc_stat, what would be the most appropriate
(or
> correct) way to test for significance with respect to track and
intensity
> errors between two models? Is there a way to do this within MET-TC?
I've
> been told that the errors can be considered significantly different
when
> the CIs don't overlap, or when the mean from one sample is outside
the
> bounds of the CIs of the other sample, but I'm not sure either of
these are
> correct.
>
> Wouldn't it be more appropriate to look at the differences in the
errors
> for each forecast (computing both the mean and variance of the
differences
> for the entire sample) and see whether the resulting CIs encompass
zero
> (not significantly different)?
>
> Thanks,
> Andy
>
> On Mon, Apr 13, 2015 at 9:23 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Andy,
> >
> > You are correct, NCL and NCL stand for normal confidence interval
lower
> and
> > upper limits.
> >
> > In the same data you sent, the first line not containing missing
data is
> a
> > mean value of 10.81587 with a normal CI of (9.88275, 11.749)
computed
> over
> > 315 values.  The standard deviation of the 315 values is 8.4498.
> >
> > The CI is computed by the function named "compute_mean_stdev()"
defined
> in:
> >    src/libcode/vx_statistics/met_stats.cc
> >
> > The CI equation is basically this:
> >   mean +/- z * stdev / sqrt(n)
> >
> > Where "z" is the critical value as determined by the alpha value
(default
> > alpha of 0.05 means a 95% CI).  For a 2-tailed 95% CI, z = 1.96.
Using
> > your data values, that's:
> >    10.81587 +/- 1.96 * 8.4498 / 17.7482 = 10.81587 +/- 0.93314 =
(9.8827,
> > 11.749)
> >
> > Hope that helps.
> >
> > Thanks,
> > John
> >
> >
> >
> >
> >
> > On Mon, Apr 13, 2015 at 8:02 AM, Andrew Penny - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Mon Apr 13 08:02:57 2015: Request 71419 was acted upon.
> > > Transaction: Ticket created by andrew.penny at noaa.gov
> > >        Queue: met_help
> > >      Subject: CI for MET-TC tc_stat
> > >        Owner: Nobody
> > >   Requestors: andrew.penny at noaa.gov
> > >       Status: new
> > >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71419
> >
> > >
> > >
> > > Hi there,
> > >
> > > In the output from MET-TC (tc_stat), I'm wondering how the
MEAN_NCL and
> > > MEAN_NCU columns are computed. I'm assuming these are the lower
and
> upper
> > > confidence intervals, is this correct? If so, how is this
computed?
> I've
> > > attached a sample output file for reference.
> > >
> > > Thanks,
> > > Andy
> > >
> > > --
> > > Andrew Penny
> > > Hurricane Model Diagnostician
> > > SRG / National Hurricane Center
> > > 11691 SW 17th Street
> > > Miami, FL 33165-2149
> > > phone: 305.229.4457
> > > email: andrew.penny at noaa.gov
> > >
> > >
> >
> >
>
>
> --
> Andrew Penny
> Hurricane Model Diagnostician
> SRG / National Hurricane Center
> 11691 SW 17th Street
> Miami, FL 33165-2149
> phone: 305.229.4457
> email: andrew.penny at noaa.gov
>
>

------------------------------------------------
Subject: CI for MET-TC tc_stat
From: Andrew Penny - NOAA Affiliate
Time: Fri Mar 03 10:25:07 2017

Thanks for the explanation, John. Very helpful.

No, I haven't use the R script before. Generally I use tc_stat to
summarize
the results, then plot things using NCL.

Is there an easy way to get at the pair-wise differences using some of
the
same filter options in tc_stat (storm classification at initial and
valid
times etc.)?

Thanks for your help,
Andy

On Fri, Mar 3, 2017 at 4:45 PM, John Halley Gotway via RT
<met_help at ucar.edu
> wrote:

> Andy,
>
> In the evaluation projects that we've done in the NCAR/DTC, the
> statisticians have advised two options at various points.  And they
are
> both options that you mentioned in your email.
>
> (1) If the confidence intervals for two lines do not overlap at all,
then
> you can interpret their difference as significant.
> (2) If the confidence interval of the pair-wise differences do not
include
> zero, then the differences are significant at the alpha value you've
chosen
> (i.e. 95% or 99% CI).
>
> And the statisticians advise that (2) is a more powerful test than
(1).
> Some differences which are NOT significant by (1) will be shown to
be
> significant by (2).
>
> In general in the DTC, we've computed pair-wise differences and
checked to
> see if the 95 or 99% confidence intervals about the mean includes 0
or
> not.  And I have not heard about the 3rd option you mentioned...
checking
> if the mean of one distribution falls outside the CI.
>
> Hope that helps clarify.
>
> Out of curiosity, are you using the plot_tcmpr.R Rscript that's
included in
> the MET release to plot the output of tc-pairs?
>
> Thanks,
> John
>
>
>
> On Fri, Mar 3, 2017 at 9:20 AM, Andrew Penny - NOAA Affiliate via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71419 >
> >
> > Hi John,
> >
> > As a follow up to our discussion a long time ago regarding the
confidence
> > intervals output from tc_stat, what would be the most appropriate
(or
> > correct) way to test for significance with respect to track and
intensity
> > errors between two models? Is there a way to do this within MET-
TC? I've
> > been told that the errors can be considered significantly
different when
> > the CIs don't overlap, or when the mean from one sample is outside
the
> > bounds of the CIs of the other sample, but I'm not sure either of
these
> are
> > correct.
> >
> > Wouldn't it be more appropriate to look at the differences in the
errors
> > for each forecast (computing both the mean and variance of the
> differences
> > for the entire sample) and see whether the resulting CIs encompass
zero
> > (not significantly different)?
> >
> > Thanks,
> > Andy
> >
> > On Mon, Apr 13, 2015 at 9:23 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Andy,
> > >
> > > You are correct, NCL and NCL stand for normal confidence
interval lower
> > and
> > > upper limits.
> > >
> > > In the same data you sent, the first line not containing missing
data
> is
> > a
> > > mean value of 10.81587 with a normal CI of (9.88275, 11.749)
computed
> > over
> > > 315 values.  The standard deviation of the 315 values is 8.4498.
> > >
> > > The CI is computed by the function named "compute_mean_stdev()"
defined
> > in:
> > >    src/libcode/vx_statistics/met_stats.cc
> > >
> > > The CI equation is basically this:
> > >   mean +/- z * stdev / sqrt(n)
> > >
> > > Where "z" is the critical value as determined by the alpha value
> (default
> > > alpha of 0.05 means a 95% CI).  For a 2-tailed 95% CI, z = 1.96.
Using
> > > your data values, that's:
> > >    10.81587 +/- 1.96 * 8.4498 / 17.7482 = 10.81587 +/- 0.93314 =
> (9.8827,
> > > 11.749)
> > >
> > > Hope that helps.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Apr 13, 2015 at 8:02 AM, Andrew Penny - NOAA Affiliate
via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Mon Apr 13 08:02:57 2015: Request 71419 was acted upon.
> > > > Transaction: Ticket created by andrew.penny at noaa.gov
> > > >        Queue: met_help
> > > >      Subject: CI for MET-TC tc_stat
> > > >        Owner: Nobody
> > > >   Requestors: andrew.penny at noaa.gov
> > > >       Status: new
> > > >  Ticket <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=71419
> > >
> > > >
> > > >
> > > > Hi there,
> > > >
> > > > In the output from MET-TC (tc_stat), I'm wondering how the
MEAN_NCL
> and
> > > > MEAN_NCU columns are computed. I'm assuming these are the
lower and
> > upper
> > > > confidence intervals, is this correct? If so, how is this
computed?
> > I've
> > > > attached a sample output file for reference.
> > > >
> > > > Thanks,
> > > > Andy
> > > >
> > > > --
> > > > Andrew Penny
> > > > Hurricane Model Diagnostician
> > > > SRG / National Hurricane Center
> > > > 11691 SW 17th Street
> > > > Miami, FL 33165-2149
> > > > phone: 305.229.4457
> > > > email: andrew.penny at noaa.gov
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > Andrew Penny
> > Hurricane Model Diagnostician
> > SRG / National Hurricane Center
> > 11691 SW 17th Street
> > Miami, FL 33165-2149
> > phone: 305.229.4457
> > email: andrew.penny at noaa.gov
> >
> >
>
>

--
Andrew Penny
Hurricane Model Diagnostician
SRG / National Hurricane Center
11691 SW 17th Street
Miami, FL 33165-2149
phone: 305.229.4457
email: andrew.penny at noaa.gov

------------------------------------------------
Subject: CI for MET-TC tc_stat
From: John Halley Gotway
Time: Fri Mar 03 17:49:53 2017

Andy,

One necessary step in computing pairwise differences is "event
equalizing"
the data.  By event equalize, I mean extracting a subset of cases that
are
common to both models.

While the tc_stat tool does not compute pairwise difference for you,
it can
apply the "event_equalization" logic to extract the cases common to
two
models.  This is done using a the config file "event_equal = TRUE;"
option
or setting "-event_equal true" on the command line.

Most of our hurricane track analysis and plotting is done using the
plot_tcmpr.R Rscript.  It makes a call to the tc_stat tool to the
track
data down to the desired subset, compute pairwise difference if
needed, and
then plot the result.  By way of example, I've attached a sample file
containing a bunch of output from tc_pairs, along with the commands
you
could use to compute a pairwise difference:

cd met-5.2
wget
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/penny_data/tc_pairs_output.tcst
setenv MET_BUILD_BASE `pwd`
Rscript scripts/Rscripts/plot_tcmpr.R -lookin tc_pairs_output.tcst
-filter
'-amodel AHWI,GFSI' -series AMODEL AHWI,GFSI,AHWI-GFSI -plot
MEAN,BOXPLOT

The resulting plots it created are attached.  They include three
series...
one for AHWI, one for GFSI, and one for their pairwise difference.

It's a bit cumbersome to understand all the options available, but I
think
you may find it really useful.  If nothing else, you could adapt it to
dump
out the pairwise differences you want.

Thanks,
John

------------------------------------------------
Subject: CI for MET-TC tc_stat
From: Andrew Penny - NOAA Affiliate
Time: Mon Mar 06 11:33:23 2017

Thanks John. I'll take a look at the R script. I really appreciate the
help.

Take care,
Andy

On Sat, Mar 4, 2017 at 12:49 AM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:

> Andy,
>
> One necessary step in computing pairwise differences is "event
equalizing"
> the data.  By event equalize, I mean extracting a subset of cases
that are
> common to both models.
>
> While the tc_stat tool does not compute pairwise difference for you,
it can
> apply the "event_equalization" logic to extract the cases common to
two
> models.  This is done using a the config file "event_equal = TRUE;"
option
> or setting "-event_equal true" on the command line.
>
> Most of our hurricane track analysis and plotting is done using the
> plot_tcmpr.R Rscript.  It makes a call to the tc_stat tool to the
track
> data down to the desired subset, compute pairwise difference if
needed, and
> then plot the result.  By way of example, I've attached a sample
file
> containing a bunch of output from tc_pairs, along with the commands
you
> could use to compute a pairwise difference:
>
> cd met-5.2
> wget
> ftp://ftp.rap.ucar.edu/incoming/irap/met_help/penny_
> data/tc_pairs_output.tcst
> setenv MET_BUILD_BASE `pwd`
> Rscript scripts/Rscripts/plot_tcmpr.R -lookin tc_pairs_output.tcst
-filter
> '-amodel AHWI,GFSI' -series AMODEL AHWI,GFSI,AHWI-GFSI -plot
MEAN,BOXPLOT
>
> The resulting plots it created are attached.  They include three
series...
> one for AHWI, one for GFSI, and one for their pairwise difference.
>
> It's a bit cumbersome to understand all the options available, but I
think
> you may find it really useful.  If nothing else, you could adapt it
to dump
> out the pairwise differences you want.
>
> Thanks,
> John
>
>

--
Andrew Penny
Hurricane Model Diagnostician
SRG / National Hurricane Center
11691 SW 17th Street
Miami, FL 33165-2149
phone: 305.229.4457
email: andrew.penny at noaa.gov

------------------------------------------------