[Met_help] [rt.rap.ucar.edu #81842] History for BRIER_NCL and BRIER_NCU Description
John Halley Gotway via RT
met_help at ucar.edu
Fri Sep 1 11:38:22 MDT 2017
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Good morning,
I have a rather specific question regarding the BRIER_NCL and BRIER_NCU
output values from grid_stat. Looking at table 7.11 wiithin
http://www.dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v6.0.pdf,
I'm not sure quantitatively what is meant by "normal upper and lower
confidence limits." Are these confidence limits controlled by the alpha
variable? If possible, I am looking for the 95% confidence interval from
BRIER_NCL and BRIER_NCU.
Thank you again!
Mike
--
Michael J. Erickson
Research Scientist
Cooperative Institute for Research in Environmental Sciences (CIRES)
NOAA/NWS/Weather Prediction Center
Phone: 301-683-1546
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: BRIER_NCL and BRIER_NCU Description
From: John Halley Gotway
Time: Fri Sep 01 09:33:50 2017
Mike,
The MET output includes both parametric and non-parametric confidence
intervals for most statistics.
The columns ending in _BCL and _BCU, which stand for "bootstrap
confidence
lower and upper limits", contain the non-parametric estimates. They
are
the result of bootstrapping the individual matched pairs. Since the
MET
statistics tools (like point_stat and grid_stat) are run once per
output
time, the confidence intervals listed in the output give an estimate
of the
spatial sampling uncertainty.
The columns ending in _NCL and _NCU, which stand for "normal
confidence
lower and upper limits", contain the parametric estimates. Using the
word
"normal" is really a misnomer. While many of the parametric
confidence
intervals are based on assumptions of normality, some of them are not.
The
parametric confidence interval estimate was chosen separately for each
statistic, as determined by the statisticians in our group.
To answer your question, yes, the alpha value is used to determine the
level of certainty you'd like to include. Setting alpha = 0.05 means
a 95%
confidence interval. alpha = 0.01 means a 99% confidence interval.
The
smaller alpha you choose, the more certain you'd like to be that the
true
statistic will fall in the interval, meaning the wider the confidence
interval must be. You should be able to specify multiple alpha values
in
the config file and check to see that with larger alphas, you get
narrower
CI's and with smaller alphas, you get wider CI's. But remember that
this
is only an estimate of the spatial sampling uncertainty.
The BRIER_NCL and BRIER_NCU values are computed by this function in
MET:
double Nx2ContingencyTable::brier_ci_halfwidth(double alpha) const
Which is defined in this file:
src/libcode/vx_statistics/contable_nx2.cc
Listed in that code is the following reference for computing Brier
score
confidence intervals:
// Bradley, A.A, S.S. Schwartz, and T. Hashino, 2008:
// Sampling Uncertainty and Confidence Intervals for the Brier Score
// and Brier Skill Score. Weather and Forecasting, 23, 992-1006.
Thanks,
John Halley Gotway
On Fri, Sep 1, 2017 at 7:00 AM, Michael Erickson - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> Fri Sep 01 07:00:42 2017: Request 81842 was acted upon.
> Transaction: Ticket created by michael.j.erickson at noaa.gov
> Queue: met_help
> Subject: BRIER_NCL and BRIER_NCU Description
> Owner: Nobody
> Requestors: michael.j.erickson at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=81842 >
>
>
> Good morning,
>
> I have a rather specific question regarding the BRIER_NCL and
BRIER_NCU
> output values from grid_stat. Looking at table 7.11 wiithin
> http://www.dtcenter.org/met/users/docs/users_guide/MET_
> Users_Guide_v6.0.pdf,
> I'm not sure quantitatively what is meant by "normal upper and lower
> confidence limits." Are these confidence limits controlled by the
alpha
> variable? If possible, I am looking for the 95% confidence interval
from
> BRIER_NCL and BRIER_NCU.
>
> Thank you again!
>
> Mike
>
> --
> Michael J. Erickson
>
> Research Scientist
> Cooperative Institute for Research in Environmental Sciences (CIRES)
> NOAA/NWS/Weather Prediction Center
> Phone: 301-683-1546
>
>
------------------------------------------------
Subject: BRIER_NCL and BRIER_NCU Description
From: Michael Erickson - NOAA Affiliate
Time: Fri Sep 01 09:50:19 2017
Hi John,
Thanks for your quick and helpful answer!
So in the process of bootstrapping the individual matched pairs, there
is
an assumption that each spatial point is independent. Is that correct?
This
is an unavoidable problem, but I can probably be more cautious by
decreasing my alpha.
Thanks again!
Mike
On Fri, Sep 1, 2017 at 11:33 AM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Mike,
>
> The MET output includes both parametric and non-parametric
confidence
> intervals for most statistics.
>
> The columns ending in _BCL and _BCU, which stand for "bootstrap
confidence
> lower and upper limits", contain the non-parametric estimates. They
are
> the result of bootstrapping the individual matched pairs. Since the
MET
> statistics tools (like point_stat and grid_stat) are run once per
output
> time, the confidence intervals listed in the output give an estimate
of the
> spatial sampling uncertainty.
>
> The columns ending in _NCL and _NCU, which stand for "normal
confidence
> lower and upper limits", contain the parametric estimates. Using
the word
> "normal" is really a misnomer. While many of the parametric
confidence
> intervals are based on assumptions of normality, some of them are
not. The
> parametric confidence interval estimate was chosen separately for
each
> statistic, as determined by the statisticians in our group.
>
> To answer your question, yes, the alpha value is used to determine
the
> level of certainty you'd like to include. Setting alpha = 0.05
means a 95%
> confidence interval. alpha = 0.01 means a 99% confidence interval.
The
> smaller alpha you choose, the more certain you'd like to be that the
true
> statistic will fall in the interval, meaning the wider the
confidence
> interval must be. You should be able to specify multiple alpha
values in
> the config file and check to see that with larger alphas, you get
narrower
> CI's and with smaller alphas, you get wider CI's. But remember that
this
> is only an estimate of the spatial sampling uncertainty.
>
> The BRIER_NCL and BRIER_NCU values are computed by this function in
MET:
> double Nx2ContingencyTable::brier_ci_halfwidth(double alpha)
const
> Which is defined in this file:
> src/libcode/vx_statistics/contable_nx2.cc
>
> Listed in that code is the following reference for computing Brier
score
> confidence intervals:
>
> // Bradley, A.A, S.S. Schwartz, and T. Hashino, 2008:
> // Sampling Uncertainty and Confidence Intervals for the Brier Score
> // and Brier Skill Score. Weather and Forecasting, 23, 992-1006.
>
> Thanks,
> John Halley Gotway
>
>
>
>
>
>
>
> On Fri, Sep 1, 2017 at 7:00 AM, Michael Erickson - NOAA Affiliate
via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Fri Sep 01 07:00:42 2017: Request 81842 was acted upon.
> > Transaction: Ticket created by michael.j.erickson at noaa.gov
> > Queue: met_help
> > Subject: BRIER_NCL and BRIER_NCU Description
> > Owner: Nobody
> > Requestors: michael.j.erickson at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=81842 >
> >
> >
> > Good morning,
> >
> > I have a rather specific question regarding the BRIER_NCL and
BRIER_NCU
> > output values from grid_stat. Looking at table 7.11 wiithin
> > http://www.dtcenter.org/met/users/docs/users_guide/MET_
> > Users_Guide_v6.0.pdf,
> > I'm not sure quantitatively what is meant by "normal upper and
lower
> > confidence limits." Are these confidence limits controlled by the
alpha
> > variable? If possible, I am looking for the 95% confidence
interval from
> > BRIER_NCL and BRIER_NCU.
> >
> > Thank you again!
> >
> > Mike
> >
> > --
> > Michael J. Erickson
> >
> > Research Scientist
> > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > NOAA/NWS/Weather Prediction Center
> > Phone: 301-683-1546
> >
> >
>
>
--
Michael J. Erickson
Research Scientist
Cooperative Institute for Research in Environmental Sciences (CIRES)
NOAA/NWS/Weather Prediction Center
Phone: 301-683-1546
------------------------------------------------
Subject: BRIER_NCL and BRIER_NCU Description
From: John Halley Gotway
Time: Fri Sep 01 10:43:02 2017
Michael,
Yes, that's correct. There is the assumption of IID
(independent-identically-distributed), which is a bad assumption.
We've discussed this over the years with statisticians here, and the
suggested way of handling spatial dependence is using a spatial block
bootstrap instead. You'd break up the pairs into spatial blocks where
the
assumption of IID is more reasonable and then bootstrap the groups of
pairs
rather than the individual ones. But in practice, it's difficult to
apply
this method generally across all the data that is processed by MET.
There's no easy way to make good default decisions for spatial block
size.
When you ask as statistician about these things, they typically
answer,
"well it really depends on your data"... because it really does!
This detail hasn't risen to the top of our development priorities.
And in
practice, we seldom use the CI's computed by MET itself. Instead, we
run
MET over many days and aggregate stats through time using METViewer.
In
that case, we're addressing temporal uncertainty instead of spatial.
I think being aware of poor IID assumption is great, and choosing a
lower
alpha value (maybe 0.01 rather than 0.05) than you otherwise would to
account for it is wise.
Thanks,
John
On Fri, Sep 1, 2017 at 9:50 AM, Michael Erickson - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=81842 >
>
> Hi John,
>
> Thanks for your quick and helpful answer!
>
> So in the process of bootstrapping the individual matched pairs,
there is
> an assumption that each spatial point is independent. Is that
correct? This
> is an unavoidable problem, but I can probably be more cautious by
> decreasing my alpha.
>
> Thanks again!
>
> Mike
>
> On Fri, Sep 1, 2017 at 11:33 AM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Mike,
> >
> > The MET output includes both parametric and non-parametric
confidence
> > intervals for most statistics.
> >
> > The columns ending in _BCL and _BCU, which stand for "bootstrap
> confidence
> > lower and upper limits", contain the non-parametric estimates.
They are
> > the result of bootstrapping the individual matched pairs. Since
the MET
> > statistics tools (like point_stat and grid_stat) are run once per
output
> > time, the confidence intervals listed in the output give an
estimate of
> the
> > spatial sampling uncertainty.
> >
> > The columns ending in _NCL and _NCU, which stand for "normal
confidence
> > lower and upper limits", contain the parametric estimates. Using
the
> word
> > "normal" is really a misnomer. While many of the parametric
confidence
> > intervals are based on assumptions of normality, some of them are
not.
> The
> > parametric confidence interval estimate was chosen separately for
each
> > statistic, as determined by the statisticians in our group.
> >
> > To answer your question, yes, the alpha value is used to determine
the
> > level of certainty you'd like to include. Setting alpha = 0.05
means a
> 95%
> > confidence interval. alpha = 0.01 means a 99% confidence
interval. The
> > smaller alpha you choose, the more certain you'd like to be that
the true
> > statistic will fall in the interval, meaning the wider the
confidence
> > interval must be. You should be able to specify multiple alpha
values in
> > the config file and check to see that with larger alphas, you get
> narrower
> > CI's and with smaller alphas, you get wider CI's. But remember
that this
> > is only an estimate of the spatial sampling uncertainty.
> >
> > The BRIER_NCL and BRIER_NCU values are computed by this function
in MET:
> > double Nx2ContingencyTable::brier_ci_halfwidth(double alpha)
const
> > Which is defined in this file:
> > src/libcode/vx_statistics/contable_nx2.cc
> >
> > Listed in that code is the following reference for computing Brier
score
> > confidence intervals:
> >
> > // Bradley, A.A, S.S. Schwartz, and T. Hashino, 2008:
> > // Sampling Uncertainty and Confidence Intervals for the Brier
Score
> > // and Brier Skill Score. Weather and Forecasting, 23, 992-1006.
> >
> > Thanks,
> > John Halley Gotway
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Sep 1, 2017 at 7:00 AM, Michael Erickson - NOAA Affiliate
via RT
> <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Fri Sep 01 07:00:42 2017: Request 81842 was acted upon.
> > > Transaction: Ticket created by michael.j.erickson at noaa.gov
> > > Queue: met_help
> > > Subject: BRIER_NCL and BRIER_NCU Description
> > > Owner: Nobody
> > > Requestors: michael.j.erickson at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=81842
> >
> > >
> > >
> > > Good morning,
> > >
> > > I have a rather specific question regarding the BRIER_NCL and
BRIER_NCU
> > > output values from grid_stat. Looking at table 7.11 wiithin
> > > http://www.dtcenter.org/met/users/docs/users_guide/MET_
> > > Users_Guide_v6.0.pdf,
> > > I'm not sure quantitatively what is meant by "normal upper and
lower
> > > confidence limits." Are these confidence limits controlled by
the alpha
> > > variable? If possible, I am looking for the 95% confidence
interval
> from
> > > BRIER_NCL and BRIER_NCU.
> > >
> > > Thank you again!
> > >
> > > Mike
> > >
> > > --
> > > Michael J. Erickson
> > >
> > > Research Scientist
> > > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > > NOAA/NWS/Weather Prediction Center
> > > Phone: 301-683-1546
> > >
> > >
> >
> >
>
>
> --
> Michael J. Erickson
>
> Research Scientist
> Cooperative Institute for Research in Environmental Sciences (CIRES)
> NOAA/NWS/Weather Prediction Center
> Phone: 301-683-1546
>
>
------------------------------------------------
Subject: BRIER_NCL and BRIER_NCU Description
From: Michael Erickson - NOAA Affiliate
Time: Fri Sep 01 11:33:59 2017
Thanks John,
That makes perfect sense and the spatial problem is definitely tough!
Thank
you for everything.
Mike
On Fri, Sep 1, 2017 at 12:43 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Michael,
>
> Yes, that's correct. There is the assumption of IID
> (independent-identically-distributed), which is a bad assumption.
>
> We've discussed this over the years with statisticians here, and the
> suggested way of handling spatial dependence is using a spatial
block
> bootstrap instead. You'd break up the pairs into spatial blocks
where the
> assumption of IID is more reasonable and then bootstrap the groups
of pairs
> rather than the individual ones. But in practice, it's difficult to
apply
> this method generally across all the data that is processed by MET.
> There's no easy way to make good default decisions for spatial block
size.
> When you ask as statistician about these things, they typically
answer,
> "well it really depends on your data"... because it really does!
>
> This detail hasn't risen to the top of our development priorities.
And in
> practice, we seldom use the CI's computed by MET itself. Instead,
we run
> MET over many days and aggregate stats through time using METViewer.
In
> that case, we're addressing temporal uncertainty instead of spatial.
>
> I think being aware of poor IID assumption is great, and choosing a
lower
> alpha value (maybe 0.01 rather than 0.05) than you otherwise would
to
> account for it is wise.
>
> Thanks,
> John
>
>
>
> On Fri, Sep 1, 2017 at 9:50 AM, Michael Erickson - NOAA Affiliate
via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=81842 >
> >
> > Hi John,
> >
> > Thanks for your quick and helpful answer!
> >
> > So in the process of bootstrapping the individual matched pairs,
there is
> > an assumption that each spatial point is independent. Is that
correct?
> This
> > is an unavoidable problem, but I can probably be more cautious by
> > decreasing my alpha.
> >
> > Thanks again!
> >
> > Mike
> >
> > On Fri, Sep 1, 2017 at 11:33 AM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Mike,
> > >
> > > The MET output includes both parametric and non-parametric
confidence
> > > intervals for most statistics.
> > >
> > > The columns ending in _BCL and _BCU, which stand for "bootstrap
> > confidence
> > > lower and upper limits", contain the non-parametric estimates.
They
> are
> > > the result of bootstrapping the individual matched pairs. Since
the
> MET
> > > statistics tools (like point_stat and grid_stat) are run once
per
> output
> > > time, the confidence intervals listed in the output give an
estimate of
> > the
> > > spatial sampling uncertainty.
> > >
> > > The columns ending in _NCL and _NCU, which stand for "normal
confidence
> > > lower and upper limits", contain the parametric estimates.
Using the
> > word
> > > "normal" is really a misnomer. While many of the parametric
confidence
> > > intervals are based on assumptions of normality, some of them
are not.
> > The
> > > parametric confidence interval estimate was chosen separately
for each
> > > statistic, as determined by the statisticians in our group.
> > >
> > > To answer your question, yes, the alpha value is used to
determine the
> > > level of certainty you'd like to include. Setting alpha = 0.05
means a
> > 95%
> > > confidence interval. alpha = 0.01 means a 99% confidence
interval.
> The
> > > smaller alpha you choose, the more certain you'd like to be that
the
> true
> > > statistic will fall in the interval, meaning the wider the
confidence
> > > interval must be. You should be able to specify multiple alpha
values
> in
> > > the config file and check to see that with larger alphas, you
get
> > narrower
> > > CI's and with smaller alphas, you get wider CI's. But remember
that
> this
> > > is only an estimate of the spatial sampling uncertainty.
> > >
> > > The BRIER_NCL and BRIER_NCU values are computed by this function
in
> MET:
> > > double Nx2ContingencyTable::brier_ci_halfwidth(double alpha)
const
> > > Which is defined in this file:
> > > src/libcode/vx_statistics/contable_nx2.cc
> > >
> > > Listed in that code is the following reference for computing
Brier
> score
> > > confidence intervals:
> > >
> > > // Bradley, A.A, S.S. Schwartz, and T. Hashino, 2008:
> > > // Sampling Uncertainty and Confidence Intervals for the Brier
Score
> > > // and Brier Skill Score. Weather and Forecasting, 23, 992-
1006.
> > >
> > > Thanks,
> > > John Halley Gotway
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Sep 1, 2017 at 7:00 AM, Michael Erickson - NOAA
Affiliate via
> RT
> > <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Fri Sep 01 07:00:42 2017: Request 81842 was acted upon.
> > > > Transaction: Ticket created by michael.j.erickson at noaa.gov
> > > > Queue: met_help
> > > > Subject: BRIER_NCL and BRIER_NCU Description
> > > > Owner: Nobody
> > > > Requestors: michael.j.erickson at noaa.gov
> > > > Status: new
> > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=81842
> > >
> > > >
> > > >
> > > > Good morning,
> > > >
> > > > I have a rather specific question regarding the BRIER_NCL and
> BRIER_NCU
> > > > output values from grid_stat. Looking at table 7.11 wiithin
> > > > http://www.dtcenter.org/met/users/docs/users_guide/MET_
> > > > Users_Guide_v6.0.pdf,
> > > > I'm not sure quantitatively what is meant by "normal upper and
lower
> > > > confidence limits." Are these confidence limits controlled by
the
> alpha
> > > > variable? If possible, I am looking for the 95% confidence
interval
> > from
> > > > BRIER_NCL and BRIER_NCU.
> > > >
> > > > Thank you again!
> > > >
> > > > Mike
> > > >
> > > > --
> > > > Michael J. Erickson
> > > >
> > > > Research Scientist
> > > > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > > > NOAA/NWS/Weather Prediction Center
> > > > Phone: 301-683-1546
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > Michael J. Erickson
> >
> > Research Scientist
> > Cooperative Institute for Research in Environmental Sciences
(CIRES)
> > NOAA/NWS/Weather Prediction Center
> > Phone: 301-683-1546
> >
> >
>
>
--
Michael J. Erickson
Research Scientist
Cooperative Institute for Research in Environmental Sciences (CIRES)
NOAA/NWS/Weather Prediction Center
Phone: 301-683-1546
------------------------------------------------
More information about the Met_help
mailing list