[Met_help] [rt.rap.ucar.edu #95174] History for a couple probabilistic verification questions
John Halley Gotway via RT
met_help at ucar.edu
Thu May 7 09:47:35 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi MET team,
Hope you’re all doing well!
I’ve got a couple questions that have come up while doing some probabilistic verification using MET.
1. Is there a way to get bootstrapped (or other) confidence intervals for the area under the ROC curve (or the individual components of the POD/POFD table needed to calculate it)? I’ve looked around and don’t find it in the documentation, but I wouldn’t be surprised if it’s lurking in there somewhere and I’m just not doing something right.
2. I noticed that in the calculation of BSS_SMPL (the Brier skill score with respect to the sample climatology), the value that gets produced in the PSTD files does not match what you would get from calculating (1 – BRIER/BASER), even though my understanding is that these should be the same. The value reported in the table for BSS_SMPL is always a bit lower than what that calculation would yield. Is there something else going on in the calculation of BSS_SMPL (like to account for varying sample climatology across the grid, or something else?)
I’m using MET9.0.1 on Linux, though I got the same results from earlier versions of MET.
Thanks much for your help!
Russ
—
Russ S. Schumacher
Director, Colorado Climate Center
Colorado State Climatologist
Associate Professor, Department of Atmospheric Science
Colorado State University
e-mail: russ.schumacher at colostate.edu<mailto:russ.schumacher at colostate.edu>
phone: 970.491.8084
web: https://www.atmos.colostate.edu/people/faculty/schumacher/
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: a couple probabilistic verification questions
From: George McCabe
Time: Wed May 06 08:46:05 2020
Hi Russ,
I assigned this ticket to John Halley Gotway. He will get back to you
soon with a response.
Thanks,
George
On Wed May 06 08:23:47 2020, russ.schumacher at colostate.edu wrote:
> Hi MET team,
>
> Hope you’re all doing well!
>
> I’ve got a couple questions that have come up while doing some
> probabilistic verification using MET.
>
>
> 1. Is there a way to get bootstrapped (or other) confidence
intervals
> for the area under the ROC curve (or the individual components of
the
> POD/POFD table needed to calculate it)? I’ve looked around and
don’t
> find it in the documentation, but I wouldn’t be surprised if it’s
> lurking in there somewhere and I’m just not doing something right.
> 2. I noticed that in the calculation of BSS_SMPL (the Brier skill
> score with respect to the sample climatology), the value that gets
> produced in the PSTD files does not match what you would get from
> calculating (1 – BRIER/BASER), even though my understanding is that
> these should be the same. The value reported in the table for
BSS_SMPL
> is always a bit lower than what that calculation would yield. Is
there
> something else going on in the calculation of BSS_SMPL (like to
> account for varying sample climatology across the grid, or something
> else?)
>
> I’m using MET9.0.1 on Linux, though I got the same results from
> earlier versions of MET.
>
> Thanks much for your help!
>
> Russ
>
>
>
>
> —
> Russ S. Schumacher
> Director, Colorado Climate Center
> Colorado State Climatologist
> Associate Professor, Department of Atmospheric Science
> Colorado State University
> e-mail:
> russ.schumacher at colostate.edu<mailto:russ.schumacher at colostate.edu>
> phone: 970.491.8084
> web: https://www.atmos.colostate.edu/people/faculty/schumacher/
>
>
------------------------------------------------
Subject: a couple probabilistic verification questions
From: John Halley Gotway
Time: Wed May 06 14:58:42 2020
Hi Russ,
I see you have some questions about probabilistic verification using
MET.
BSS_SMPL is the Brier Skill Score relative to the sample climatology,
and
it's computation can be found starting on line 600 of the file
contable_nx2.cc:
https://github.com/NCAR/MET/blob/master_v9.0/met/src/libcode/vx_statistics/contable_nx2.cc
That code includes this reference to an equation from Wilks:
Reference: Equation 8.43, page 340 in Wilks, 3rd Ed.
And it's computed from the reliability, resolution, and
uncertainty statistics as:
bss = ( res - rel ) / unc;
I haven't worked through these equations to prove that that's not the
same
as computing 1 - BRIER/BASER, but I suspect that it is not. If you do
think
they should match, please let me know, and I'll ask one of the
statisticians to take a look.
Your second question is about confidence intervals for the ROC_AUC
statistic. That statistic is included in the PSTD line type produced
by
Point-Stat and Grid-Stat tools when evaluating probabilistic
forecasts. And
as you probably know, the line type does NOT include parametric or
non-parametric confidence intervals for ROC_AUC. So MET itself
currently
includes no functionality for this.
However, the METviewer database and display system IS able to add CI's
to
ROC_AUC, but there are 2 choices for how to do this. I used some
sample
data available at this instance of METviewer:
http://www.dtcenter.org/met/metviewer/metviewer1.jsp
And I've attached 2 plots and 2 corresponding XML files. Each plot
shows a
time series of ROC_AUC stats for the probability of 3-hour precip >
0.1"
over CONUS. The red and blue lines are for probabilities derived from
two
different ensembles. The gray line is their pairwise difference.
(1) Compute the mean of the ROC_AUC statistics for each lead time
across
multiple initializations and compute the CI as the standard error of
the
mean. This is quick and dirty.
(2) Use 1000 bootstrapping replications to the aggregate the ROC_AUC
statistic for each lead time across multiple initializations. Compute
the
CI's using the bootstrap percentile method. This is much, much slower
but
generally considered to be more statistically robust.
Anywhere the gray pairwise difference line does not include the 0-line
(which is at all lead times), the difference is considered to be
statistically significant. Comparing those, the pattern of the results
remain the same but the details differ.
I included the XML so you could easily go to the METviewer URL, click
the
"Load XML" button in the top-right corner, and see how I set this up.
So I'd recommend using METviewer for this. And if you guys haven't
had the
opportunity to use METviewer at CSU, perhaps this is a good
opportunity to
try it out. METviewer also include the ability to compute "scorecards"
which provide a more concise way of measuring and displaying
statistically
significant pairwise differences across multiple variables, levels,
regions, and statistics. Figured that may also be of interest.
Hope that helps clarify.
Thanks,
John
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #95174] a couple probabilistic verification questions
From: Russ Schumacher
Time: Thu May 07 08:19:12 2020
Thanks, John - much appreciated. Once I can go back to the office
I'll dig through Wilks and remind myself of all of the connections
between these different variables - I know he walks through all the
different ways you can formulate these equations.
And the METViewer thing sounds like a cool solution - I'll check it
out as soon as I can and let you know if I have other questions.
Thanks!
Russ
On 5/6/20, 2:58 PM, "John Halley Gotway via RT" <met_help at ucar.edu>
wrote:
Hi Russ,
I see you have some questions about probabilistic verification
using MET.
BSS_SMPL is the Brier Skill Score relative to the sample
climatology, and
it's computation can be found starting on line 600 of the file
contable_nx2.cc:
https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNCAR%2FMET%2Fblob%2Fmaster_v9.0%2Fmet%2Fsrc%2Flibcode%2Fvx_statistics%2Fcontable_nx2.cc&data=02%7C01%7Cruss.schumacher%40colostate.edu%7C734e6d4b89f84a46ee4e08d7f20043cf%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637243955264938764&sdata=SU024BhgQaicm68DfObFb1HO6QxIsMhlnYaj23AZJoI%3D&reserved=0
That code includes this reference to an equation from Wilks:
Reference: Equation 8.43, page 340 in Wilks, 3rd Ed.
And it's computed from the reliability, resolution, and
uncertainty statistics as:
bss = ( res - rel ) / unc;
I haven't worked through these equations to prove that that's not
the same
as computing 1 - BRIER/BASER, but I suspect that it is not. If you
do think
they should match, please let me know, and I'll ask one of the
statisticians to take a look.
Your second question is about confidence intervals for the ROC_AUC
statistic. That statistic is included in the PSTD line type
produced by
Point-Stat and Grid-Stat tools when evaluating probabilistic
forecasts. And
as you probably know, the line type does NOT include parametric or
non-parametric confidence intervals for ROC_AUC. So MET itself
currently
includes no functionality for this.
However, the METviewer database and display system IS able to add
CI's to
ROC_AUC, but there are 2 choices for how to do this. I used some
sample
data available at this instance of METviewer:
https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dtcenter.org%2Fmet%2Fmetviewer%2Fmetviewer1.jsp&data=02%7C01%7Cruss.schumacher%40colostate.edu%7C734e6d4b89f84a46ee4e08d7f20043cf%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637243955264938764&sdata=JbkMCooXrXUFaTvK4alu9%2BXcTPysxNND4CCQENdbUSU%3D&reserved=0
And I've attached 2 plots and 2 corresponding XML files. Each
plot shows a
time series of ROC_AUC stats for the probability of 3-hour precip
> 0.1"
over CONUS. The red and blue lines are for probabilities derived
from two
different ensembles. The gray line is their pairwise difference.
(1) Compute the mean of the ROC_AUC statistics for each lead time
across
multiple initializations and compute the CI as the standard error
of the
mean. This is quick and dirty.
(2) Use 1000 bootstrapping replications to the aggregate the
ROC_AUC
statistic for each lead time across multiple initializations.
Compute the
CI's using the bootstrap percentile method. This is much, much
slower but
generally considered to be more statistically robust.
Anywhere the gray pairwise difference line does not include the 0-
line
(which is at all lead times), the difference is considered to be
statistically significant. Comparing those, the pattern of the
results
remain the same but the details differ.
I included the XML so you could easily go to the METviewer URL,
click the
"Load XML" button in the top-right corner, and see how I set this
up.
So I'd recommend using METviewer for this. And if you guys
haven't had the
opportunity to use METviewer at CSU, perhaps this is a good
opportunity to
try it out. METviewer also include the ability to compute
"scorecards"
which provide a more concise way of measuring and displaying
statistically
significant pairwise differences across multiple variables,
levels,
regions, and statistics. Figured that may also be of interest.
Hope that helps clarify.
Thanks,
John
------------------------------------------------
Subject: a couple probabilistic verification questions
From: John Halley Gotway
Time: Thu May 07 09:47:34 2020
Russ,
I'll go ahead and resolve this ticket. But let us know if any more
questions arise. And let us know if you'd like help getting METviewer
installed somewhere.
Thanks,
John
On Thu, May 7, 2020 at 8:19 AM Russ Schumacher via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95174 >
>
> Thanks, John - much appreciated. Once I can go back to the office
I'll
> dig through Wilks and remind myself of all of the connections
between these
> different variables - I know he walks through all the different ways
you
> can formulate these equations.
>
> And the METViewer thing sounds like a cool solution - I'll check it
out as
> soon as I can and let you know if I have other questions. Thanks!
>
> Russ
>
>
>
> On 5/6/20, 2:58 PM, "John Halley Gotway via RT" <met_help at ucar.edu>
> wrote:
>
> Hi Russ,
>
> I see you have some questions about probabilistic verification
using
> MET.
>
> BSS_SMPL is the Brier Skill Score relative to the sample
climatology,
> and
> it's computation can be found starting on line 600 of the file
> contable_nx2.cc:
>
>
https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FNCAR%2FMET%2Fblob%2Fmaster_v9.0%2Fmet%2Fsrc%2Flibcode%2Fvx_statistics%2Fcontable_nx2.cc&data=02%7C01%7Cruss.schumacher%40colostate.edu%7C734e6d4b89f84a46ee4e08d7f20043cf%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637243955264938764&sdata=SU024BhgQaicm68DfObFb1HO6QxIsMhlnYaj23AZJoI%3D&reserved=0
>
> That code includes this reference to an equation from Wilks:
> Reference: Equation 8.43, page 340 in Wilks, 3rd Ed.
>
> And it's computed from the reliability, resolution, and
> uncertainty statistics as:
> bss = ( res - rel ) / unc;
>
> I haven't worked through these equations to prove that that's
not the
> same
> as computing 1 - BRIER/BASER, but I suspect that it is not. If
you do
> think
> they should match, please let me know, and I'll ask one of the
> statisticians to take a look.
>
> Your second question is about confidence intervals for the
ROC_AUC
> statistic. That statistic is included in the PSTD line type
produced by
> Point-Stat and Grid-Stat tools when evaluating probabilistic
> forecasts. And
> as you probably know, the line type does NOT include parametric
or
> non-parametric confidence intervals for ROC_AUC. So MET itself
> currently
> includes no functionality for this.
>
> However, the METviewer database and display system IS able to
add CI's
> to
> ROC_AUC, but there are 2 choices for how to do this. I used some
sample
> data available at this instance of METviewer:
>
>
https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dtcenter.org%2Fmet%2Fmetviewer%2Fmetviewer1.jsp&data=02%7C01%7Cruss.schumacher%40colostate.edu%7C734e6d4b89f84a46ee4e08d7f20043cf%7Cafb58802ff7a4bb1ab21367ff2ecfc8b%7C0%7C0%7C637243955264938764&sdata=JbkMCooXrXUFaTvK4alu9%2BXcTPysxNND4CCQENdbUSU%3D&reserved=0
>
> And I've attached 2 plots and 2 corresponding XML files. Each
plot
> shows a
> time series of ROC_AUC stats for the probability of 3-hour
precip >
> 0.1"
> over CONUS. The red and blue lines are for probabilities
derived from
> two
> different ensembles. The gray line is their pairwise
difference.
>
> (1) Compute the mean of the ROC_AUC statistics for each lead
time
> across
> multiple initializations and compute the CI as the standard
error of
> the
> mean. This is quick and dirty.
>
> (2) Use 1000 bootstrapping replications to the aggregate the
ROC_AUC
> statistic for each lead time across multiple initializations.
Compute
> the
> CI's using the bootstrap percentile method. This is much, much
slower
> but
> generally considered to be more statistically robust.
>
> Anywhere the gray pairwise difference line does not include the
0-line
> (which is at all lead times), the difference is considered to be
> statistically significant. Comparing those, the pattern of the
results
> remain the same but the details differ.
>
> I included the XML so you could easily go to the METviewer URL,
click
> the
> "Load XML" button in the top-right corner, and see how I set
this up.
>
> So I'd recommend using METviewer for this. And if you guys
haven't
> had the
> opportunity to use METviewer at CSU, perhaps this is a good
> opportunity to
> try it out. METviewer also include the ability to compute
"scorecards"
> which provide a more concise way of measuring and displaying
> statistically
> significant pairwise differences across multiple variables,
levels,
> regions, and statistics. Figured that may also be of interest.
>
> Hope that helps clarify.
>
> Thanks,
> John
>
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list