[Met_help] [rt.rap.ucar.edu #86962] History for Add statistical significance (box/whisker) to METviewer scores plot?
John Halley Gotway via RT
met_help at ucar.edu
Mon Sep 17 09:18:34 MDT 2018
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello. Is there a METviewer example (png & xml) I can look up showing
how to add the statistical significance to a scores plot? I see this:
http://dtcenter.org/met/metviewer/doc/plot.html
but other than bootstrapping replication=1 means no bootstrapping, I'm
not sure how to set up the rest.
Attached are 1) an FVS plot showing what I'd like to see from the
METviewer plot and 2) a pair of METviewer png/xml (GSS - same as the top
graph in fvs.gif), on which I'd like to add the statistical significance.
Thank you very much for your help!
Ying
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Add statistical significance (box/whisker) to METviewer scores plot?
From: John Halley Gotway
Time: Fri Sep 14 10:10:45 2018
Hi Ying,
I see you have a question about how to add statistical significance to
your
plots using METviewer. Thanks for sending along the sample plots/xml
file. That helps a lot. I logged on to NCEP VPN and regenerated the
plot
you sent.
METviewer supports two statistical derivation methods...
(1) plot the mean of the daily stats with confidence intervals defined
by
the "standard error"... that uses the standard deviation to add
confidence
intervals and assumes normality
(2) plot the aggregated statistics with confidence intervals defined
by
bootstrapping that aggregation
To test, I tried method (1) and it works fine for some of the
categorical
stats, like BASER and FMEAN. However, applying it to GSS results in
this
error:
==== Start Rscript error ==== Error in if (0 < sum(listDataInd)) { :
missing value where TRUE/FALSE needed Execution halted ==== End
Rscript
error ====
For high thresholds, GSS is undefined and that missing data is causing
this
error message. When I restrict the plot to the 2 lowest thresholds
for
which there are always events, then method (1) works fine.
But you asked about method (2). Here are the details for applying
bootstrap confidence intervals...
- In the "Statistics" box make sure you have "Aggregation Statistics"
selected... and you do.
- In the "Statistics" box in the dropdown menu select the line type to
be
aggregated. For GSS, you aggregate CTC lines, which is what you have
selected.
- In the "Statistics" box select the number of bootstrapping
replicates,
using in general, at least 1000. I changed your "1" to "1000".
- Note that if you want to make your plot exactly repeatable in
subsequent
runs, set the boostrapping seed to any number.
- In the "Series Formatting" table in the "Conf Interval" column,
select
"boot".
Some other suggestions...
- The default alpha value is 0.05 for 95% confidence interval. You
can
change that in the "Common" tab. Setting alpha = 0.01 would yield a
99% CI.
- On the "Common" tab, select "Y1 Stagger Points". That prevents the
CI
bars from being plotted on top of each other.
Bootstrapping 1000 times takes longer to generate the plot. The
resulting
plot is attached as "plot_20180914_155508.xml/.png".
To really compare the difference of these lines, I'd suggest plotting
their
pairwise difference:
- In the bottom-left corner, select "+Add Derived Curve".
- In the pop-up, choose GFS in the first box and FV3GFS in the second,
make
sure "DIFF" is selected, and click "Create Derived Curve".
- We now have a new entry in the "Series Formatting" table. Select
"boot"
in the "Conf Interval" column. Change the color to black and line
width to
3. And set "Show Significance" to Yes.
- In "Series Formatting" set the "Legend Text" to FV3GFS, GFS, and
GFS-FV3GFS.
- In the "Y1" tab get rid of the "Y1 Limits" setting so that R can fit
all
the data on the plot.
- In the "Formatting" tab, set the "Plot Script Commands" to draw a
gray
dashed reference line at 0:
abline(h=0,col="gray",lty=2,lwd=3)
The resulting plot is attached as "plot_20180914_160322.xml/.png".
This shows that at GFS has a statistically significant higher GSS
score
than FV3GFS at the lowest threshold. But the results are not
statistically
significant at higher thresholds. Visually, when the confidence
interval
for the pairwise difference line does not include 0, the difference is
significant and METviewer plots the CI in bold. That's the effect of
turning on "Show Significance".
Hope that helps clarify.
Thanks,
John
On Thu, Sep 13, 2018 at 12:15 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> Thu Sep 13 12:15:22 2018: Request 86962 was acted upon.
> Transaction: Ticket created by ying.lin at noaa.gov
> Queue: met_help
> Subject: Add statistical significance (box/whisker) to
METviewer
> scores plot?
> Owner: Nobody
> Requestors: ying.lin at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86962 >
>
>
> Hello. Is there a METviewer example (png & xml) I can look up
showing
> how to add the statistical significance to a scores plot? I see
this:
>
> http://dtcenter.org/met/metviewer/doc/plot.html
>
> but other than bootstrapping replication=1 means no bootstrapping,
I'm
> not sure how to set up the rest.
>
> Attached are 1) an FVS plot showing what I'd like to see from the
> METviewer plot and 2) a pair of METviewer png/xml (GSS - same as the
top
> graph in fvs.gif), on which I'd like to add the statistical
significance.
>
> Thank you very much for your help!
>
> Ying
>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: Add statistical significance (box/whisker) to METviewer scores plot?
From: John Halley Gotway
Time: Fri Sep 14 10:15:47 2018
Ying,
Sorry, I had my conclusion backwards. FV3GFS is better than GFS at
the
lowest threshold in a statistically significant way. Looking at
trends
(not statistical significance), FV3GFS generally does better for lower
thresholds and GFS does better for higher thresholds... which I'm sure
you
already determined.
John
On Fri, Sep 14, 2018 at 10:11 AM The RT System itself via RT <
met_help at ucar.edu> wrote:
>
> Fri Sep 14 10:10:46 2018: Request 86962 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by RT_System
> Queue: met_help
> Subject: Add statistical significance (box/whisker) to
METviewer
> scores plot?
> Owner: johnhg
> Requestors: ying.lin at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86962 >
>
>
> This transaction appears to have no content
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #86962] Add statistical significance (box/whisker) to METviewer scores plot?
From: Ying Lin
Time: Fri Sep 14 10:20:29 2018
Thanks John! yes I was just reading that line and double-checking the
plots ... Ying
On 09/14/2018 12:15 PM, John Halley Gotway via RT wrote:
> Ying,
>
> Sorry, I had my conclusion backwards. FV3GFS is better than GFS at
the
> lowest threshold in a statistically significant way. Looking at
trends
> (not statistical significance), FV3GFS generally does better for
lower
> thresholds and GFS does better for higher thresholds... which I'm
sure you
> already determined.
>
> John
>
> On Fri, Sep 14, 2018 at 10:11 AM The RT System itself via RT <
> met_help at ucar.edu> wrote:
>
>> Fri Sep 14 10:10:46 2018: Request 86962 was acted upon.
>> Transaction: Given to johnhg (John Halley Gotway) by RT_System
>> Queue: met_help
>> Subject: Add statistical significance (box/whisker) to
METviewer
>> scores plot?
>> Owner: johnhg
>> Requestors: ying.lin at noaa.gov
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86962 >
>>
>>
>> This transaction appears to have no content
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: Add statistical significance (box/whisker) to METviewer scores plot?
From: Ying Lin
Time: Fri Sep 14 11:47:23 2018
[Binbin, Tracey - you've both asked about this recently. Here's the
answer plus a lot of good stuff from John]
Hi John, thank you so much for the solutions and a lot of great bonus
information and subtleties like slightly off-setting the CI bars.
Hurray for METviewer!
I was able to add the CI bars now following the steps you outlined
(didn't set "Conf Interval" in Series Formatting before, aha). Couple
of
follow-up questions:
1) You said that "if you want to make your plot exactly repeatable in
subsequent runs, set the boostrapping seed to any number". Does that
mean the same number (e.g. '3') in the initial and subsequent runs, or
any number each time (e.g. '3' at first, then '50', '12' etc.?)
2) Great stuff in "show significance". For the difference line, I
think
you meant that the difference is significant if the entire CI line
clears the 0 line cleanly (staying above or below the 0 line), and
that
makes sense. Then I turned on the "show significance" for both the
FV3GFS and GFS score lines themselves (see attached), and it seems
that
the CI bars for them shows significance for all thresholds below 2".
How should we interpret this different significance? (back in the FVS
days I remember Geoff DiMego would say that the difference was
significant at a threshold if the distance between the two scores are
greater than half of the CI).
Thanks again,
Ying
On 09/14/2018 12:15 PM, John Halley Gotway via RT wrote:
> Ying,
>
> Sorry, I had my conclusion backwards. FV3GFS is better than GFS at
the
> lowest threshold in a statistically significant way. Looking at
trends
> (not statistical significance), FV3GFS generally does better for
lower
> thresholds and GFS does better for higher thresholds... which I'm
sure you
> already determined.
>
> John
On 09/14/2018 12:10 PM, John Halley Gotway via RT wrote:
> Hi Ying,
>
> I see you have a question about how to add statistical significance
to your
> plots using METviewer. Thanks for sending along the sample
plots/xml
> file. That helps a lot. I logged on to NCEP VPN and regenerated
the plot
> you sent.
>
> METviewer supports two statistical derivation methods...
> (1) plot the mean of the daily stats with confidence intervals
defined by
> the "standard error"... that uses the standard deviation to add
confidence
> intervals and assumes normality
> (2) plot the aggregated statistics with confidence intervals defined
by
> bootstrapping that aggregation
>
> To test, I tried method (1) and it works fine for some of the
categorical
> stats, like BASER and FMEAN. However, applying it to GSS results in
this
> error:
>
> ==== Start Rscript error ==== Error in if (0 < sum(listDataInd)) { :
> missing value where TRUE/FALSE needed Execution halted ==== End
Rscript
> error ====
>
> For high thresholds, GSS is undefined and that missing data is
causing this
> error message. When I restrict the plot to the 2 lowest thresholds
for
> which there are always events, then method (1) works fine.
>
> But you asked about method (2). Here are the details for applying
> bootstrap confidence intervals...
> - In the "Statistics" box make sure you have "Aggregation
Statistics"
> selected... and you do.
> - In the "Statistics" box in the dropdown menu select the line type
to be
> aggregated. For GSS, you aggregate CTC lines, which is what you
have
> selected.
> - In the "Statistics" box select the number of bootstrapping
replicates,
> using in general, at least 1000. I changed your "1" to "1000".
> - Note that if you want to make your plot exactly repeatable in
subsequent
> runs, set the boostrapping seed to any number.
> - In the "Series Formatting" table in the "Conf Interval" column,
select
> "boot".
>
> Some other suggestions...
> - The default alpha value is 0.05 for 95% confidence interval. You
can
> change that in the "Common" tab. Setting alpha = 0.01 would yield a
99% CI.
> - On the "Common" tab, select "Y1 Stagger Points". That prevents
the CI
> bars from being plotted on top of each other.
>
> Bootstrapping 1000 times takes longer to generate the plot. The
resulting
> plot is attached as "plot_20180914_155508.xml/.png".
>
> To really compare the difference of these lines, I'd suggest
plotting their
> pairwise difference:
>
> - In the bottom-left corner, select "+Add Derived Curve".
> - In the pop-up, choose GFS in the first box and FV3GFS in the
second, make
> sure "DIFF" is selected, and click "Create Derived Curve".
> - We now have a new entry in the "Series Formatting" table. Select
"boot"
> in the "Conf Interval" column. Change the color to black and line
width to
> 3. And set "Show Significance" to Yes.
> - In "Series Formatting" set the "Legend Text" to FV3GFS, GFS, and
> GFS-FV3GFS.
> - In the "Y1" tab get rid of the "Y1 Limits" setting so that R can
fit all
> the data on the plot.
> - In the "Formatting" tab, set the "Plot Script Commands" to draw a
gray
> dashed reference line at 0:
> abline(h=0,col="gray",lty=2,lwd=3)
>
> The resulting plot is attached as "plot_20180914_160322.xml/.png".
>
> This shows that at GFS has a statistically significant higher GSS
score
> than FV3GFS at the lowest threshold. But the results are not
statistically
> significant at higher thresholds. Visually, when the confidence
interval
> for the pairwise difference line does not include 0, the difference
is
> significant and METviewer plots the CI in bold. That's the effect
of
> turning on "Show Significance".
>
> Hope that helps clarify.
>
> Thanks,
> John
>
>
> On Thu, Sep 13, 2018 at 12:15 PM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> Thu Sep 13 12:15:22 2018: Request 86962 was acted upon.
>> Transaction: Ticket created by ying.lin at noaa.gov
>> Queue: met_help
>> Subject: Add statistical significance (box/whisker) to
METviewer
>> scores plot?
>> Owner: Nobody
>> Requestors: ying.lin at noaa.gov
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86962 >
>>
>>
>> Hello. Is there a METviewer example (png & xml) I can look up
showing
>> how to add the statistical significance to a scores plot? I see
this:
>>
>> http://dtcenter.org/met/metviewer/doc/plot.html
>>
>> but other than bootstrapping replication=1 means no bootstrapping,
I'm
>> not sure how to set up the rest.
>>
>> Attached are 1) an FVS plot showing what I'd like to see from the
>> METviewer plot and 2) a pair of METviewer png/xml (GSS - same as
the top
>> graph in fvs.gif), on which I'd like to add the statistical
significance.
>>
>> Thank you very much for your help!
>>
>> Ying
>>
>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
Subject: Add statistical significance (box/whisker) to METviewer scores plot?
From: John Halley Gotway
Time: Fri Sep 14 14:47:04 2018
Ying,
(1) Yes, to make the plot exactly repeatable on the same system, set
the
bootstrapping seed and keep it the same in subsequent runs. I usually
just
set it to "1", but you could use any number. When you reuse that
number
for *EXACTLY* the same plot, you should get exactly the same result.
And
you can check the actual number by looking at the "R Data" tab. In
the
file, there's a column with the statistic value listed. The next two
columns are the lower and upper CI values. If the seed is not set,
those
CI's will change slightly each time... since bootstrapping is an
inherently
random process. Setting the seed should yield exactly the same CI's
on
subsequent runs that use the same seed value.
FYI, you can also increase the number of bootstrapping replicates.
Generally, the larger the number of replicates, the more stable the CI
estimate is. But the bigger it is, the slower it runs. This is
pretty
frustrating advice, but statisticians say... keep increasing the
number of
replicates and rerunning until the CI stops changing. In the DTC, we
often
use 1,000. But if time is no object, you could use 5,000.
And a note about the "Confidence Interval Method" option for
bootstrapping. The percentile method (perc) is fastest, but
generally, the
bias-corrected and adjusted (bca) method is slower but yields more
accurate
results.
When generating plots interactively with METviewer, I often use only
100
replicates with the perc method... at least until I get the plot
looking
good. But when automating graphics with the batch engine, I switch to
using a more accurate 1000 or 5000 replicates with the bca method.
(2) Sorry for stating it incorrectly. Yes, the results are only
significant when 0 falls outside of the CI for the pairwise difference
line. Turning "Show Significance" on really only makes sense for the
pairwise difference line... but you can use it however you'd like.
The
logic is basically... for any CI that doesn't touch 0, plot it as
bold.
Thanks,
John
On Fri, Sep 14, 2018 at 11:47 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86962 >
>
> [Binbin, Tracey - you've both asked about this recently. Here's the
> answer plus a lot of good stuff from John]
>
> Hi John, thank you so much for the solutions and a lot of great
bonus
> information and subtleties like slightly off-setting the CI bars.
> Hurray for METviewer!
>
> I was able to add the CI bars now following the steps you outlined
> (didn't set "Conf Interval" in Series Formatting before, aha).
Couple of
> follow-up questions:
>
> 1) You said that "if you want to make your plot exactly repeatable
in
> subsequent runs, set the boostrapping seed to any number". Does
that
> mean the same number (e.g. '3') in the initial and subsequent runs,
or
> any number each time (e.g. '3' at first, then '50', '12' etc.?)
>
> 2) Great stuff in "show significance". For the difference line, I
think
> you meant that the difference is significant if the entire CI line
> clears the 0 line cleanly (staying above or below the 0 line), and
that
> makes sense. Then I turned on the "show significance" for both the
> FV3GFS and GFS score lines themselves (see attached), and it seems
that
> the CI bars for them shows significance for all thresholds below 2".
> How should we interpret this different significance? (back in the
FVS
> days I remember Geoff DiMego would say that the difference was
> significant at a threshold if the distance between the two scores
are
> greater than half of the CI).
>
> Thanks again,
>
> Ying
>
> On 09/14/2018 12:15 PM, John Halley Gotway via RT wrote:
> > Ying,
> >
> > Sorry, I had my conclusion backwards. FV3GFS is better than GFS
at the
> > lowest threshold in a statistically significant way. Looking at
trends
> > (not statistical significance), FV3GFS generally does better for
lower
> > thresholds and GFS does better for higher thresholds... which I'm
sure
> you
> > already determined.
> >
> > John
>
> On 09/14/2018 12:10 PM, John Halley Gotway via RT wrote:
> > Hi Ying,
> >
> > I see you have a question about how to add statistical
significance to
> your
> > plots using METviewer. Thanks for sending along the sample
plots/xml
> > file. That helps a lot. I logged on to NCEP VPN and regenerated
the
> plot
> > you sent.
> >
> > METviewer supports two statistical derivation methods...
> > (1) plot the mean of the daily stats with confidence intervals
defined by
> > the "standard error"... that uses the standard deviation to add
> confidence
> > intervals and assumes normality
> > (2) plot the aggregated statistics with confidence intervals
defined by
> > bootstrapping that aggregation
> >
> > To test, I tried method (1) and it works fine for some of the
categorical
> > stats, like BASER and FMEAN. However, applying it to GSS results
in this
> > error:
> >
> > ==== Start Rscript error ==== Error in if (0 < sum(listDataInd)) {
:
> > missing value where TRUE/FALSE needed Execution halted ==== End
Rscript
> > error ====
> >
> > For high thresholds, GSS is undefined and that missing data is
causing
> this
> > error message. When I restrict the plot to the 2 lowest
thresholds for
> > which there are always events, then method (1) works fine.
> >
> > But you asked about method (2). Here are the details for applying
> > bootstrap confidence intervals...
> > - In the "Statistics" box make sure you have "Aggregation
Statistics"
> > selected... and you do.
> > - In the "Statistics" box in the dropdown menu select the line
type to be
> > aggregated. For GSS, you aggregate CTC lines, which is what you
have
> > selected.
> > - In the "Statistics" box select the number of bootstrapping
replicates,
> > using in general, at least 1000. I changed your "1" to "1000".
> > - Note that if you want to make your plot exactly repeatable in
> subsequent
> > runs, set the boostrapping seed to any number.
> > - In the "Series Formatting" table in the "Conf Interval" column,
select
> > "boot".
> >
> > Some other suggestions...
> > - The default alpha value is 0.05 for 95% confidence interval.
You can
> > change that in the "Common" tab. Setting alpha = 0.01 would yield
a 99%
> CI.
> > - On the "Common" tab, select "Y1 Stagger Points". That prevents
the CI
> > bars from being plotted on top of each other.
> >
> > Bootstrapping 1000 times takes longer to generate the plot. The
> resulting
> > plot is attached as "plot_20180914_155508.xml/.png".
> >
> > To really compare the difference of these lines, I'd suggest
plotting
> their
> > pairwise difference:
> >
> > - In the bottom-left corner, select "+Add Derived Curve".
> > - In the pop-up, choose GFS in the first box and FV3GFS in the
second,
> make
> > sure "DIFF" is selected, and click "Create Derived Curve".
> > - We now have a new entry in the "Series Formatting" table.
Select
> "boot"
> > in the "Conf Interval" column. Change the color to black and line
width
> to
> > 3. And set "Show Significance" to Yes.
> > - In "Series Formatting" set the "Legend Text" to FV3GFS, GFS, and
> > GFS-FV3GFS.
> > - In the "Y1" tab get rid of the "Y1 Limits" setting so that R can
fit
> all
> > the data on the plot.
> > - In the "Formatting" tab, set the "Plot Script Commands" to draw
a gray
> > dashed reference line at 0:
> > abline(h=0,col="gray",lty=2,lwd=3)
> >
> > The resulting plot is attached as "plot_20180914_160322.xml/.png".
> >
> > This shows that at GFS has a statistically significant higher GSS
score
> > than FV3GFS at the lowest threshold. But the results are not
> statistically
> > significant at higher thresholds. Visually, when the confidence
interval
> > for the pairwise difference line does not include 0, the
difference is
> > significant and METviewer plots the CI in bold. That's the effect
of
> > turning on "Show Significance".
> >
> > Hope that helps clarify.
> >
> > Thanks,
> > John
> >
> >
> > On Thu, Sep 13, 2018 at 12:15 PM Ying Lin via RT
<met_help at ucar.edu>
> wrote:
> >
> >> Thu Sep 13 12:15:22 2018: Request 86962 was acted upon.
> >> Transaction: Ticket created by ying.lin at noaa.gov
> >> Queue: met_help
> >> Subject: Add statistical significance (box/whisker) to
METviewer
> >> scores plot?
> >> Owner: Nobody
> >> Requestors: ying.lin at noaa.gov
> >> Status: new
> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86962
> >
> >>
> >>
> >> Hello. Is there a METviewer example (png & xml) I can look up
showing
> >> how to add the statistical significance to a scores plot? I see
this:
> >>
> >> http://dtcenter.org/met/metviewer/doc/plot.html
> >>
> >> but other than bootstrapping replication=1 means no
bootstrapping, I'm
> >> not sure how to set up the rest.
> >>
> >> Attached are 1) an FVS plot showing what I'd like to see from the
> >> METviewer plot and 2) a pair of METviewer png/xml (GSS - same as
the top
> >> graph in fvs.gif), on which I'd like to add the statistical
> significance.
> >>
> >> Thank you very much for your help!
> >>
> >> Ying
> >>
> >>
> >> --
> >> Ying Lin
> >> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
> >> NCWCP Cubicle No. 2015
> >> Ying.Lin at noaa.gov
> >>
> >>
> >>
> >>
>
> --
> Ying Lin
> NCEP/EMC/Verification, Post-processing and Product Generation Branch
> NCWCP Cubicle No. 2015
> Ying.Lin at noaa.gov
>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #86962] Add statistical significance (box/whisker) to METviewer scores plot?
From: Ying Lin
Time: Fri Sep 14 17:38:37 2018
Thank you very much John! I really appreciate your spending the time
to
explain the finer points. I was able to plot with the bias-corrected
and adjusted (bca) method - much higher CI lines at higher thresholds,
but got two more significant pair-wise differences at lower
thresholds.
I'm learning a lot from this one ticket. Please feel free to close
this
one now at your convenience.
Have a great weekend -
Ying
On 09/14/2018 04:47 PM, John Halley Gotway via RT wrote:
> Ying,
>
> (1) Yes, to make the plot exactly repeatable on the same system, set
the
> bootstrapping seed and keep it the same in subsequent runs. I
usually just
> set it to "1", but you could use any number. When you reuse that
number
> for *EXACTLY* the same plot, you should get exactly the same result.
And
> you can check the actual number by looking at the "R Data" tab. In
the
> file, there's a column with the statistic value listed. The next
two
> columns are the lower and upper CI values. If the seed is not set,
those
> CI's will change slightly each time... since bootstrapping is an
inherently
> random process. Setting the seed should yield exactly the same CI's
on
> subsequent runs that use the same seed value.
>
> FYI, you can also increase the number of bootstrapping replicates.
> Generally, the larger the number of replicates, the more stable the
CI
> estimate is. But the bigger it is, the slower it runs. This is
pretty
> frustrating advice, but statisticians say... keep increasing the
number of
> replicates and rerunning until the CI stops changing. In the DTC,
we often
> use 1,000. But if time is no object, you could use 5,000.
>
> And a note about the "Confidence Interval Method" option for
> bootstrapping. The percentile method (perc) is fastest, but
generally, the
> bias-corrected and adjusted (bca) method is slower but yields more
accurate
> results.
>
> When generating plots interactively with METviewer, I often use only
100
> replicates with the perc method... at least until I get the plot
looking
> good. But when automating graphics with the batch engine, I switch
to
> using a more accurate 1000 or 5000 replicates with the bca method.
>
> (2) Sorry for stating it incorrectly. Yes, the results are only
> significant when 0 falls outside of the CI for the pairwise
difference
> line. Turning "Show Significance" on really only makes sense for
the
> pairwise difference line... but you can use it however you'd like.
The
> logic is basically... for any CI that doesn't touch 0, plot it as
bold.
>
> Thanks,
> John
>
> On Fri, Sep 14, 2018 at 11:47 AM Ying Lin via RT <met_help at ucar.edu>
wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86962 >
>>
>> [Binbin, Tracey - you've both asked about this recently. Here's
the
>> answer plus a lot of good stuff from John]
>>
>> Hi John, thank you so much for the solutions and a lot of great
bonus
>> information and subtleties like slightly off-setting the CI bars.
>> Hurray for METviewer!
>>
>> I was able to add the CI bars now following the steps you outlined
>> (didn't set "Conf Interval" in Series Formatting before, aha).
Couple of
>> follow-up questions:
>>
>> 1) You said that "if you want to make your plot exactly repeatable
in
>> subsequent runs, set the boostrapping seed to any number". Does
that
>> mean the same number (e.g. '3') in the initial and subsequent runs,
or
>> any number each time (e.g. '3' at first, then '50', '12' etc.?)
>>
>> 2) Great stuff in "show significance". For the difference line, I
think
>> you meant that the difference is significant if the entire CI line
>> clears the 0 line cleanly (staying above or below the 0 line), and
that
>> makes sense. Then I turned on the "show significance" for both the
>> FV3GFS and GFS score lines themselves (see attached), and it seems
that
>> the CI bars for them shows significance for all thresholds below
2".
>> How should we interpret this different significance? (back in the
FVS
>> days I remember Geoff DiMego would say that the difference was
>> significant at a threshold if the distance between the two scores
are
>> greater than half of the CI).
>>
>> Thanks again,
>>
>> Ying
>>
>> On 09/14/2018 12:15 PM, John Halley Gotway via RT wrote:
>>> Ying,
>>>
>>> Sorry, I had my conclusion backwards. FV3GFS is better than GFS
at the
>>> lowest threshold in a statistically significant way. Looking at
trends
>>> (not statistical significance), FV3GFS generally does better for
lower
>>> thresholds and GFS does better for higher thresholds... which I'm
sure
>> you
>>> already determined.
>>>
>>> John
>> On 09/14/2018 12:10 PM, John Halley Gotway via RT wrote:
>>> Hi Ying,
>>>
>>> I see you have a question about how to add statistical
significance to
>> your
>>> plots using METviewer. Thanks for sending along the sample
plots/xml
>>> file. That helps a lot. I logged on to NCEP VPN and regenerated
the
>> plot
>>> you sent.
>>>
>>> METviewer supports two statistical derivation methods...
>>> (1) plot the mean of the daily stats with confidence intervals
defined by
>>> the "standard error"... that uses the standard deviation to add
>> confidence
>>> intervals and assumes normality
>>> (2) plot the aggregated statistics with confidence intervals
defined by
>>> bootstrapping that aggregation
>>>
>>> To test, I tried method (1) and it works fine for some of the
categorical
>>> stats, like BASER and FMEAN. However, applying it to GSS results
in this
>>> error:
>>>
>>> ==== Start Rscript error ==== Error in if (0 < sum(listDataInd)) {
:
>>> missing value where TRUE/FALSE needed Execution halted ==== End
Rscript
>>> error ====
>>>
>>> For high thresholds, GSS is undefined and that missing data is
causing
>> this
>>> error message. When I restrict the plot to the 2 lowest
thresholds for
>>> which there are always events, then method (1) works fine.
>>>
>>> But you asked about method (2). Here are the details for applying
>>> bootstrap confidence intervals...
>>> - In the "Statistics" box make sure you have "Aggregation
Statistics"
>>> selected... and you do.
>>> - In the "Statistics" box in the dropdown menu select the line
type to be
>>> aggregated. For GSS, you aggregate CTC lines, which is what you
have
>>> selected.
>>> - In the "Statistics" box select the number of bootstrapping
replicates,
>>> using in general, at least 1000. I changed your "1" to "1000".
>>> - Note that if you want to make your plot exactly repeatable in
>> subsequent
>>> runs, set the boostrapping seed to any number.
>>> - In the "Series Formatting" table in the "Conf Interval" column,
select
>>> "boot".
>>>
>>> Some other suggestions...
>>> - The default alpha value is 0.05 for 95% confidence interval.
You can
>>> change that in the "Common" tab. Setting alpha = 0.01 would yield
a 99%
>> CI.
>>> - On the "Common" tab, select "Y1 Stagger Points". That prevents
the CI
>>> bars from being plotted on top of each other.
>>>
>>> Bootstrapping 1000 times takes longer to generate the plot. The
>> resulting
>>> plot is attached as "plot_20180914_155508.xml/.png".
>>>
>>> To really compare the difference of these lines, I'd suggest
plotting
>> their
>>> pairwise difference:
>>>
>>> - In the bottom-left corner, select "+Add Derived Curve".
>>> - In the pop-up, choose GFS in the first box and FV3GFS in the
second,
>> make
>>> sure "DIFF" is selected, and click "Create Derived Curve".
>>> - We now have a new entry in the "Series Formatting" table.
Select
>> "boot"
>>> in the "Conf Interval" column. Change the color to black and line
width
>> to
>>> 3. And set "Show Significance" to Yes.
>>> - In "Series Formatting" set the "Legend Text" to FV3GFS, GFS, and
>>> GFS-FV3GFS.
>>> - In the "Y1" tab get rid of the "Y1 Limits" setting so that R can
fit
>> all
>>> the data on the plot.
>>> - In the "Formatting" tab, set the "Plot Script Commands" to draw
a gray
>>> dashed reference line at 0:
>>> abline(h=0,col="gray",lty=2,lwd=3)
>>>
>>> The resulting plot is attached as "plot_20180914_160322.xml/.png".
>>>
>>> This shows that at GFS has a statistically significant higher GSS
score
>>> than FV3GFS at the lowest threshold. But the results are not
>> statistically
>>> significant at higher thresholds. Visually, when the confidence
interval
>>> for the pairwise difference line does not include 0, the
difference is
>>> significant and METviewer plots the CI in bold. That's the effect
of
>>> turning on "Show Significance".
>>>
>>> Hope that helps clarify.
>>>
>>> Thanks,
>>> John
>>>
>>>
>>> On Thu, Sep 13, 2018 at 12:15 PM Ying Lin via RT
<met_help at ucar.edu>
>> wrote:
>>>> Thu Sep 13 12:15:22 2018: Request 86962 was acted upon.
>>>> Transaction: Ticket created by ying.lin at noaa.gov
>>>> Queue: met_help
>>>> Subject: Add statistical significance (box/whisker) to
METviewer
>>>> scores plot?
>>>> Owner: Nobody
>>>> Requestors: ying.lin at noaa.gov
>>>> Status: new
>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86962
>>>>
>>>> Hello. Is there a METviewer example (png & xml) I can look up
showing
>>>> how to add the statistical significance to a scores plot? I see
this:
>>>>
>>>> http://dtcenter.org/met/metviewer/doc/plot.html
>>>>
>>>> but other than bootstrapping replication=1 means no
bootstrapping, I'm
>>>> not sure how to set up the rest.
>>>>
>>>> Attached are 1) an FVS plot showing what I'd like to see from the
>>>> METviewer plot and 2) a pair of METviewer png/xml (GSS - same as
the top
>>>> graph in fvs.gif), on which I'd like to add the statistical
>> significance.
>>>> Thank you very much for your help!
>>>>
>>>> Ying
>>>>
>>>>
>>>> --
>>>> Ying Lin
>>>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>>>> NCWCP Cubicle No. 2015
>>>> Ying.Lin at noaa.gov
>>>>
>>>>
>>>>
>>>>
>> --
>> Ying Lin
>> NCEP/EMC/Verification, Post-processing and Product Generation
Branch
>> NCWCP Cubicle No. 2015
>> Ying.Lin at noaa.gov
>>
>>
>>
>>
--
Ying Lin
NCEP/EMC/Verification, Post-processing and Product Generation Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov
------------------------------------------------
More information about the Met_help
mailing list