[Met_help] [rt.rap.ucar.edu #83486] History for Plotting CIs for RMSE
John Halley Gotway via RT
met_help at ucar.edu
Fri Jan 5 15:48:22 MST 2018
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi,
I have been using Grid Stat to generate Fractions Brier Score statistics.
I use Grid Stat to create NetCDF matched pairs files containing observed
fractions, and then I run Grid Stat a second time to calculate RMSE with
the CNT line type to compare the observed fractions directly to forecast
probabilities. From previous email exchanges with John and Tressa, RMSE
from the CNT line type will be equivalent to FBS if the data have already
been transformed into fractions.
However, I've been having trouble plotting confidence intervals for RMSE.
I noticed in METViewer that the CNT line type is not an option under
aggregation statistics, and the only way I am able to make plots of RMSE is
by selecting the "none" option with agg stat.
In my Grid Stat Config file (attached), I only have CNT turned on. To plot
CIs for RMSE, do I also need to turn on the SL1L2 line type?
Thanks!
Ben
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon>
Virus-free.
www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Plotting CIs for RMSE
From: John Halley Gotway
Time: Thu Jan 04 15:21:12 2018
Ben,
The continuous statistics line type (CNT) is *derivable* from the
SL1L2
partial sums line type. If you're using aggregation logic, you'd get
CI's
by bootstrapping the aggregation of SL1L2 in CNT. So for
bootstrapping the
aggregation, yes, you'd also need to SL1L2 partial sums.
In the METViewer GUI, you'd use:
- In "Statistics" box, select "Aggregation statistics" radio button
- In dropdown, select "SL1L2" option
- Select 1000 (or so) "Bootsrapping replications"
- In the "Series Formatting" box in the "Conf Interval" column, select
the
"boot" option
The other alternative is to just plot the mean of the daily RMSE
values and
put a confidence interval on that mean value.
In the METViewer GUI, you'd use:
- In "Statistics" box, select "Summary" radio button
- In dropdown, just leave it as "None"
- For "Plot statistic", change from "Median" to "Mean"
- In the "Series Formatting" box in the "Conf Interval" column, select
the
"std" option for the standard error
So that's two slightly different CI methods.
In general, when you're writing out CNT lines, I'd suggest also
writing out
the SL1L2 lines.
Thanks,
John
On Thu, Jan 4, 2018 at 9:25 AM, Benjamin Blake - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> Thu Jan 04 09:25:11 2018: Request 83486 was acted upon.
> Transaction: Ticket created by benjamin.blake at noaa.gov
> Queue: met_help
> Subject: Plotting CIs for RMSE
> Owner: Nobody
> Requestors: benjamin.blake at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83486 >
>
>
> Hi,
>
> I have been using Grid Stat to generate Fractions Brier Score
statistics.
> I use Grid Stat to create NetCDF matched pairs files containing
observed
> fractions, and then I run Grid Stat a second time to calculate RMSE
with
> the CNT line type to compare the observed fractions directly to
forecast
> probabilities. From previous email exchanges with John and Tressa,
RMSE
> from the CNT line type will be equivalent to FBS if the data have
already
> been transformed into fractions.
>
> However, I've been having trouble plotting confidence intervals for
RMSE.
> I noticed in METViewer that the CNT line type is not an option under
> aggregation statistics, and the only way I am able to make plots of
RMSE is
> by selecting the "none" option with agg stat.
>
> In my Grid Stat Config file (attached), I only have CNT turned on.
To plot
> CIs for RMSE, do I also need to turn on the SL1L2 line type?
>
> Thanks!
> Ben
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_
> source=link&utm_campaign=sig-
email&utm_content=webmail&utm_term=icon>
> Virus-free.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_
> source=link&utm_campaign=sig-
email&utm_content=webmail&utm_term=link>
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
>
------------------------------------------------
Subject: Plotting CIs for RMSE
From: Benjamin Blake - NOAA Affiliate
Time: Fri Jan 05 06:08:55 2018
Hi John,
That makes sense, thanks for the clarification!
I am interested in doing bootstrapping so I will generate the SL1L2
partial
sums in addition to the CNT lines.
Ben
On Thu, Jan 4, 2018 at 5:21 PM, John Halley Gotway via RT
<met_help at ucar.edu
> wrote:
> Ben,
>
> The continuous statistics line type (CNT) is *derivable* from the
SL1L2
> partial sums line type. If you're using aggregation logic, you'd
get CI's
> by bootstrapping the aggregation of SL1L2 in CNT. So for
bootstrapping the
> aggregation, yes, you'd also need to SL1L2 partial sums.
>
> In the METViewer GUI, you'd use:
> - In "Statistics" box, select "Aggregation statistics" radio button
> - In dropdown, select "SL1L2" option
> - Select 1000 (or so) "Bootsrapping replications"
> - In the "Series Formatting" box in the "Conf Interval" column,
select the
> "boot" option
>
> The other alternative is to just plot the mean of the daily RMSE
values and
> put a confidence interval on that mean value.
>
> In the METViewer GUI, you'd use:
> - In "Statistics" box, select "Summary" radio button
> - In dropdown, just leave it as "None"
> - For "Plot statistic", change from "Median" to "Mean"
> - In the "Series Formatting" box in the "Conf Interval" column,
select the
> "std" option for the standard error
>
> So that's two slightly different CI methods.
>
> In general, when you're writing out CNT lines, I'd suggest also
writing out
> the SL1L2 lines.
>
> Thanks,
> John
>
>
> On Thu, Jan 4, 2018 at 9:25 AM, Benjamin Blake - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Jan 04 09:25:11 2018: Request 83486 was acted upon.
> > Transaction: Ticket created by benjamin.blake at noaa.gov
> > Queue: met_help
> > Subject: Plotting CIs for RMSE
> > Owner: Nobody
> > Requestors: benjamin.blake at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83486 >
> >
> >
> > Hi,
> >
> > I have been using Grid Stat to generate Fractions Brier Score
statistics.
> > I use Grid Stat to create NetCDF matched pairs files containing
observed
> > fractions, and then I run Grid Stat a second time to calculate
RMSE with
> > the CNT line type to compare the observed fractions directly to
forecast
> > probabilities. From previous email exchanges with John and
Tressa, RMSE
> > from the CNT line type will be equivalent to FBS if the data have
already
> > been transformed into fractions.
> >
> > However, I've been having trouble plotting confidence intervals
for RMSE.
> > I noticed in METViewer that the CNT line type is not an option
under
> > aggregation statistics, and the only way I am able to make plots
of RMSE
> is
> > by selecting the "none" option with agg stat.
> >
> > In my Grid Stat Config file (attached), I only have CNT turned on.
To
> plot
> > CIs for RMSE, do I also need to turn on the SL1L2 line type?
> >
> > Thanks!
> > Ben
> >
> > <https://www.avast.com/sig-email?utm_medium=email&utm_
> > source=link&utm_campaign=sig-
email&utm_content=webmail&utm_term=icon>
> > Virus-free.
> > www.avast.com
> > <https://www.avast.com/sig-email?utm_medium=email&utm_
> > source=link&utm_campaign=sig-
email&utm_content=webmail&utm_term=link>
> > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list