[Met_help] [rt.rap.ucar.edu #98415] History for StatAnalysis aggregate and Bootstrap
John Halley Gotway via RT
met_help at ucar.edu
Wed Feb 17 10:49:13 MST 2021
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
MET Help,
I'm trying to aggregate RMSE statistics with the bootstrap confidence
interval from a time series of GridStat CNT output. I can get bootstrapped
CI when running GridStat for individual times, however, after running
stat_analysis to aggregate, the Bootstrap values are still NA. I must be
missing something to make this work, and hoped you could point me in the
right direction.
Some details:
MET Version
9.1
StatAnalysis Job
"-job aggregate_stat -line_type SL1L2 -out_line_type CNT -by VX_MASK -by
FCST_VAR -by FCST_LEAD -by FCST_LEV -out_stat ./aggregate_job.stat"
I turned on bootstrapping in the StatAnalaysis Config file
boot = {
interval = PCTILE;
rep_prop = 1.0;
n_rep = 1000;
rng = "mt19937";
seed = "";
}
Attached are my StatAnalysis config file and two STAT files in case that is
helpful.
Respectfully,
Brian Blaylock
NRC Postdoc
Marine Meteorology Division
Naval Research Laboratory
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: StatAnalysis aggregate and Bootstrap
From: George McCabe
Time: Wed Jan 27 17:02:03 2021
Hi Brian,
I assigned this ticket to John Halley Gotway. He will get back to you
soon with a response.
Thanks,
George
On Wed Jan 27 12:32:24 2021, brian.blaylock.ctr at nrlmry.navy.mil wrote:
> MET Help,
>
>
>
> I'm trying to aggregate RMSE statistics with the bootstrap
confidence
> interval from a time series of GridStat CNT output. I can get
bootstrapped
> CI when running GridStat for individual times, however, after
running
> stat_analysis to aggregate, the Bootstrap values are still NA. I
must be
> missing something to make this work, and hoped you could point me in
the
> right direction.
>
>
>
> Some details:
>
>
>
> MET Version
>
> 9.1
>
>
>
> StatAnalysis Job
>
> "-job aggregate_stat -line_type SL1L2 -out_line_type CNT -by VX_MASK
-by
> FCST_VAR -by FCST_LEAD -by FCST_LEV -out_stat ./aggregate_job.stat"
>
>
>
> I turned on bootstrapping in the StatAnalaysis Config file
>
> boot = {
>
> interval = PCTILE;
>
> rep_prop = 1.0;
>
> n_rep = 1000;
>
> rng = "mt19937";
>
> seed = "";
>
> }
>
>
>
> Attached are my StatAnalysis config file and two STAT files in case
that is
> helpful.
>
>
>
> Respectfully,
>
>
>
> Brian Blaylock
>
> NRC Postdoc
>
> Marine Meteorology Division
>
> Naval Research Laboratory
>
>
>
------------------------------------------------
Subject: StatAnalysis aggregate and Bootstrap
From: John Halley Gotway
Time: Wed Jan 27 17:35:25 2021
Hello Brian,
I see that you're asking about bootstrap confidence intervals in
Stat-Analysis. The short answer is no, it's not supported.
Here's the longer answer. The MET statistics tools (like Grid-Stat and
Point-Stat) perform spatial aggregations of statistics by
bootstrapping the
aggregation of the fcst/obs matched pairs. The Stat-Analysis tool can
also
compute bootstrap CI's BUT ONLY WHEN aggregating the MPR lines from
the
Point-Stat tool.
What you're trying to do is bootstrap the temporal aggregation of
SL1L2
partial sums when computing CNT statistics. And Stat-Analysis does not
include that logic. So why not? It certainly is doable and a sensible
thing
to do. In fact, here's a GitHub issue about it from 10 years ago!
https://github.com/dtcenter/MET/issues/59
Over those last 10 years, we've been using METviewer to do exactly
that...
bootstrapping temporal aggregations of contingency table counts and
partial
sums. So if possible, I'd recommend using METviewer to do this. If
that's
not sufficient or possible for some reason, I'd recommend coordinating
with
Tara Jensen, the METplus project manager to consider pushing this
feature
request higher up in the queue.
Hopefully that helps clarify.
Thanks,
John Halley Gotway
On Wed, Jan 27, 2021 at 5:02 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98415 >
>
> Hi Brian,
>
> I assigned this ticket to John Halley Gotway. He will get back to
you soon
> with a response.
>
> Thanks,
> George
>
> On Wed Jan 27 12:32:24 2021, brian.blaylock.ctr at nrlmry.navy.mil
wrote:
> > MET Help,
> >
> >
> >
> > I'm trying to aggregate RMSE statistics with the bootstrap
confidence
> > interval from a time series of GridStat CNT output. I can get
> bootstrapped
> > CI when running GridStat for individual times, however, after
running
> > stat_analysis to aggregate, the Bootstrap values are still NA. I
must be
> > missing something to make this work, and hoped you could point me
in the
> > right direction.
> >
> >
> >
> > Some details:
> >
> >
> >
> > MET Version
> >
> > 9.1
> >
> >
> >
> > StatAnalysis Job
> >
> > "-job aggregate_stat -line_type SL1L2 -out_line_type CNT -by
VX_MASK -by
> > FCST_VAR -by FCST_LEAD -by FCST_LEV -out_stat
./aggregate_job.stat"
> >
> >
> >
> > I turned on bootstrapping in the StatAnalaysis Config file
> >
> > boot = {
> >
> > interval = PCTILE;
> >
> > rep_prop = 1.0;
> >
> > n_rep = 1000;
> >
> > rng = "mt19937";
> >
> > seed = "";
> >
> > }
> >
> >
> >
> > Attached are my StatAnalysis config file and two STAT files in
case that
> is
> > helpful.
> >
> >
> >
> > Respectfully,
> >
> >
> >
> > Brian Blaylock
> >
> > NRC Postdoc
> >
> > Marine Meteorology Division
> >
> > Naval Research Laboratory
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: StatAnalysis aggregate and Bootstrap
From: Blaylock, Dr. Brian, Contractor, Code 7531
Time: Wed Jan 27 18:09:30 2021
Thanks for the explanation John. I'll take a look at MET Viewer and
see what I
can do with that.
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Wednesday, January 27, 2021 4:35 PM
To: Blaylock, Dr. Brian, Contractor, Code 7531
<brian.blaylock.ctr at nrlmry.navy.mil>
Subject: Re: [rt.rap.ucar.edu #98415] StatAnalysis aggregate and
Bootstrap
Hello Brian,
I see that you're asking about bootstrap confidence intervals in
Stat-Analysis. The short answer is no, it's not supported.
Here's the longer answer. The MET statistics tools (like Grid-Stat and
Point-Stat) perform spatial aggregations of statistics by
bootstrapping the
aggregation of the fcst/obs matched pairs. The Stat-Analysis tool can
also
compute bootstrap CI's BUT ONLY WHEN aggregating the MPR lines from
the
Point-Stat tool.
What you're trying to do is bootstrap the temporal aggregation of
SL1L2
partial sums when computing CNT statistics. And Stat-Analysis does not
include
that logic. So why not? It certainly is doable and a sensible thing to
do. In
fact, here's a GitHub issue about it from 10 years ago!
https://github.com/dtcenter/MET/issues/59
Over those last 10 years, we've been using METviewer to do exactly
that...
bootstrapping temporal aggregations of contingency table counts and
partial
sums. So if possible, I'd recommend using METviewer to do this. If
that's not
sufficient or possible for some reason, I'd recommend coordinating
with Tara
Jensen, the METplus project manager to consider pushing this feature
request
higher up in the queue.
Hopefully that helps clarify.
Thanks,
John Halley Gotway
On Wed, Jan 27, 2021 at 5:02 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98415 >
>
> Hi Brian,
>
> I assigned this ticket to John Halley Gotway. He will get back to
you
> soon with a response.
>
> Thanks,
> George
>
> On Wed Jan 27 12:32:24 2021, brian.blaylock.ctr at nrlmry.navy.mil
wrote:
> > MET Help,
> >
> >
> >
> > I'm trying to aggregate RMSE statistics with the bootstrap
> > confidence interval from a time series of GridStat CNT output. I
can
> > get
> bootstrapped
> > CI when running GridStat for individual times, however, after
> > running stat_analysis to aggregate, the Bootstrap values are still
> > NA. I must be missing something to make this work, and hoped you
> > could point me in the right direction.
> >
> >
> >
> > Some details:
> >
> >
> >
> > MET Version
> >
> > 9.1
> >
> >
> >
> > StatAnalysis Job
> >
> > "-job aggregate_stat -line_type SL1L2 -out_line_type CNT -by
VX_MASK
> > -by FCST_VAR -by FCST_LEAD -by FCST_LEV -out_stat
./aggregate_job.stat"
> >
> >
> >
> > I turned on bootstrapping in the StatAnalaysis Config file
> >
> > boot = {
> >
> > interval = PCTILE;
> >
> > rep_prop = 1.0;
> >
> > n_rep = 1000;
> >
> > rng = "mt19937";
> >
> > seed = "";
> >
> > }
> >
> >
> >
> > Attached are my StatAnalysis config file and two STAT files in
case
> > that
> is
> > helpful.
> >
> >
> >
> > Respectfully,
> >
> >
> >
> > Brian Blaylock
> >
> > NRC Postdoc
> >
> > Marine Meteorology Division
> >
> > Naval Research Laboratory
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: StatAnalysis aggregate and Bootstrap
From: Blaylock, Dr. Brian, Contractor, Code 7531
Time: Fri Jan 29 15:18:46 2021
Hi Tara,
Just wanted to add another voice to this feature request for the MET
package:
the ability to aggregate confidence interval statistics with Stat
Analysis. I
talked to some of the other folks here at NRL who use MET and it
sounds like
MET Viewer is not something that we can use, (yet??).
Thanks!
Brian Blaylock
NRC Postdoc
Marine Meteorology Division
Naval Research Laboratory
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Wednesday, January 27, 2021 4:35 PM
To: Blaylock, Dr. Brian, Contractor, Code 7531
<brian.blaylock.ctr at nrlmry.navy.mil>
Subject: Re: [rt.rap.ucar.edu #98415] StatAnalysis aggregate and
Bootstrap
Hello Brian,
I see that you're asking about bootstrap confidence intervals in
Stat-Analysis. The short answer is no, it's not supported.
Here's the longer answer. The MET statistics tools (like Grid-Stat and
Point-Stat) perform spatial aggregations of statistics by
bootstrapping the
aggregation of the fcst/obs matched pairs. The Stat-Analysis tool can
also
compute bootstrap CI's BUT ONLY WHEN aggregating the MPR lines from
the
Point-Stat tool.
What you're trying to do is bootstrap the temporal aggregation of
SL1L2
partial sums when computing CNT statistics. And Stat-Analysis does not
include
that logic. So why not? It certainly is doable and a sensible thing to
do. In
fact, here's a GitHub issue about it from 10 years ago!
https://github.com/dtcenter/MET/issues/59
Over those last 10 years, we've been using METviewer to do exactly
that...
bootstrapping temporal aggregations of contingency table counts and
partial
sums. So if possible, I'd recommend using METviewer to do this. If
that's not
sufficient or possible for some reason, I'd recommend coordinating
with Tara
Jensen, the METplus project manager to consider pushing this feature
request
higher up in the queue.
Hopefully that helps clarify.
Thanks,
John Halley Gotway
On Wed, Jan 27, 2021 at 5:02 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98415 >
>
> Hi Brian,
>
> I assigned this ticket to John Halley Gotway. He will get back to
you
> soon with a response.
>
> Thanks,
> George
>
> On Wed Jan 27 12:32:24 2021, brian.blaylock.ctr at nrlmry.navy.mil
wrote:
> > MET Help,
> >
> >
> >
> > I'm trying to aggregate RMSE statistics with the bootstrap
> > confidence interval from a time series of GridStat CNT output. I
can
> > get
> bootstrapped
> > CI when running GridStat for individual times, however, after
> > running stat_analysis to aggregate, the Bootstrap values are still
> > NA. I must be missing something to make this work, and hoped you
> > could point me in the right direction.
> >
> >
> >
> > Some details:
> >
> >
> >
> > MET Version
> >
> > 9.1
> >
> >
> >
> > StatAnalysis Job
> >
> > "-job aggregate_stat -line_type SL1L2 -out_line_type CNT -by
VX_MASK
> > -by FCST_VAR -by FCST_LEAD -by FCST_LEV -out_stat
./aggregate_job.stat"
> >
> >
> >
> > I turned on bootstrapping in the StatAnalaysis Config file
> >
> > boot = {
> >
> > interval = PCTILE;
> >
> > rep_prop = 1.0;
> >
> > n_rep = 1000;
> >
> > rng = "mt19937";
> >
> > seed = "";
> >
> > }
> >
> >
> >
> > Attached are my StatAnalysis config file and two STAT files in
case
> > that
> is
> > helpful.
> >
> >
> >
> > Respectfully,
> >
> >
> >
> > Brian Blaylock
> >
> > NRC Postdoc
> >
> > Marine Meteorology Division
> >
> > Naval Research Laboratory
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: StatAnalysis aggregate and Bootstrap
From: John Halley Gotway
Time: Fri Jan 29 15:39:17 2021
Tara and Brian,
We are in the process of migrating METviewer over from calling R to
calling
Python instead. In that process, we're migrating functionality from
METviewer itself into the METplotpy and METcalcpy repositories. The
plan is
to enable METcalcpy to be callable directly, instead of only being
useful
via METviewer. So in a perfect world, this bootstrap aggregation
functionality could be addressed with the METcalcpy package.
But it also sure would be nice if the STAT-Analysis tool could just do
it
directly without needing to make calls to additional packages.
Tara, is it realistic to assign MET #59 to MET-10.0.0 or MET-10.1.0?
Could
this be done within the existing NRL funding framework?
Thanks,
John
On Fri, Jan 29, 2021 at 3:19 PM Blaylock, Dr. Brian, Contractor, Code
7531
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98415 >
>
> Hi Tara,
>
> Just wanted to add another voice to this feature request for the MET
> package:
> the ability to aggregate confidence interval statistics with Stat
> Analysis. I
> talked to some of the other folks here at NRL who use MET and it
sounds
> like
> MET Viewer is not something that we can use, (yet??).
>
> Thanks!
>
>
> Brian Blaylock
> NRC Postdoc
> Marine Meteorology Division
> Naval Research Laboratory
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Wednesday, January 27, 2021 4:35 PM
> To: Blaylock, Dr. Brian, Contractor, Code 7531
> <brian.blaylock.ctr at nrlmry.navy.mil>
> Subject: Re: [rt.rap.ucar.edu #98415] StatAnalysis aggregate and
Bootstrap
>
> Hello Brian,
>
> I see that you're asking about bootstrap confidence intervals in
> Stat-Analysis. The short answer is no, it's not supported.
>
> Here's the longer answer. The MET statistics tools (like Grid-Stat
and
> Point-Stat) perform spatial aggregations of statistics by
bootstrapping the
> aggregation of the fcst/obs matched pairs. The Stat-Analysis tool
can also
> compute bootstrap CI's BUT ONLY WHEN aggregating the MPR lines from
the
> Point-Stat tool.
>
> What you're trying to do is bootstrap the temporal aggregation of
SL1L2
> partial sums when computing CNT statistics. And Stat-Analysis does
not
> include
> that logic. So why not? It certainly is doable and a sensible thing
to do.
> In
> fact, here's a GitHub issue about it from 10 years ago!
> https://github.com/dtcenter/MET/issues/59
>
> Over those last 10 years, we've been using METviewer to do exactly
that...
> bootstrapping temporal aggregations of contingency table counts and
partial
> sums. So if possible, I'd recommend using METviewer to do this. If
that's
> not
> sufficient or possible for some reason, I'd recommend coordinating
with
> Tara
> Jensen, the METplus project manager to consider pushing this feature
> request
> higher up in the queue.
>
> Hopefully that helps clarify.
>
> Thanks,
> John Halley Gotway
>
> On Wed, Jan 27, 2021 at 5:02 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98415 >
> >
> > Hi Brian,
> >
> > I assigned this ticket to John Halley Gotway. He will get back to
you
> > soon with a response.
> >
> > Thanks,
> > George
> >
> > On Wed Jan 27 12:32:24 2021, brian.blaylock.ctr at nrlmry.navy.mil
wrote:
> > > MET Help,
> > >
> > >
> > >
> > > I'm trying to aggregate RMSE statistics with the bootstrap
> > > confidence interval from a time series of GridStat CNT output. I
can
> > > get
> > bootstrapped
> > > CI when running GridStat for individual times, however, after
> > > running stat_analysis to aggregate, the Bootstrap values are
still
> > > NA. I must be missing something to make this work, and hoped you
> > > could point me in the right direction.
> > >
> > >
> > >
> > > Some details:
> > >
> > >
> > >
> > > MET Version
> > >
> > > 9.1
> > >
> > >
> > >
> > > StatAnalysis Job
> > >
> > > "-job aggregate_stat -line_type SL1L2 -out_line_type CNT -by
VX_MASK
> > > -by FCST_VAR -by FCST_LEAD -by FCST_LEV -out_stat
./aggregate_job.stat"
> > >
> > >
> > >
> > > I turned on bootstrapping in the StatAnalaysis Config file
> > >
> > > boot = {
> > >
> > > interval = PCTILE;
> > >
> > > rep_prop = 1.0;
> > >
> > > n_rep = 1000;
> > >
> > > rng = "mt19937";
> > >
> > > seed = "";
> > >
> > > }
> > >
> > >
> > >
> > > Attached are my StatAnalysis config file and two STAT files in
case
> > > that
> > is
> > > helpful.
> > >
> > >
> > >
> > > Respectfully,
> > >
> > >
> > >
> > > Brian Blaylock
> > >
> > > NRC Postdoc
> > >
> > > Marine Meteorology Division
> > >
> > > Naval Research Laboratory
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
------------------------------------------------
More information about the Met_help
mailing list