[Met_help] [rt.rap.ucar.edu #90895] History for Stat Anal Boot Strapping
John Halley Gotway via RT
met_help at ucar.edu
Mon Jul 8 11:16:44 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
John or Julie, I need to get error bars on my stat_analysis output. It appears I would do this by setting the n_rep field in the stat_anal config file to 1000. The question is can I set this in my stat_anal JOB COMMAND LINE and not use the config_file? If the answer is no then can I add the -config option to the end of my command line like below:
/h/MET/bin/stat_analysis -lookin /h/data/global/WXQC/data/met/ptstat/mdlob_pairs/korea_4/TT -out /h/data/global/WXQC/data/met/summary/tt_korea_4_10p9 -tmp_dir /h/data/global/WXQC/d ata/met/tmp -job aggregate_stat -line_type MPR -out_line_type CNT -fcst_valid_beg 20180101_000000 -fcst_valid_end 20180531_180000 -by OBS_LEV -fcst_var TMP -obs_var TMP -by FCST_LE AD -dump_row /h/data/global/WXQC/data/met/filter_job -column_max -obs 50 -column_min -obs -70 -v 4 -config /h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_bootstrap
When I do this I get the following error:
DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file: /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file: /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
DEBUG 1: Creating STAT-Analysis output file "/h/data/global/WXQC/data/met/summary/tt_korea_4_10p9"
DEBUG 1: Default Config File: /h/MET/share/met/config/STATAnalysisConfig_default
DEBUG 1: User Config File: /h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_bootstrap
ERROR :
"RROR : recursive_envs() -> unable to open input file "/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_bootstrap
ERROR :
Thanks
Bob
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Stat Anal Boot Strapping
From: Julie Prestopnik
Time: Wed Jul 03 10:02:58 2019
Hi Bob.
Unfortunately, I cannot help you with this one. I have assigned the
ticket
to John. Please allow a few days for follow up.
Thanks,
Julie
On Wed, Jul 3, 2019 at 9:58 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Wed Jul 03 09:58:41 2019: Request 90895 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: Stat Anal Boot Strapping
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90895 >
>
>
> John or Julie, I need to get error bars on my stat_analysis output.
It
> appears I would do this by setting the n_rep field in the stat_anal
config
> file to 1000. The question is can I set this in my stat_anal JOB
COMMAND
> LINE and not use the config_file? If the answer is no then can I
add the
> -config option to the end of my command line like below:
>
> /h/MET/bin/stat_analysis -lookin
> /h/data/global/WXQC/data/met/ptstat/mdlob_pairs/korea_4/TT -out
> /h/data/global/WXQC/data/met/summary/tt_korea_4_10p9 -tmp_dir
> /h/data/global/WXQC/d ata/met/tmp -job aggregate_stat -line_type
MPR
> -out_line_type CNT -fcst_valid_beg 20180101_000000 -fcst_valid_end
> 20180531_180000 -by OBS_LEV -fcst_var TMP -obs_var TMP -by FCST_LE
AD
> -dump_row /h/data/global/WXQC/data/met/filter_job -column_max -obs
50
> -column_min -obs -70 -v 4 -config
>
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_bootstrap
>
> When I do this I get the following error:
>
> DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> DEBUG 1: Creating STAT-Analysis output file
> "/h/data/global/WXQC/data/met/summary/tt_korea_4_10p9"
> DEBUG 1: Default Config File:
> /h/MET/share/met/config/STATAnalysisConfig_default
> DEBUG 1: User Config File:
>
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_bootstrap
> ERROR :
> "RROR : recursive_envs() -> unable to open input file
>
"/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_bootstrap
> ERROR :
>
> Thanks
> Bob
>
>
------------------------------------------------
Subject: Stat Anal Boot Strapping
From: John Halley Gotway
Time: Thu Jul 04 07:04:47 2019
Bob,
The immediate error is that Stat-Analysis can’t read the config file
from
the path you provided:
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_bootstrap
I suspect you just have the path listed incorrectly there? Or perhaps
the
permissions are set in a way that make it unreadable by Stat-Analysis?
But there may be a bigger issue in the computation of bootstrap CI’s.
Stat-Analysis only computes them when processing matched pair (MPR)
input
lines. If that’s what your jobs are doing (reading MPR and computing
CNT,
for example) that should be fine.
But if you’re aggregating SL1L2 partial sums to compute CNT stats, for
example, bootstrapping won’t work. We’ve always used METviewer to
compute
bootstrap CIs for those aggregations. For completeness sake, that
functionality could also be added to Stat-Analysis. In fact the
development issue for this is over 7 years old!
https://github.com/NCAR/MET/issues/59
Since we’ve had METviewer, that feature request has never bubbled up
to the
top of our priority list. If this is needed in your workflow, please
let
management know so it can be included in a future DTC work plan.
Thanks,
John
On Wed, Jul 3, 2019 at 10:03 AM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90895 >
>
> Hi Bob.
>
> Unfortunately, I cannot help you with this one. I have assigned the
ticket
> to John. Please allow a few days for follow up.
>
> Thanks,
> Julie
>
> On Wed, Jul 3, 2019 at 9:58 AM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Jul 03 09:58:41 2019: Request 90895 was acted upon.
> > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > Queue: met_help
> > Subject: Stat Anal Boot Strapping
> > Owner: Nobody
> > Requestors: robert.craig.2 at us.af.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90895 >
> >
> >
> > John or Julie, I need to get error bars on my stat_analysis
output. It
> > appears I would do this by setting the n_rep field in the
stat_anal
> config
> > file to 1000. The question is can I set this in my stat_anal JOB
COMMAND
> > LINE and not use the config_file? If the answer is no then can I
add the
> > -config option to the end of my command line like below:
> >
> > /h/MET/bin/stat_analysis -lookin
> > /h/data/global/WXQC/data/met/ptstat/mdlob_pairs/korea_4/TT -out
> > /h/data/global/WXQC/data/met/summary/tt_korea_4_10p9 -tmp_dir
> > /h/data/global/WXQC/d ata/met/tmp -job aggregate_stat
-line_type MPR
> > -out_line_type CNT -fcst_valid_beg 20180101_000000 -fcst_valid_end
> > 20180531_180000 -by OBS_LEV -fcst_var TMP -obs_var TMP -by FCST_LE
AD
> > -dump_row /h/data/global/WXQC/data/met/filter_job -column_max
-obs 50
> > -column_min -obs -70 -v 4 -config
> >
>
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_bootstrap
> >
> > When I do this I get the following error:
> >
> > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> > /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> > DEBUG 1: Creating STAT-Analysis output file
> > "/h/data/global/WXQC/data/met/summary/tt_korea_4_10p9"
> > DEBUG 1: Default Config File:
> > /h/MET/share/met/config/STATAnalysisConfig_default
> > DEBUG 1: User Config File:
> >
>
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_bootstrap
> > ERROR :
> > "RROR : recursive_envs() -> unable to open input file
> >
>
"/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_bootstrap
> > ERROR :
> >
> > Thanks
> > Bob
> >
> >
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #90895] Stat Anal Boot Strapping
From: robert.craig.2 at us.af.mil
Time: Fri Jul 05 07:11:12 2019
Thanks John, The path to the config file was correct. I got the error
to go away by placing the job command in the config file instead of
the command line - maybe your not allowed to have it in two places.
As far as the confidence intervals, we can wait to see what MET Viewer
provides. The big thing of interest here is to be able to say one
model is significantly better than another. We have generated some
hypothesis testing code but would love to have MET be able to answer
this question. How do you guys calculate practical significance
versus true significance?
Thanks
Bob
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Thursday, July 4, 2019 8:05 AM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #90895] Stat Anal Boot
Strapping
Bob,
The immediate error is that Stat-Analysis can’t read the config file
from the path you provided:
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_bootstrap
I suspect you just have the path listed incorrectly there? Or perhaps
the permissions are set in a way that make it unreadable by Stat-
Analysis?
But there may be a bigger issue in the computation of bootstrap CI’s.
Stat-Analysis only computes them when processing matched pair (MPR)
input lines. If that’s what your jobs are doing (reading MPR and
computing CNT, for example) that should be fine.
But if you’re aggregating SL1L2 partial sums to compute CNT stats, for
example, bootstrapping won’t work. We’ve always used METviewer to
compute bootstrap CIs for those aggregations. For completeness sake,
that functionality could also be added to Stat-Analysis. In fact the
development issue for this is over 7 years old!
https://github.com/NCAR/MET/issues/59
Since we’ve had METviewer, that feature request has never bubbled up
to the top of our priority list. If this is needed in your workflow,
please let management know so it can be included in a future DTC work
plan.
Thanks,
John
On Wed, Jul 3, 2019 at 10:03 AM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90895 >
>
> Hi Bob.
>
> Unfortunately, I cannot help you with this one. I have assigned the
> ticket to John. Please allow a few days for follow up.
>
> Thanks,
> Julie
>
> On Wed, Jul 3, 2019 at 9:58 AM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Jul 03 09:58:41 2019: Request 90895 was acted upon.
> > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > Queue: met_help
> > Subject: Stat Anal Boot Strapping
> > Owner: Nobody
> > Requestors: robert.craig.2 at us.af.mil
> > Status: new
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90895 >
> >
> >
> > John or Julie, I need to get error bars on my stat_analysis
output. It
> > appears I would do this by setting the n_rep field in the
stat_anal
> config
> > file to 1000. The question is can I set this in my stat_anal JOB
> > COMMAND LINE and not use the config_file? If the answer is no
then
> > can I add the -config option to the end of my command line like
below:
> >
> > /h/MET/bin/stat_analysis -lookin
> > /h/data/global/WXQC/data/met/ptstat/mdlob_pairs/korea_4/TT -out
> > /h/data/global/WXQC/data/met/summary/tt_korea_4_10p9 -tmp_dir
> > /h/data/global/WXQC/d ata/met/tmp -job aggregate_stat
-line_type MPR
> > -out_line_type CNT -fcst_valid_beg 20180101_000000 -fcst_valid_end
> > 20180531_180000 -by OBS_LEV -fcst_var TMP -obs_var TMP -by FCST_LE
AD
> > -dump_row /h/data/global/WXQC/data/met/filter_job -column_max
-obs
> > 50 -column_min -obs -70 -v 4 -config
> >
>
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_boot
> strap
> >
> > When I do this I get the following error:
> >
> > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> > /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> > DEBUG 1: Creating STAT-Analysis output file
> > "/h/data/global/WXQC/data/met/summary/tt_korea_4_10p9"
> > DEBUG 1: Default Config File:
> > /h/MET/share/met/config/STATAnalysisConfig_default
> > DEBUG 1: User Config File:
> >
>
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_boot
> strap
> > ERROR :
> > "RROR : recursive_envs() -> unable to open input file
> >
>
"/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_boo
> tstrap
> > ERROR :
> >
> > Thanks
> > Bob
> >
> >
>
>
------------------------------------------------
Subject: Stat Anal Boot Strapping
From: John Halley Gotway
Time: Fri Jul 05 08:17:54 2019
Bob,
OK, I understand. We primarily use METviewer to compare the
performance
between two models. We do this by plotting the pairwise difference of
statistics for multiple variables and levels and looking for
statistically
significant differences. And we also create “scorecards” to quickly
summarize the performance across multiple variables, levels, lead
times,
and statistics.
Really the only thing in MET itself which directly compares
performance is
the GO index.
Thanks,
John
On Fri, Jul 5, 2019 at 7:11 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90895 >
>
> Thanks John, The path to the config file was correct. I got the
error to
> go away by placing the job command in the config file instead of the
> command line - maybe your not allowed to have it in two places. As
far as
> the confidence intervals, we can wait to see what MET Viewer
provides. The
> big thing of interest here is to be able to say one model is
significantly
> better than another. We have generated some hypothesis testing code
but
> would love to have MET be able to answer this question. How do you
guys
> calculate practical significance versus true significance?
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, July 4, 2019 8:05 AM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #90895] Stat Anal
Boot
> Strapping
>
> Bob,
>
> The immediate error is that Stat-Analysis can’t read the config file
from
> the path you provided:
>
>
>
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_bootstrap
>
> I suspect you just have the path listed incorrectly there? Or
perhaps the
> permissions are set in a way that make it unreadable by Stat-
Analysis?
>
> But there may be a bigger issue in the computation of bootstrap
CI’s.
> Stat-Analysis only computes them when processing matched pair (MPR)
input
> lines. If that’s what your jobs are doing (reading MPR and
computing CNT,
> for example) that should be fine.
>
> But if you’re aggregating SL1L2 partial sums to compute CNT stats,
for
> example, bootstrapping won’t work. We’ve always used METviewer to
compute
> bootstrap CIs for those aggregations. For completeness sake, that
> functionality could also be added to Stat-Analysis. In fact the
> development issue for this is over 7 years old!
>
> https://github.com/NCAR/MET/issues/59
>
> Since we’ve had METviewer, that feature request has never bubbled up
to
> the top of our priority list. If this is needed in your workflow,
please
> let management know so it can be included in a future DTC work plan.
>
> Thanks,
> John
>
> On Wed, Jul 3, 2019 at 10:03 AM Julie Prestopnik via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90895 >
> >
> > Hi Bob.
> >
> > Unfortunately, I cannot help you with this one. I have assigned
the
> > ticket to John. Please allow a few days for follow up.
> >
> > Thanks,
> > Julie
> >
> > On Wed, Jul 3, 2019 at 9:58 AM robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Jul 03 09:58:41 2019: Request 90895 was acted upon.
> > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > Queue: met_help
> > > Subject: Stat Anal Boot Strapping
> > > Owner: Nobody
> > > Requestors: robert.craig.2 at us.af.mil
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90895 >
> > >
> > >
> > > John or Julie, I need to get error bars on my stat_analysis
output.
> It
> > > appears I would do this by setting the n_rep field in the
stat_anal
> > config
> > > file to 1000. The question is can I set this in my stat_anal
JOB
> > > COMMAND LINE and not use the config_file? If the answer is no
then
> > > can I add the -config option to the end of my command line like
below:
> > >
> > > /h/MET/bin/stat_analysis -lookin
> > > /h/data/global/WXQC/data/met/ptstat/mdlob_pairs/korea_4/TT -out
> > > /h/data/global/WXQC/data/met/summary/tt_korea_4_10p9 -tmp_dir
> > > /h/data/global/WXQC/d ata/met/tmp -job aggregate_stat
-line_type MPR
> > > -out_line_type CNT -fcst_valid_beg 20180101_000000
-fcst_valid_end
> > > 20180531_180000 -by OBS_LEV -fcst_var TMP -obs_var TMP -by
FCST_LE
> AD
> > > -dump_row /h/data/global/WXQC/data/met/filter_job -column_max
-obs
> > > 50 -column_min -obs -70 -v 4 -config
> > >
> >
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_boot
> > strap
> > >
> > > When I do this I get the following error:
> > >
> > > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> > > /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> > > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> > > DEBUG 1: Creating STAT-Analysis output file
> > > "/h/data/global/WXQC/data/met/summary/tt_korea_4_10p9"
> > > DEBUG 1: Default Config File:
> > > /h/MET/share/met/config/STATAnalysisConfig_default
> > > DEBUG 1: User Config File:
> > >
> >
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_boot
> > strap
> > > ERROR :
> > > "RROR : recursive_envs() -> unable to open input file
> > >
> >
"/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_boo
> > tstrap
> > > ERROR :
> > >
> > > Thanks
> > > Bob
> > >
> > >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #90895] Stat Anal Boot Strapping
From: robert.craig.2 at us.af.mil
Time: Fri Jul 05 08:26:34 2019
John, one last question on this issue - in your score card how do you
come up with practical significance versus "true" significance?
Thanks
Bob
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Friday, July 5, 2019 9:18 AM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #90895] Stat Anal
Boot Strapping
Bob,
OK, I understand. We primarily use METviewer to compare the
performance between two models. We do this by plotting the pairwise
difference of statistics for multiple variables and levels and looking
for statistically significant differences. And we also create
“scorecards” to quickly summarize the performance across multiple
variables, levels, lead times, and statistics.
Really the only thing in MET itself which directly compares
performance is the GO index.
Thanks,
John
On Fri, Jul 5, 2019 at 7:11 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90895 >
>
> Thanks John, The path to the config file was correct. I got the
error
> to go away by placing the job command in the config file instead of
> the command line - maybe your not allowed to have it in two places.
> As far as the confidence intervals, we can wait to see what MET
Viewer
> provides. The big thing of interest here is to be able to say one
> model is significantly better than another. We have generated some
> hypothesis testing code but would love to have MET be able to answer
> this question. How do you guys calculate practical significance
versus true significance?
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Thursday, July 4, 2019 8:05 AM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> <robert.craig.2 at us.af.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #90895] Stat Anal
Boot
> Strapping
>
> Bob,
>
> The immediate error is that Stat-Analysis can’t read the config file
> from the path you provided:
>
>
>
>
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_boot
> strap
>
> I suspect you just have the path listed incorrectly there? Or
perhaps
> the permissions are set in a way that make it unreadable by Stat-
Analysis?
>
> But there may be a bigger issue in the computation of bootstrap
CI’s.
> Stat-Analysis only computes them when processing matched pair (MPR)
> input lines. If that’s what your jobs are doing (reading MPR and
> computing CNT, for example) that should be fine.
>
> But if you’re aggregating SL1L2 partial sums to compute CNT stats,
for
> example, bootstrapping won’t work. We’ve always used METviewer to
> compute bootstrap CIs for those aggregations. For completeness
sake,
> that functionality could also be added to Stat-Analysis. In fact
the
> development issue for this is over 7 years old!
>
> https://github.com/NCAR/MET/issues/59
>
> Since we’ve had METviewer, that feature request has never bubbled up
> to the top of our priority list. If this is needed in your
workflow,
> please let management know so it can be included in a future DTC
work plan.
>
> Thanks,
> John
>
> On Wed, Jul 3, 2019 at 10:03 AM Julie Prestopnik via RT
> <met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90895 >
> >
> > Hi Bob.
> >
> > Unfortunately, I cannot help you with this one. I have assigned
the
> > ticket to John. Please allow a few days for follow up.
> >
> > Thanks,
> > Julie
> >
> > On Wed, Jul 3, 2019 at 9:58 AM robert.craig.2 at us.af.mil via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Jul 03 09:58:41 2019: Request 90895 was acted upon.
> > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > Queue: met_help
> > > Subject: Stat Anal Boot Strapping
> > > Owner: Nobody
> > > Requestors: robert.craig.2 at us.af.mil
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90895 >
> > >
> > >
> > > John or Julie, I need to get error bars on my stat_analysis
output.
> It
> > > appears I would do this by setting the n_rep field in the
> > > stat_anal
> > config
> > > file to 1000. The question is can I set this in my stat_anal
JOB
> > > COMMAND LINE and not use the config_file? If the answer is no
> > > then can I add the -config option to the end of my command line
like below:
> > >
> > > /h/MET/bin/stat_analysis -lookin
> > > /h/data/global/WXQC/data/met/ptstat/mdlob_pairs/korea_4/TT -out
> > > /h/data/global/WXQC/data/met/summary/tt_korea_4_10p9 -tmp_dir
> > > /h/data/global/WXQC/d ata/met/tmp -job aggregate_stat
-line_type MPR
> > > -out_line_type CNT -fcst_valid_beg 20180101_000000
-fcst_valid_end
> > > 20180531_180000 -by OBS_LEV -fcst_var TMP -obs_var TMP -by
FCST_LE
> AD
> > > -dump_row /h/data/global/WXQC/data/met/filter_job -column_max
> > > -obs
> > > 50 -column_min -obs -70 -v 4 -config
> > >
> >
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_bo
> > ot
> > strap
> > >
> > > When I do this I get the following error:
> > >
> > > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> > > /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> > > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> > > DEBUG 1: Creating STAT-Analysis output file
> > > "/h/data/global/WXQC/data/met/summary/tt_korea_4_10p9"
> > > DEBUG 1: Default Config File:
> > > /h/MET/share/met/config/STATAnalysisConfig_default
> > > DEBUG 1: User Config File:
> > >
> >
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_bo
> > ot
> > strap
> > > ERROR :
> > > "RROR : recursive_envs() -> unable to open input file
> > >
> >
"/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_b
> > oo
> > tstrap
> > > ERROR :
> > >
> > > Thanks
> > > Bob
> > >
> > >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Stat Anal Boot Strapping
From: John Halley Gotway
Time: Fri Jul 05 10:57:56 2019
Bob,
I haven’t done this myself, but I believe the scientists pick a
threshold
to define “practical” significance for each variable/statistic combo.
For
example, if the difference is less than the precision of the
observations,
it probably isn’t practically significant. But I can’t remember
if/how
practical significance is included in the scorecards made by
METviewer.
This would be a good item to discuss during the regular telecons
between
DTC and the AF.
Thanks,
John
On Fri, Jul 5, 2019 at 8:27 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90895 >
>
> John, one last question on this issue - in your score card how do
you come
> up with practical significance versus "true" significance?
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Friday, July 5, 2019 9:18 AM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #90895] Stat Anal
Boot
> Strapping
>
> Bob,
>
> OK, I understand. We primarily use METviewer to compare the
performance
> between two models. We do this by plotting the pairwise difference
of
> statistics for multiple variables and levels and looking for
statistically
> significant differences. And we also create “scorecards” to quickly
> summarize the performance across multiple variables, levels, lead
times,
> and statistics.
>
> Really the only thing in MET itself which directly compares
performance is
> the GO index.
>
> Thanks,
> John
>
> On Fri, Jul 5, 2019 at 7:11 AM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90895 >
> >
> > Thanks John, The path to the config file was correct. I got the
error
> > to go away by placing the job command in the config file instead
of
> > the command line - maybe your not allowed to have it in two
places.
> > As far as the confidence intervals, we can wait to see what MET
Viewer
> > provides. The big thing of interest here is to be able to say one
> > model is significantly better than another. We have generated
some
> > hypothesis testing code but would love to have MET be able to
answer
> > this question. How do you guys calculate practical significance
versus
> true significance?
> >
> > Thanks
> > Bob
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Thursday, July 4, 2019 8:05 AM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #90895] Stat Anal
Boot
> > Strapping
> >
> > Bob,
> >
> > The immediate error is that Stat-Analysis can’t read the config
file
> > from the path you provided:
> >
> >
> >
> >
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_boot
> > strap
> >
> > I suspect you just have the path listed incorrectly there? Or
perhaps
> > the permissions are set in a way that make it unreadable by
> Stat-Analysis?
> >
> > But there may be a bigger issue in the computation of bootstrap
CI’s.
> > Stat-Analysis only computes them when processing matched pair
(MPR)
> > input lines. If that’s what your jobs are doing (reading MPR and
> > computing CNT, for example) that should be fine.
> >
> > But if you’re aggregating SL1L2 partial sums to compute CNT stats,
for
> > example, bootstrapping won’t work. We’ve always used METviewer to
> > compute bootstrap CIs for those aggregations. For completeness
sake,
> > that functionality could also be added to Stat-Analysis. In fact
the
> > development issue for this is over 7 years old!
> >
> > https://github.com/NCAR/MET/issues/59
> >
> > Since we’ve had METviewer, that feature request has never bubbled
up
> > to the top of our priority list. If this is needed in your
workflow,
> > please let management know so it can be included in a future DTC
work
> plan.
> >
> > Thanks,
> > John
> >
> > On Wed, Jul 3, 2019 at 10:03 AM Julie Prestopnik via RT
> > <met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90895 >
> > >
> > > Hi Bob.
> > >
> > > Unfortunately, I cannot help you with this one. I have assigned
the
> > > ticket to John. Please allow a few days for follow up.
> > >
> > > Thanks,
> > > Julie
> > >
> > > On Wed, Jul 3, 2019 at 9:58 AM robert.craig.2 at us.af.mil via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Wed Jul 03 09:58:41 2019: Request 90895 was acted upon.
> > > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > > Queue: met_help
> > > > Subject: Stat Anal Boot Strapping
> > > > Owner: Nobody
> > > > Requestors: robert.craig.2 at us.af.mil
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90895 >
> > > >
> > > >
> > > > John or Julie, I need to get error bars on my stat_analysis
output.
> > It
> > > > appears I would do this by setting the n_rep field in the
> > > > stat_anal
> > > config
> > > > file to 1000. The question is can I set this in my stat_anal
JOB
> > > > COMMAND LINE and not use the config_file? If the answer is no
> > > > then can I add the -config option to the end of my command
line like
> below:
> > > >
> > > > /h/MET/bin/stat_analysis -lookin
> > > > /h/data/global/WXQC/data/met/ptstat/mdlob_pairs/korea_4/TT
-out
> > > > /h/data/global/WXQC/data/met/summary/tt_korea_4_10p9 -tmp_dir
> > > > /h/data/global/WXQC/d ata/met/tmp -job aggregate_stat
-line_type
> MPR
> > > > -out_line_type CNT -fcst_valid_beg 20180101_000000
-fcst_valid_end
> > > > 20180531_180000 -by OBS_LEV -fcst_var TMP -obs_var TMP -by
FCST_LE
> > AD
> > > > -dump_row /h/data/global/WXQC/data/met/filter_job -column_max
> > > > -obs
> > > > 50 -column_min -obs -70 -v 4 -config
> > > >
> > >
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_bo
> > > ot
> > > strap
> > > >
> > > > When I do this I get the following error:
> > > >
> > > > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> > > > /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> > > > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > > > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> > > > DEBUG 1: Creating STAT-Analysis output file
> > > > "/h/data/global/WXQC/data/met/summary/tt_korea_4_10p9"
> > > > DEBUG 1: Default Config File:
> > > > /h/MET/share/met/config/STATAnalysisConfig_default
> > > > DEBUG 1: User Config File:
> > > >
> > >
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_bo
> > > ot
> > > strap
> > > > ERROR :
> > > > "RROR : recursive_envs() -> unable to open input file
> > > >
> > >
"/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_b
> > > oo
> > > tstrap
> > > > ERROR :
> > > >
> > > > Thanks
> > > > Bob
> > > >
> > > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #90895] Stat Anal Boot Strapping
From: robert.craig.2 at us.af.mil
Time: Fri Jul 05 12:45:17 2019
Okay, thanks for your help.
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Friday, July 5, 2019 11:58 AM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #90895] Stat Anal
Boot Strapping
Bob,
I haven’t done this myself, but I believe the scientists pick a
threshold to define “practical” significance for each
variable/statistic combo. For example, if the difference is less than
the precision of the observations, it probably isn’t practically
significant. But I can’t remember if/how practical significance is
included in the scorecards made by METviewer.
This would be a good item to discuss during the regular telecons
between DTC and the AF.
Thanks,
John
On Fri, Jul 5, 2019 at 8:27 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90895 >
>
> John, one last question on this issue - in your score card how do
you
> come up with practical significance versus "true" significance?
>
> Thanks
> Bob
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Friday, July 5, 2019 9:18 AM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #90895] Stat Anal
> Boot Strapping
>
> Bob,
>
> OK, I understand. We primarily use METviewer to compare the
> performance between two models. We do this by plotting the pairwise
> difference of statistics for multiple variables and levels and
looking
> for statistically significant differences. And we also create
> “scorecards” to quickly summarize the performance across multiple
> variables, levels, lead times, and statistics.
>
> Really the only thing in MET itself which directly compares
> performance is the GO index.
>
> Thanks,
> John
>
> On Fri, Jul 5, 2019 at 7:11 AM robert.craig.2 at us.af.mil via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90895 >
> >
> > Thanks John, The path to the config file was correct. I got the
> > error to go away by placing the job command in the config file
> > instead of the command line - maybe your not allowed to have it in
two places.
> > As far as the confidence intervals, we can wait to see what MET
> > Viewer provides. The big thing of interest here is to be able to
> > say one model is significantly better than another. We have
> > generated some hypothesis testing code but would love to have MET
be
> > able to answer this question. How do you guys calculate practical
> > significance versus
> true significance?
> >
> > Thanks
> > Bob
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu>
> > Sent: Thursday, July 4, 2019 8:05 AM
> > To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
> > <robert.craig.2 at us.af.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #90895] Stat Anal
> > Boot Strapping
> >
> > Bob,
> >
> > The immediate error is that Stat-Analysis can’t read the config
file
> > from the path you provided:
> >
> >
> >
> >
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_bo
> > ot
> > strap
> >
> > I suspect you just have the path listed incorrectly there? Or
> > perhaps the permissions are set in a way that make it unreadable
by
> Stat-Analysis?
> >
> > But there may be a bigger issue in the computation of bootstrap
CI’s.
> > Stat-Analysis only computes them when processing matched pair
(MPR)
> > input lines. If that’s what your jobs are doing (reading MPR and
> > computing CNT, for example) that should be fine.
> >
> > But if you’re aggregating SL1L2 partial sums to compute CNT stats,
> > for example, bootstrapping won’t work. We’ve always used
METviewer
> > to compute bootstrap CIs for those aggregations. For completeness
> > sake, that functionality could also be added to Stat-Analysis. In
> > fact the development issue for this is over 7 years old!
> >
> > https://github.com/NCAR/MET/issues/59
> >
> > Since we’ve had METviewer, that feature request has never bubbled
up
> > to the top of our priority list. If this is needed in your
> > workflow, please let management know so it can be included in a
> > future DTC work
> plan.
> >
> > Thanks,
> > John
> >
> > On Wed, Jul 3, 2019 at 10:03 AM Julie Prestopnik via RT
> > <met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90895 >
> > >
> > > Hi Bob.
> > >
> > > Unfortunately, I cannot help you with this one. I have assigned
> > > the ticket to John. Please allow a few days for follow up.
> > >
> > > Thanks,
> > > Julie
> > >
> > > On Wed, Jul 3, 2019 at 9:58 AM robert.craig.2 at us.af.mil via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Wed Jul 03 09:58:41 2019: Request 90895 was acted upon.
> > > > Transaction: Ticket created by robert.craig.2 at us.af.mil
> > > > Queue: met_help
> > > > Subject: Stat Anal Boot Strapping
> > > > Owner: Nobody
> > > > Requestors: robert.craig.2 at us.af.mil
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90895 >
> > > >
> > > >
> > > > John or Julie, I need to get error bars on my stat_analysis
output.
> > It
> > > > appears I would do this by setting the n_rep field in the
> > > > stat_anal
> > > config
> > > > file to 1000. The question is can I set this in my stat_anal
> > > > JOB COMMAND LINE and not use the config_file? If the answer
is
> > > > no then can I add the -config option to the end of my command
> > > > line like
> below:
> > > >
> > > > /h/MET/bin/stat_analysis -lookin
> > > > /h/data/global/WXQC/data/met/ptstat/mdlob_pairs/korea_4/TT
-out
> > > > /h/data/global/WXQC/data/met/summary/tt_korea_4_10p9 -tmp_dir
> > > > /h/data/global/WXQC/d ata/met/tmp -job aggregate_stat
-line_type
> MPR
> > > > -out_line_type CNT -fcst_valid_beg 20180101_000000
> > > > -fcst_valid_end
> > > > 20180531_180000 -by OBS_LEV -fcst_var TMP -obs_var TMP -by
> > > > FCST_LE
> > AD
> > > > -dump_row /h/data/global/WXQC/data/met/filter_job -column_max
> > > > -obs
> > > > 50 -column_min -obs -70 -v 4 -config
> > > >
> > >
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_
> > > bo
> > > ot
> > > strap
> > > >
> > > > When I do this I get the following error:
> > > >
> > > > DEBUG 1: Reading user-defined grib1 MET_GRIB_TABLES file:
> > > > /h/data/global/WXQC/data/met/config/grib2/grib1_af_131_57.txt
> > > > DEBUG 1: Reading user-defined grib2 MET_GRIB_TABLES file:
> > > > /h/data/global/WXQC/data/met/config/grib2/grib2_geps.txt
> > > > DEBUG 1: Creating STAT-Analysis output file
> > > > "/h/data/global/WXQC/data/met/summary/tt_korea_4_10p9"
> > > > DEBUG 1: Default Config File:
> > > > /h/MET/share/met/config/STATAnalysisConfig_default
> > > > DEBUG 1: User Config File:
> > > >
> > >
/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig_
> > > bo
> > > ot
> > > strap
> > > > ERROR :
> > > > "RROR : recursive_envs() -> unable to open input file
> > > >
> > >
"/h/data/global/WXQC/data/met/config/met_config/STATAnalysisConfig
> > > _b
> > > oo
> > > tstrap
> > > > ERROR :
> > > >
> > > > Thanks
> > > > Bob
> > > >
> > > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list