[Met_help] [rt.rap.ucar.edu #43402] History for Question about Confidence Intervals (MET V2.0) (UNCLASSIFIED)
RAL HelpDesk {for John Halley Gotway}
met_help at ucar.edu
Tue Jan 11 13:26:26 MST 2011
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Classification: UNCLASSIFIED
Caveats: NONE
In my Point-Stat config file, I have the value of "n_boot_rep" set to 0 to
disable the computation of bootstrap confidence intervals (CI). The output
text file (_cnt.txt file) confirms that only the columns for the normal CI
have values and the columns for the bootstrap CI have "NA" as expected.
However, when I look at my Stat-Analysis text output files I find that the
bootstrap CI columns show data values as if the bootstrap CI IS being
computed. I do not use a config file for running Stat-Analysis. The job I
run is aggregate_stat using MPR input to get CNT output.
The V 2.0 Users Guide from March 2009 (pp 8-1 and 8-2) does state that both
the normal and bootstrap methods are computed by Stat-Analysis by resampling
the matched pairs.
Are the bootstrap CI values I'm getting from Stat-Analysis valid if I
disabled their computation in Point-Stat?
I would like to use the CI values produced by Stat-Analysis, but I am
concerned that there is an inconsistency because the bootstrap CI values are
not being computed by Point-Stat and yet I am seeing values for bootstrap CI
in my Stat-Analysis output.
If the Stat-Analysis values for the bootstrap CI ARE valid, then what are
the specifications being used to generate these CI values? (such as
ci_alpha, boot_interval, boot_rep_prop, ...etc)
Thanks.
R/
John
Mr John W. Raby, Meteorologist
U.S. Army Research Laboratory
White Sands Missile Range, NM 88002
(575) 678-2004 DSN 258-2004
FAX (575) 678-1230 DSN 258-1230
Email: john.raby at us.army.mil
"When you can measure what you are speaking about and express it in numbers,
you know something about it, but when you cannot measure it, when you cannot
express it in number, your knowledge is of a meagre and unsatisfactory
kind". - Lord Kelvin
Classification: UNCLASSIFIED
Caveats: NONE
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #43402] Question about Confidence Intervals (MET V2.0) (UNCLASSIFIED)
From: John Halley Gotway
Time: Tue Jan 11 10:38:46 2011
John,
Good questions.
Yes, the bootstrap confidence intervals produced by STAT-Analysis are
valid. And it's fine that you're not computing them in Point-Stat.
The bootstrap CI's produced by Point-Stat and STAT-Analysis
are completely independent. The difference really is what set of
matched pairs are used for the computation of bootstrap CI's.
If you compute them in Point-Stat, the matched pairs consist of all
the pairs falling the verification region at a single point in time.
If you compute them in STAT-Analysis, you're able to group together
matched pairs from multiple output times, and then use that larger set
to compute bootstrap CI's.
I'm assuming here that you're using METv3.0. If that's not the case,
please let me know.
When you run STAT-Analysis jobs, most of the settings have reasonable
defaults which can be overridden on the command line or in the config
file. This is true of all of the bootstrap settings as
well. To see what those defaults are, you could either:
(1) Look in the file
"METv3.0/src/stat_analysis/STATAnalysisConfig_default"
(2) Run a STAT-Analysis job and look at the output. For example,
try running the following commands that run on the output of the MET
test scripts:
> cd METv3.0
> bin/stat_analysis -lookin out/point_stat -job aggregate_stat
-line_type MPR -out_line_type CNT -fcst_var TMP -fcst_lev P900-750
Listed below is the output...
JOB_LIST: -job aggregate_stat-fcst_var TMP-fcst_lev P900-750-
line_type MPR-out_line_type CNT-out_alpha 0.05-boot_interval 1-
boot_rep_prop 1.00-n_boot_rep 1000-boot_rng mt19937-boot_seed
-rank_corr_flag 1
COL_NAME: TOTAL FBAR FBAR_NCL FBAR_NCU FBAR_BCL FBAR_BCU
FSTDEV FSTDEV_NCL FSTDEV_NCU FSTDEV_BCL FSTDEV_BCU OBAR OBAR_NCL
OBAR_NCU OBAR_BCL OBAR_BCU OSTDEV OSTDEV_NCL OSTDEV_NCU
OSTDEV_BCL OSTDEV_BCU PR_CORR PR_CORR_NCL PR_CORR_NCU PR_CORR_BCL
PR_CORR_BCU SP_CORR KT_CORR RANKS FRANK_TIES ORANK_TIES ME ME_NCL
ME_NCU ME_BCL ME_BCU ESTDEV ESTDEV_NCL ESTDEV_NCU
ESTDEV_BCL ESTDEV_BCU MBIAS MBIAS_BCL MBIAS_BCU MAE MAE_BCL
MAE_BCU MSE MSE_BCL MSE_BCU BCMSE BCMSE_BCL BCMSE_BCU RMSE
RMSE_BCL RMSE_BCU E10 E10_BCL E10_BCU E25 E25_BCL
E25_BCU E50 E50_BCL E50_BCU E75 E75_BCL E75_BCU E90
E90_BCL E90_BCU
CNT: 2508 277.43125 277.20217 277.66032 277.19815 277.66503
5.85327 5.69565 6.01992 5.64205 6.06687 275.75789
275.50294 276.01284 275.51125 276.00568 6.51435 6.33894 6.69982
6.33504 6.70909 0.85559 0.84474 0.86574 0.84227
0.86739 0.81325 0.63270 2508 1227 2333 1.67335
1.54093 1.80578 1.53824 1.79508 3.38374 3.29263 3.48008 3.28550
3.48192 1.00607 1.00558 1.00651 2.77947 2.68364 2.87494
14.24527 13.28130 15.15132 11.44515 10.79019 12.11896 3.77429
3.64435 3.89247 -2.08348 -2.29510 -1.91335 -0.75013 -0.90712
-0.65183 0.90993 0.75596 1.10766 3.69092 3.48483 3.90753 6.50509
6.04324 6.65989
If you take a look at the "JOB_LIST" line, you'll see a listing of all
of the filtering criteria and processing options used by this job.
This includes what we specified on the command line and the
default options. In there you'll see the bootstrapping related
settings: out_alpha=0.05, boot_interval=1, boot_rep_prop=1.00,
n_boot_rep=1000, and boot_rng=mt19937. You can override any of these
using the appropriate command line option.
I see that we have a slight bug in the code - that JOB_LIST should
contain spaces between each option listed.
Hope that helps.
John
On 01/11/2011 10:16 AM, RAL HelpDesk {for Raby, John} wrote:
>
> Tue Jan 11 10:16:17 2011: Request 43402 was acted upon.
> Transaction: Ticket created by john.raby at us.army.mil
> Queue: met_help
> Subject: Question about Confidence Intervals (MET V2.0)
(UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.raby at us.army.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=43402 >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> In my Point-Stat config file, I have the value of "n_boot_rep" set
to 0 to
> disable the computation of bootstrap confidence intervals (CI). The
output
> text file (_cnt.txt file) confirms that only the columns for the
normal CI
> have values and the columns for the bootstrap CI have "NA" as
expected.
>
> However, when I look at my Stat-Analysis text output files I find
that the
> bootstrap CI columns show data values as if the bootstrap CI IS
being
> computed. I do not use a config file for running Stat-Analysis. The
job I
> run is aggregate_stat using MPR input to get CNT output.
>
> The V 2.0 Users Guide from March 2009 (pp 8-1 and 8-2) does state
that both
> the normal and bootstrap methods are computed by Stat-Analysis by
resampling
> the matched pairs.
>
> Are the bootstrap CI values I'm getting from Stat-Analysis valid if
I
> disabled their computation in Point-Stat?
>
> I would like to use the CI values produced by Stat-Analysis, but I
am
> concerned that there is an inconsistency because the bootstrap CI
values are
> not being computed by Point-Stat and yet I am seeing values for
bootstrap CI
> in my Stat-Analysis output.
>
> If the Stat-Analysis values for the bootstrap CI ARE valid, then
what are
> the specifications being used to generate these CI values? (such as
> ci_alpha, boot_interval, boot_rep_prop, ...etc)
>
> Thanks.
>
> R/
> John
>
> Mr John W. Raby, Meteorologist
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> (575) 678-2004 DSN 258-2004
> FAX (575) 678-1230 DSN 258-1230
> Email: john.raby at us.army.mil
>
> "When you can measure what you are speaking about and express it in
numbers,
> you know something about it, but when you cannot measure it, when
you cannot
> express it in number, your knowledge is of a meagre and
unsatisfactory
> kind". - Lord Kelvin
>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
------------------------------------------------
Subject: Question about Confidence Intervals (MET V2.0) (UNCLASSIFIED)
From: Raby, John
Time: Tue Jan 11 11:46:32 2011
Classification: UNCLASSIFIED
Caveats: NONE
John -
Thanks for the quick response.
I am using MET V2.0.
I would think that using the larger set of matched pairs gives you a
better
sampling basis for estimating the CI, so I'm happy with the way I'm
running
Point-Stat and Stat-Analysis.
I was able to examine one of my Stat-Analysis output text files and
confirm
the bootstrap settings you show in your output below.
The output JOB_LIST line I'm seeing here DOES have a space between
each
option.
To override these settings you said that you use the appropriate
command line
option, but where in the command line are these settings specified? I
thought
that you had to use the config file to specify the settings you want,
if you
want something other than the default values.
Thanks for clarifying how the CIs are computed. Good info!
R/
John
-----Original Message-----
From: RAL HelpDesk {for John Halley Gotway} [mailto:met_help at ucar.edu]
Sent: Tuesday, January 11, 2011 10:39 AM
To: Raby, John (Civ, ARL/CISD)
Cc: Brown, Bob (Civ, ARL/CISD); Raby, Yasmina (Civ, ARL/CISD)
Subject: Re: [rt.rap.ucar.edu #43402] Question about Confidence
Intervals (MET
V2.0) (UNCLASSIFIED)
John,
Good questions.
Yes, the bootstrap confidence intervals produced by STAT-Analysis are
valid.
And it's fine that you're not computing them in Point-Stat. The
bootstrap
CI's produced by Point-Stat and STAT-Analysis are completely
independent. The
difference really is what set of matched pairs are used for the
computation of
bootstrap CI's.
If you compute them in Point-Stat, the matched pairs consist of all
the pairs
falling the verification region at a single point in time.
If you compute them in STAT-Analysis, you're able to group together
matched
pairs from multiple output times, and then use that larger set to
compute
bootstrap CI's.
I'm assuming here that you're using METv3.0. If that's not the case,
please
let me know.
When you run STAT-Analysis jobs, most of the settings have reasonable
defaults
which can be overridden on the command line or in the config file.
This is
true of all of the bootstrap settings as well. To see what those
defaults
are, you could either:
(1) Look in the file
"METv3.0/src/stat_analysis/STATAnalysisConfig_default"
(2) Run a STAT-Analysis job and look at the output. For example,
try
running the following commands that run on the output of the MET test
scripts:
> cd METv3.0
> bin/stat_analysis -lookin out/point_stat -job aggregate_stat
> -line_type MPR -out_line_type CNT -fcst_var TMP -fcst_lev P900-750
Listed below is the output...
JOB_LIST: -job aggregate_stat-fcst_var TMP-fcst_lev P900-750-
line_type
MPR-out_line_type CNT-out_alpha 0.05-boot_interval 1-boot_rep_prop
1.00-n_boot_rep 1000-boot_rng mt19937-boot_seed
-rank_corr_flag 1
COL_NAME: TOTAL FBAR FBAR_NCL FBAR_NCU FBAR_BCL FBAR_BCU
FSTDEV
FSTDEV_NCL FSTDEV_NCU FSTDEV_BCL FSTDEV_BCU OBAR OBAR_NCL
OBAR_NCU
OBAR_BCL OBAR_BCU OSTDEV OSTDEV_NCL OSTDEV_NCU
OSTDEV_BCL OSTDEV_BCU PR_CORR PR_CORR_NCL PR_CORR_NCU PR_CORR_BCL
PR_CORR_BCU
SP_CORR KT_CORR RANKS FRANK_TIES ORANK_TIES ME ME_NCL ME_NCU
ME_BCL
ME_BCU ESTDEV ESTDEV_NCL ESTDEV_NCU
ESTDEV_BCL ESTDEV_BCU MBIAS MBIAS_BCL MBIAS_BCU MAE MAE_BCL
MAE_BCU MSE
MSE_BCL MSE_BCU BCMSE BCMSE_BCL BCMSE_BCU RMSE RMSE_BCL
RMSE_BCU E10
E10_BCL E10_BCU E25 E25_BCL
E25_BCU E50 E50_BCL E50_BCU E75 E75_BCL E75_BCU E90
E90_BCL
E90_BCU
CNT: 2508 277.43125 277.20217 277.66032 277.19815 277.66503
5.85327
5.69565 6.01992 5.64205 6.06687 275.75789 275.50294
276.01284
275.51125 276.00568 6.51435 6.33894 6.69982
6.33504 6.70909 0.85559 0.84474 0.86574 0.84227
0.86739
0.81325 0.63270 2508 1227 2333 1.67335 1.54093 1.80578
1.53824
1.79508 3.38374 3.29263 3.48008 3.28550
3.48192 1.00607 1.00558 1.00651 2.77947 2.68364 2.87494
14.24527
13.28130 15.15132 11.44515 10.79019 12.11896 3.77429 3.64435
3.89247 -2.08348 -2.29510 -1.91335 -0.75013 -0.90712
-0.65183 0.90993 0.75596 1.10766 3.69092 3.48483 3.90753 6.50509
6.04324
6.65989
If you take a look at the "JOB_LIST" line, you'll see a listing of all
of the
filtering criteria and processing options used by this job. This
includes
what we specified on the command line and the default options. In
there
you'll see the bootstrapping related settings: out_alpha=0.05,
boot_interval=1, boot_rep_prop=1.00, n_boot_rep=1000, and
boot_rng=mt19937.
You can override any of these using the appropriate command line
option.
I see that we have a slight bug in the code - that JOB_LIST should
contain
spaces between each option listed.
Hope that helps.
John
On 01/11/2011 10:16 AM, RAL HelpDesk {for Raby, John} wrote:
>
> Tue Jan 11 10:16:17 2011: Request 43402 was acted upon.
> Transaction: Ticket created by john.raby at us.army.mil
> Queue: met_help
> Subject: Question about Confidence Intervals (MET V2.0)
(UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.raby at us.army.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=43402
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> In my Point-Stat config file, I have the value of "n_boot_rep" set
to
> 0 to disable the computation of bootstrap confidence intervals (CI).
> The output text file (_cnt.txt file) confirms that only the columns
> for the normal CI have values and the columns for the bootstrap CI
have "NA"
> as expected.
>
> However, when I look at my Stat-Analysis text output files I find
that
> the bootstrap CI columns show data values as if the bootstrap CI IS
> being computed. I do not use a config file for running Stat-
Analysis.
> The job I run is aggregate_stat using MPR input to get CNT output.
>
> The V 2.0 Users Guide from March 2009 (pp 8-1 and 8-2) does state
that
> both the normal and bootstrap methods are computed by Stat-Analysis
by
> resampling the matched pairs.
>
> Are the bootstrap CI values I'm getting from Stat-Analysis valid if
I
> disabled their computation in Point-Stat?
>
> I would like to use the CI values produced by Stat-Analysis, but I
am
> concerned that there is an inconsistency because the bootstrap CI
> values are not being computed by Point-Stat and yet I am seeing
values
> for bootstrap CI in my Stat-Analysis output.
>
> If the Stat-Analysis values for the bootstrap CI ARE valid, then
what
> are the specifications being used to generate these CI values? (such
> as ci_alpha, boot_interval, boot_rep_prop, ...etc)
>
> Thanks.
>
> R/
> John
>
> Mr John W. Raby, Meteorologist
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> (575) 678-2004 DSN 258-2004
> FAX (575) 678-1230 DSN 258-1230
> Email: john.raby at us.army.mil
>
> "When you can measure what you are speaking about and express it in
> numbers, you know something about it, but when you cannot measure
it,
> when you cannot express it in number, your knowledge is of a meagre
> and unsatisfactory kind". - Lord Kelvin
>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
Classification: UNCLASSIFIED
Caveats: NONE
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #43402] Question about Confidence Intervals (MET V2.0) (UNCLASSIFIED)
From: John Halley Gotway
Time: Tue Jan 11 12:58:39 2011
John,
Please take a look in the file
METv2.0/src/stat_analysis/STATAnalysisConfig_default. For each one of
those filtering options, starting with "model" and ending with
"line_type", there is a
corresponding command line option that you can use: "-model" ... "-
fcst_lead". This is also described in the comments listed above each
option. For example, in that job I'd sent you earlier you
could specify the number of replicates and alpha value using command
line options:
bin/stat_analysis -lookin out/point_stat -job aggregate_stat
-line_type MPR -out_line_type CNT -fcst_var TMP -fcst_lev P900-750
-n_boot_rep 2500 -alpha 0.10
Thanks for letting me know that the JOB_LIST looks fine in your
output. It turns out that it was a bug in the development version of
the code - not the released version.
Thanks,
John
On 01/11/2011 11:46 AM, RAL HelpDesk {for Raby, John} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=43402 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for the quick response.
>
> I am using MET V2.0.
>
> I would think that using the larger set of matched pairs gives you a
better
> sampling basis for estimating the CI, so I'm happy with the way I'm
running
> Point-Stat and Stat-Analysis.
>
> I was able to examine one of my Stat-Analysis output text files and
confirm
> the bootstrap settings you show in your output below.
>
> The output JOB_LIST line I'm seeing here DOES have a space between
each
> option.
>
> To override these settings you said that you use the appropriate
command line
> option, but where in the command line are these settings specified?
I thought
> that you had to use the config file to specify the settings you
want, if you
> want something other than the default values.
>
> Thanks for clarifying how the CIs are computed. Good info!
>
> R/
> John
>
>
> -----Original Message-----
> From: RAL HelpDesk {for John Halley Gotway}
[mailto:met_help at ucar.edu]
> Sent: Tuesday, January 11, 2011 10:39 AM
> To: Raby, John (Civ, ARL/CISD)
> Cc: Brown, Bob (Civ, ARL/CISD); Raby, Yasmina (Civ, ARL/CISD)
> Subject: Re: [rt.rap.ucar.edu #43402] Question about Confidence
Intervals (MET
> V2.0) (UNCLASSIFIED)
>
> John,
>
> Good questions.
>
> Yes, the bootstrap confidence intervals produced by STAT-Analysis
are valid.
> And it's fine that you're not computing them in Point-Stat. The
bootstrap
> CI's produced by Point-Stat and STAT-Analysis are completely
independent. The
> difference really is what set of matched pairs are used for the
computation of
> bootstrap CI's.
>
> If you compute them in Point-Stat, the matched pairs consist of all
the pairs
> falling the verification region at a single point in time.
>
> If you compute them in STAT-Analysis, you're able to group together
matched
> pairs from multiple output times, and then use that larger set to
compute
> bootstrap CI's.
>
> I'm assuming here that you're using METv3.0. If that's not the
case, please
> let me know.
>
> When you run STAT-Analysis jobs, most of the settings have
reasonable defaults
> which can be overridden on the command line or in the config file.
This is
> true of all of the bootstrap settings as well. To see what those
defaults
> are, you could either:
> (1) Look in the file
"METv3.0/src/stat_analysis/STATAnalysisConfig_default"
> (2) Run a STAT-Analysis job and look at the output. For example,
try
> running the following commands that run on the output of the MET
test scripts:
>
>> cd METv3.0
>> bin/stat_analysis -lookin out/point_stat -job aggregate_stat
>> -line_type MPR -out_line_type CNT -fcst_var TMP -fcst_lev P900-750
>
> Listed below is the output...
>
> JOB_LIST: -job aggregate_stat-fcst_var TMP-fcst_lev P900-750-
line_type
> MPR-out_line_type CNT-out_alpha 0.05-boot_interval 1-boot_rep_prop
> 1.00-n_boot_rep 1000-boot_rng mt19937-boot_seed
> -rank_corr_flag 1
> COL_NAME: TOTAL FBAR FBAR_NCL FBAR_NCU FBAR_BCL FBAR_BCU
FSTDEV
> FSTDEV_NCL FSTDEV_NCU FSTDEV_BCL FSTDEV_BCU OBAR OBAR_NCL
OBAR_NCU
> OBAR_BCL OBAR_BCU OSTDEV OSTDEV_NCL OSTDEV_NCU
> OSTDEV_BCL OSTDEV_BCU PR_CORR PR_CORR_NCL PR_CORR_NCU PR_CORR_BCL
PR_CORR_BCU
> SP_CORR KT_CORR RANKS FRANK_TIES ORANK_TIES ME ME_NCL ME_NCU
ME_BCL
> ME_BCU ESTDEV ESTDEV_NCL ESTDEV_NCU
> ESTDEV_BCL ESTDEV_BCU MBIAS MBIAS_BCL MBIAS_BCU MAE MAE_BCL
MAE_BCU MSE
> MSE_BCL MSE_BCU BCMSE BCMSE_BCL BCMSE_BCU RMSE RMSE_BCL
RMSE_BCU E10
> E10_BCL E10_BCU E25 E25_BCL
> E25_BCU E50 E50_BCL E50_BCU E75 E75_BCL E75_BCU E90
E90_BCL
> E90_BCU
> CNT: 2508 277.43125 277.20217 277.66032 277.19815 277.66503
5.85327
> 5.69565 6.01992 5.64205 6.06687 275.75789 275.50294
276.01284
> 275.51125 276.00568 6.51435 6.33894 6.69982
> 6.33504 6.70909 0.85559 0.84474 0.86574 0.84227
0.86739
> 0.81325 0.63270 2508 1227 2333 1.67335 1.54093 1.80578
1.53824
> 1.79508 3.38374 3.29263 3.48008 3.28550
> 3.48192 1.00607 1.00558 1.00651 2.77947 2.68364 2.87494
14.24527
> 13.28130 15.15132 11.44515 10.79019 12.11896 3.77429 3.64435
> 3.89247 -2.08348 -2.29510 -1.91335 -0.75013 -0.90712
> -0.65183 0.90993 0.75596 1.10766 3.69092 3.48483 3.90753 6.50509
6.04324
> 6.65989
>
> If you take a look at the "JOB_LIST" line, you'll see a listing of
all of the
> filtering criteria and processing options used by this job. This
includes
> what we specified on the command line and the default options. In
there
> you'll see the bootstrapping related settings: out_alpha=0.05,
> boot_interval=1, boot_rep_prop=1.00, n_boot_rep=1000, and
boot_rng=mt19937.
> You can override any of these using the appropriate command line
option.
>
> I see that we have a slight bug in the code - that JOB_LIST should
contain
> spaces between each option listed.
>
> Hope that helps.
>
> John
>
>
> On 01/11/2011 10:16 AM, RAL HelpDesk {for Raby, John} wrote:
>>
>> Tue Jan 11 10:16:17 2011: Request 43402 was acted upon.
>> Transaction: Ticket created by john.raby at us.army.mil
>> Queue: met_help
>> Subject: Question about Confidence Intervals (MET V2.0)
(UNCLASSIFIED)
>> Owner: Nobody
>> Requestors: john.raby at us.army.mil
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=43402
>>>
>>
>>
>> Classification: UNCLASSIFIED
>> Caveats: NONE
>>
>> In my Point-Stat config file, I have the value of "n_boot_rep" set
to
>> 0 to disable the computation of bootstrap confidence intervals
(CI).
>> The output text file (_cnt.txt file) confirms that only the columns
>> for the normal CI have values and the columns for the bootstrap CI
have "NA"
>> as expected.
>>
>> However, when I look at my Stat-Analysis text output files I find
that
>> the bootstrap CI columns show data values as if the bootstrap CI IS
>> being computed. I do not use a config file for running Stat-
Analysis.
>> The job I run is aggregate_stat using MPR input to get CNT output.
>>
>> The V 2.0 Users Guide from March 2009 (pp 8-1 and 8-2) does state
that
>> both the normal and bootstrap methods are computed by Stat-Analysis
by
>> resampling the matched pairs.
>>
>> Are the bootstrap CI values I'm getting from Stat-Analysis valid if
I
>> disabled their computation in Point-Stat?
>>
>> I would like to use the CI values produced by Stat-Analysis, but I
am
>> concerned that there is an inconsistency because the bootstrap CI
>> values are not being computed by Point-Stat and yet I am seeing
values
>> for bootstrap CI in my Stat-Analysis output.
>>
>> If the Stat-Analysis values for the bootstrap CI ARE valid, then
what
>> are the specifications being used to generate these CI values?
(such
>> as ci_alpha, boot_interval, boot_rep_prop, ...etc)
>>
>> Thanks.
>>
>> R/
>> John
>>
>> Mr John W. Raby, Meteorologist
>> U.S. Army Research Laboratory
>> White Sands Missile Range, NM 88002
>> (575) 678-2004 DSN 258-2004
>> FAX (575) 678-1230 DSN 258-1230
>> Email: john.raby at us.army.mil
>>
>> "When you can measure what you are speaking about and express it in
>> numbers, you know something about it, but when you cannot measure
it,
>> when you cannot express it in number, your knowledge is of a meagre
>> and unsatisfactory kind". - Lord Kelvin
>>
>>
>>
>> Classification: UNCLASSIFIED
>> Caveats: NONE
>>
>>
>>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
------------------------------------------------
Subject: Question about Confidence Intervals (MET V2.0) (UNCLASSIFIED)
From: Raby, John
Time: Tue Jan 11 13:22:47 2011
Classification: UNCLASSIFIED
Caveats: NONE
John -
Thanks for explaining that. I looked at that config file and found the
options. I was unaware that you could specify them as command line
options in
lieu of using a config file. I'm learning something new about MET all
the
time!
R/
John
-----Original Message-----
From: RAL HelpDesk {for John Halley Gotway} [mailto:met_help at ucar.edu]
Sent: Tuesday, January 11, 2011 12:59 PM
To: Raby, John (Civ, ARL/CISD)
Cc: Brown, Bob (Civ, ARL/CISD); Raby, Yasmina (Civ, ARL/CISD)
Subject: Re: [rt.rap.ucar.edu #43402] Question about Confidence
Intervals (MET
V2.0) (UNCLASSIFIED)
John,
Please take a look in the file
METv2.0/src/stat_analysis/STATAnalysisConfig_default. For each one of
those
filtering options, starting with "model" and ending with "line_type",
there is
a corresponding command line option that you can use: "-model" ...
"-fcst_lead". This is also described in the comments listed above
each
option. For example, in that job I'd sent you earlier you could
specify the
number of replicates and alpha value using command line options:
bin/stat_analysis -lookin out/point_stat -job aggregate_stat
-line_type
MPR -out_line_type CNT -fcst_var TMP -fcst_lev P900-750 -n_boot_rep
2500 -alpha 0.10
Thanks for letting me know that the JOB_LIST looks fine in your
output. It
turns out that it was a bug in the development version of the code -
not the
released version.
Thanks,
John
On 01/11/2011 11:46 AM, RAL HelpDesk {for Raby, John} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=43402 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for the quick response.
>
> I am using MET V2.0.
>
> I would think that using the larger set of matched pairs gives you a
> better sampling basis for estimating the CI, so I'm happy with the
way
> I'm running Point-Stat and Stat-Analysis.
>
> I was able to examine one of my Stat-Analysis output text files and
> confirm the bootstrap settings you show in your output below.
>
> The output JOB_LIST line I'm seeing here DOES have a space between
> each option.
>
> To override these settings you said that you use the appropriate
> command line option, but where in the command line are these
settings
> specified? I thought that you had to use the config file to specify
> the settings you want, if you want something other than the default
values.
>
> Thanks for clarifying how the CIs are computed. Good info!
>
> R/
> John
>
>
> -----Original Message-----
> From: RAL HelpDesk {for John Halley Gotway}
[mailto:met_help at ucar.edu]
> Sent: Tuesday, January 11, 2011 10:39 AM
> To: Raby, John (Civ, ARL/CISD)
> Cc: Brown, Bob (Civ, ARL/CISD); Raby, Yasmina (Civ, ARL/CISD)
> Subject: Re: [rt.rap.ucar.edu #43402] Question about Confidence
> Intervals (MET
> V2.0) (UNCLASSIFIED)
>
> John,
>
> Good questions.
>
> Yes, the bootstrap confidence intervals produced by STAT-Analysis
are valid.
> And it's fine that you're not computing them in Point-Stat. The
> bootstrap CI's produced by Point-Stat and STAT-Analysis are
completely
> independent. The difference really is what set of matched pairs are
> used for the computation of bootstrap CI's.
>
> If you compute them in Point-Stat, the matched pairs consist of all
> the pairs falling the verification region at a single point in time.
>
> If you compute them in STAT-Analysis, you're able to group together
> matched pairs from multiple output times, and then use that larger
set
> to compute bootstrap CI's.
>
> I'm assuming here that you're using METv3.0. If that's not the
case,
> please let me know.
>
> When you run STAT-Analysis jobs, most of the settings have
reasonable
> defaults which can be overridden on the command line or in the
config
> file. This is true of all of the bootstrap settings as well. To
see
> what those defaults are, you could either:
> (1) Look in the file
> "METv3.0/src/stat_analysis/STATAnalysisConfig_default"
> (2) Run a STAT-Analysis job and look at the output. For example,
> try running the following commands that run on the output of the MET
test
> scripts:
>
>> cd METv3.0
>> bin/stat_analysis -lookin out/point_stat -job aggregate_stat
>> -line_type MPR -out_line_type CNT -fcst_var TMP -fcst_lev P900-750
>
> Listed below is the output...
>
> JOB_LIST: -job aggregate_stat-fcst_var TMP-fcst_lev P900-750-
line_type
> MPR-out_line_type CNT-out_alpha 0.05-boot_interval 1-boot_rep_prop
> 1.00-n_boot_rep 1000-boot_rng mt19937-boot_seed -rank_corr_flag 1
> COL_NAME: TOTAL FBAR FBAR_NCL FBAR_NCU FBAR_BCL FBAR_BCU
FSTDEV
> FSTDEV_NCL FSTDEV_NCU FSTDEV_BCL FSTDEV_BCU OBAR OBAR_NCL
OBAR_NCU
> OBAR_BCL OBAR_BCU OSTDEV OSTDEV_NCL OSTDEV_NCU OSTDEV_BCL
> OSTDEV_BCU PR_CORR PR_CORR_NCL PR_CORR_NCU PR_CORR_BCL PR_CORR_BCU
> SP_CORR KT_CORR RANKS FRANK_TIES ORANK_TIES ME ME_NCL ME_NCU
ME_BCL
> ME_BCU ESTDEV ESTDEV_NCL ESTDEV_NCU
> ESTDEV_BCL ESTDEV_BCU MBIAS MBIAS_BCL MBIAS_BCU MAE MAE_BCL
MAE_BCU
> MSE
> MSE_BCL MSE_BCU BCMSE BCMSE_BCL BCMSE_BCU RMSE RMSE_BCL
RMSE_BCU E10
> E10_BCL E10_BCU E25 E25_BCL
> E25_BCU E50 E50_BCL E50_BCU E75 E75_BCL E75_BCU E90
E90_BCL
> E90_BCU
> CNT: 2508 277.43125 277.20217 277.66032 277.19815 277.66503
5.85327
> 5.69565 6.01992 5.64205 6.06687 275.75789 275.50294
276.01284
> 275.51125 276.00568 6.51435 6.33894 6.69982
> 6.33504 6.70909 0.85559 0.84474 0.86574 0.84227
0.86739
> 0.81325 0.63270 2508 1227 2333 1.67335 1.54093 1.80578
1.53824
> 1.79508 3.38374 3.29263 3.48008 3.28550
> 3.48192 1.00607 1.00558 1.00651 2.77947 2.68364 2.87494
14.24527
> 13.28130 15.15132 11.44515 10.79019 12.11896 3.77429 3.64435
> 3.89247 -2.08348 -2.29510 -1.91335 -0.75013 -0.90712
> -0.65183 0.90993 0.75596 1.10766 3.69092 3.48483 3.90753 6.50509
> 6.04324
> 6.65989
>
> If you take a look at the "JOB_LIST" line, you'll see a listing of
all
> of the filtering criteria and processing options used by this job.
> This includes what we specified on the command line and the default
> options. In there you'll see the bootstrapping related settings:
> out_alpha=0.05, boot_interval=1, boot_rep_prop=1.00,
n_boot_rep=1000, and
> boot_rng=mt19937.
> You can override any of these using the appropriate command line
option.
>
> I see that we have a slight bug in the code - that JOB_LIST should
> contain spaces between each option listed.
>
> Hope that helps.
>
> John
>
>
> On 01/11/2011 10:16 AM, RAL HelpDesk {for Raby, John} wrote:
>>
>> Tue Jan 11 10:16:17 2011: Request 43402 was acted upon.
>> Transaction: Ticket created by john.raby at us.army.mil
>> Queue: met_help
>> Subject: Question about Confidence Intervals (MET V2.0)
(UNCLASSIFIED)
>> Owner: Nobody
>> Requestors: john.raby at us.army.mil
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=43402
>>>
>>
>>
>> Classification: UNCLASSIFIED
>> Caveats: NONE
>>
>> In my Point-Stat config file, I have the value of "n_boot_rep" set
to
>> 0 to disable the computation of bootstrap confidence intervals
(CI).
>> The output text file (_cnt.txt file) confirms that only the columns
>> for the normal CI have values and the columns for the bootstrap CI
have
>> "NA"
>> as expected.
>>
>> However, when I look at my Stat-Analysis text output files I find
>> that the bootstrap CI columns show data values as if the bootstrap
CI
>> IS being computed. I do not use a config file for running Stat-
Analysis.
>> The job I run is aggregate_stat using MPR input to get CNT output.
>>
>> The V 2.0 Users Guide from March 2009 (pp 8-1 and 8-2) does state
>> that both the normal and bootstrap methods are computed by
>> Stat-Analysis by resampling the matched pairs.
>>
>> Are the bootstrap CI values I'm getting from Stat-Analysis valid if
I
>> disabled their computation in Point-Stat?
>>
>> I would like to use the CI values produced by Stat-Analysis, but I
am
>> concerned that there is an inconsistency because the bootstrap CI
>> values are not being computed by Point-Stat and yet I am seeing
>> values for bootstrap CI in my Stat-Analysis output.
>>
>> If the Stat-Analysis values for the bootstrap CI ARE valid, then
what
>> are the specifications being used to generate these CI values?
(such
>> as ci_alpha, boot_interval, boot_rep_prop, ...etc)
>>
>> Thanks.
>>
>> R/
>> John
>>
>> Mr John W. Raby, Meteorologist
>> U.S. Army Research Laboratory
>> White Sands Missile Range, NM 88002
>> (575) 678-2004 DSN 258-2004
>> FAX (575) 678-1230 DSN 258-1230
>> Email: john.raby at us.army.mil
>>
>> "When you can measure what you are speaking about and express it in
>> numbers, you know something about it, but when you cannot measure
it,
>> when you cannot express it in number, your knowledge is of a meagre
>> and unsatisfactory kind". - Lord Kelvin
>>
>>
>>
>> Classification: UNCLASSIFIED
>> Caveats: NONE
>>
>>
>>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
Classification: UNCLASSIFIED
Caveats: NONE
------------------------------------------------
More information about the Met_help
mailing list