[Met_help] [rt.rap.ucar.edu #68757] History for Question about Stat-Analysis (UNCLASSIFIED)
John Halley Gotway via RT
met_help at ucar.edu
Tue Aug 26 15:48:16 MDT 2014
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Classification: UNCLASSIFIED
Caveats: NONE
I wanted the confidence intervals for two different alpha values in
Stat-Analysis output, so I specified the two values in the command line as:
-alpha 0.05 -alpha 0.0071.
However, when I ran this , it gave a WARNING about "no matching STAT lines
found". Is this because I had not run Point-Stat with these two alpha values
specified?
Thanks.
R/
John
Classification: UNCLASSIFIED
Caveats: NONE
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Question about Stat-Analysis (UNCLASSIFIED)
From: John Halley Gotway
Time: Mon Aug 25 15:41:18 2014
John,
Depends on the type of job you're running...
In general, the "-column_name" options define filtering criteria for
the
input STAT lines. So the "-alpha" option look at the column named
"ALPHA"
in the input STAT lines and only keeps the lines that match the
value(s)
you specified. If you're trying to filter the input STAT lines, then
"-alpha" will allow you to subset the input lines by the value in the
ALPHA
column.
However, I'm guessing you're trying to run an aggregate-stat job on
matched
pair (MPR), contingency table (CTC), or partial sums (SL1L2) lines and
want
to specify an alpha value to be used for the confidence intervals. If
that's the case, you should use the "-out_alpha" job command option.
That
specifies the alpha value to be used when computing confidence
intervals.
It's is a bit confusing but "-column_name" (like -alpha) type options
filter the input column names, and "-out_name" (like -out_alpha)
defines
setting to be used in the output.
Also, an aggregate-stat job can only handle one "-out_alpha" setting
at a
time. If you want to use 2 of them, you'll need to run that job
twice.
Thanks,
John
On Mon, Aug 25, 2014 at 3:13 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Mon Aug 25 15:13:07 2014: Request 68757 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: Question about Stat-Analysis (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68757 >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> I wanted the confidence intervals for two different alpha values in
> Stat-Analysis output, so I specified the two values in the command
line as:
>
> -alpha 0.05 -alpha 0.0071.
>
> However, when I ran this , it gave a WARNING about "no matching STAT
lines
> found". Is this because I had not run Point-Stat with these two
alpha
> values
> specified?
>
> Thanks.
>
> R/
> John
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #68757] Question about Stat-Analysis (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Mon Aug 25 16:05:16 2014
John -
Thanks for the quick response. I should have specified the way I ran
Stat-Analysis. I used the following command:
stat_analysis -lookin
~/MET_PointStat_Reen_FINAL/results_m3o3_R/20120207 -out
~/MET_StatAnalysis/Summary_byHour/m3o3_Reen_1km/20120207/hr00/m3o3_Reen_1km_SFC_TMP_hr00.txt
-v 3 -job aggregate_stat -line_type MPR -out_line_type CNT -obtype
ADPSFC -fcst_var TMP -fcst_lev Z2 -fcst_lead 000000 -alpha 0.05 -alpha
0.0071 -log
~/MET_StatAnalysis/logs/Summary_by_hour/m3o3_Reen_1km_sfctmp/20120207/hr00_log
Your guess about the way I ran it (aggregate-stat job) and then the
explanation based on that guess fixed the problem. Just ran with
-out_alpha 0.0071 instead of -alpha 0.0071 and it ran with no WARNING
and produced the expected output. Will the confidence limits be based
on .0071 for alpha or just .01? The "processing job" dialog shows .01.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Monday, August 25, 2014 3:41 PM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #68757] Question about Stat-Analysis
(UNCLASSIFIED)
John,
Depends on the type of job you're running...
In general, the "-column_name" options define filtering criteria for
the
input STAT lines. So the "-alpha" option look at the column named
"ALPHA"
in the input STAT lines and only keeps the lines that match the
value(s)
you specified. If you're trying to filter the input STAT lines, then
"-alpha" will allow you to subset the input lines by the value in the
ALPHA
column.
However, I'm guessing you're trying to run an aggregate-stat job on
matched
pair (MPR), contingency table (CTC), or partial sums (SL1L2) lines and
want
to specify an alpha value to be used for the confidence intervals. If
that's the case, you should use the "-out_alpha" job command option.
That
specifies the alpha value to be used when computing confidence
intervals.
It's is a bit confusing but "-column_name" (like -alpha) type options
filter the input column names, and "-out_name" (like -out_alpha)
defines
setting to be used in the output.
Also, an aggregate-stat job can only handle one "-out_alpha" setting
at a
time. If you want to use 2 of them, you'll need to run that job
twice.
Thanks,
John
On Mon, Aug 25, 2014 at 3:13 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Mon Aug 25 15:13:07 2014: Request 68757 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: Question about Stat-Analysis (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68757 >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> I wanted the confidence intervals for two different alpha values in
> Stat-Analysis output, so I specified the two values in the command
line as:
>
> -alpha 0.05 -alpha 0.0071.
>
> However, when I ran this , it gave a WARNING about "no matching STAT
lines
> found". Is this because I had not run Point-Stat with these two
alpha
> values
> specified?
>
> Thanks.
>
> R/
> John
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>
------------------------------------------------
Subject: Question about Stat-Analysis (UNCLASSIFIED)
From: John Halley Gotway
Time: Mon Aug 25 17:01:08 2014
John,
If you closely look at the MPR lines you're passing to STAT-Analysis,
you'll see that "ALPHA" column contains "NA" in all of them. It's NA
because the confidence interval alpha value does not apply to a
matched
pair line. It only applies to line types that contains confidence
intervals, like CNT and CTS, for example. The ALPHA column in those
lines
should be a real value, like 0.05 or whatever you've chosen.
Therefore when defining a STAT-Analysis aggregate_stat job, you should
not
specify an "-alpha" value at all. In fact, specifying one should
cause you
to match 0 MPR lines because the column value should be NA.
Using the information you sent, your job should look like this:
stat_analysis \
-lookin ~/MET_PointStat_Reen_FINAL/results_m3o3_R/20120207 \
-out
~/MET_StatAnalysis/Summary_byHour/m3o3_Reen_1km/20120207/hr00/m3o3_Reen_1km_SFC_TMP_hr00.txt
\
-v 3 \
-job aggregate_stat -line_type MPR -out_line_type CNT \
-obtype ADPSFC -fcst_var TMP -fcst_lev Z2 -fcst_lead 000000 \
-out_alpha 0.0071 \
-log
~/MET_StatAnalysis/logs/Summary_by_hour/m3o3_Reen_1km_sfctmp/20120207/hr00_log
Note that I've only used "-out_alpha 0.0071".
When I run a job like this, I do see "-out_alpha 0.01" in the job
command
line. I ran it through the debugger and see that the value of 0.0071
is
actually being used, but its being rounded to 2 decimal points in the
output string. I'll take a look at that to see if we can make that
behave
better in the next release.
Thanks,
John
On Mon, Aug 25, 2014 at 4:05 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68757 >
>
> John -
>
> Thanks for the quick response. I should have specified the way I ran
> Stat-Analysis. I used the following command:
>
> stat_analysis -lookin
~/MET_PointStat_Reen_FINAL/results_m3o3_R/20120207
> -out
>
~/MET_StatAnalysis/Summary_byHour/m3o3_Reen_1km/20120207/hr00/m3o3_Reen_1km_SFC_TMP_hr00.txt
> -v 3 -job aggregate_stat -line_type MPR -out_line_type CNT -obtype
ADPSFC
> -fcst_var TMP -fcst_lev Z2 -fcst_lead 000000 -alpha 0.05 -alpha
0.0071 -log
>
~/MET_StatAnalysis/logs/Summary_by_hour/m3o3_Reen_1km_sfctmp/20120207/hr00_log
>
> Your guess about the way I ran it (aggregate-stat job) and then the
> explanation based on that guess fixed the problem. Just ran with
-out_alpha
> 0.0071 instead of -alpha 0.0071 and it ran with no WARNING and
produced the
> expected output. Will the confidence limits be based on .0071 for
alpha or
> just .01? The "processing job" dialog shows .01.
>
> R/
> John
>
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Monday, August 25, 2014 3:41 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #68757] Question about Stat-Analysis
> (UNCLASSIFIED)
>
> John,
>
> Depends on the type of job you're running...
>
> In general, the "-column_name" options define filtering criteria for
the
> input STAT lines. So the "-alpha" option look at the column named
"ALPHA"
> in the input STAT lines and only keeps the lines that match the
value(s)
> you specified. If you're trying to filter the input STAT lines,
then
> "-alpha" will allow you to subset the input lines by the value in
the ALPHA
> column.
>
> However, I'm guessing you're trying to run an aggregate-stat job on
matched
> pair (MPR), contingency table (CTC), or partial sums (SL1L2) lines
and want
> to specify an alpha value to be used for the confidence intervals.
If
> that's the case, you should use the "-out_alpha" job command option.
That
> specifies the alpha value to be used when computing confidence
intervals.
>
> It's is a bit confusing but "-column_name" (like -alpha) type
options
> filter the input column names, and "-out_name" (like -out_alpha)
defines
> setting to be used in the output.
>
> Also, an aggregate-stat job can only handle one "-out_alpha" setting
at a
> time. If you want to use 2 of them, you'll need to run that job
twice.
>
> Thanks,
> John
>
>
> On Mon, Aug 25, 2014 at 3:13 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Aug 25 15:13:07 2014: Request 68757 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: Question about Stat-Analysis (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68757 >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > I wanted the confidence intervals for two different alpha values
in
> > Stat-Analysis output, so I specified the two values in the command
line
> as:
> >
> > -alpha 0.05 -alpha 0.0071.
> >
> > However, when I ran this , it gave a WARNING about "no matching
STAT
> lines
> > found". Is this because I had not run Point-Stat with these two
alpha
> > values
> > specified?
> >
> > Thanks.
> >
> > R/
> > John
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Question about Stat-Analysis (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Aug 26 07:41:52 2014
Classification: UNCLASSIFIED
Caveats: NONE
John -
Thanks for the details on how -alpha and -out_alpha work and for your
consideration of possible changes for better behavior
I checked my CNT output from Point-Stat and it does specify my
intended alpha
of 0.05.
Yesterday, I confirmed the behavior you mention below when specifying
-alpha.
It stated "no matching STAT lines" and the output .txt file was
created, but 0
contents.
Later, yesterday, I did successfully run after changing the
specification
to -out_alpha 0.0071. The log file for that run reports -out_alpha
0.01
confirming the behavior you saw in the job command line below. Would
be nice
to see more decimal places, because it is an issue for the following
reason:
I would like to generate Stat-Analysis output with two different alpha
values.
One for the 95% CI and another for the CI defined by applying the
Bonferroni
Correction so I can compare a family of error statistics to see if
they are
within the overall Bonferroni limits. These limits are calculated by
taking
the alpha value of .05 and dividing by the number of members in the
family.
This will typically (at least as I have learned from my limited
testing)
result in a much smaller alpha value which becomes important to
establish the
new limits. In my case, the new alpha value is 0.0071 which gives an
effective
level of confidence of 99.3% instead of 95%. This widens the error
bars to
allow for the fact that you are comparing a family of statistical
values
rather than just one statistic.
So it would be nice to have confirmation that the alpha value used was
0.0071
rather than just 0.01.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, August 25, 2014 5:01 PM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #68757] Question about Stat-Analysis
(UNCLASSIFIED)
John,
If you closely look at the MPR lines you're passing to STAT-Analysis,
you'll
see that "ALPHA" column contains "NA" in all of them. It's NA because
the
confidence interval alpha value does not apply to a matched pair line.
It
only applies to line types that contains confidence intervals, like
CNT and
CTS, for example. The ALPHA column in those lines should be a real
value,
like 0.05 or whatever you've chosen.
Therefore when defining a STAT-Analysis aggregate_stat job, you should
not
specify an "-alpha" value at all. In fact, specifying one should
cause you to
match 0 MPR lines because the column value should be NA.
Using the information you sent, your job should look like this:
stat_analysis \
-lookin ~/MET_PointStat_Reen_FINAL/results_m3o3_R/20120207 \ -out
~/MET_StatAnalysis/Summary_byHour/m3o3_Reen_1km/20120207/hr00/m3o3_Reen_1km_SFC_TMP_hr00.txt
\
-v 3 \
-job aggregate_stat -line_type MPR -out_line_type CNT \ -obtype
ADPSFC -fcst_var TMP -fcst_lev Z2 -fcst_lead 000000 \ -out_alpha
0.0071 \ -log
~/MET_StatAnalysis/logs/Summary_by_hour/m3o3_Reen_1km_sfctmp/20120207/hr00_log
Note that I've only used "-out_alpha 0.0071".
When I run a job like this, I do see "-out_alpha 0.01" in the job
command
line. I ran it through the debugger and see that the value of 0.0071
is
actually being used, but its being rounded to 2 decimal points in the
output
string. I'll take a look at that to see if we can make that behave
better in
the next release.
Thanks,
John
On Mon, Aug 25, 2014 at 4:05 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68757 >
>
> John -
>
> Thanks for the quick response. I should have specified the way I ran
> Stat-Analysis. I used the following command:
>
> stat_analysis -lookin
> ~/MET_PointStat_Reen_FINAL/results_m3o3_R/20120207
> -out
>
~/MET_StatAnalysis/Summary_byHour/m3o3_Reen_1km/20120207/hr00/m3o3_Ree
> n_1km_SFC_TMP_hr00.txt -v 3 -job aggregate_stat -line_type MPR
> -out_line_type CNT -obtype ADPSFC -fcst_var TMP -fcst_lev Z2
> -fcst_lead 000000 -alpha 0.05 -alpha 0.0071 -log
>
~/MET_StatAnalysis/logs/Summary_by_hour/m3o3_Reen_1km_sfctmp/20120207/
> hr00_log
>
> Your guess about the way I ran it (aggregate-stat job) and then the
> explanation based on that guess fixed the problem. Just ran with
> -out_alpha
> 0.0071 instead of -alpha 0.0071 and it ran with no WARNING and
> produced the expected output. Will the confidence limits be based on
> .0071 for alpha or just .01? The "processing job" dialog shows .01.
>
> R/
> John
>
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Monday, August 25, 2014 3:41 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #68757] Question about Stat-Analysis
> (UNCLASSIFIED)
>
> John,
>
> Depends on the type of job you're running...
>
> In general, the "-column_name" options define filtering criteria for
> the input STAT lines. So the "-alpha" option look at the column
named
> "ALPHA"
> in the input STAT lines and only keeps the lines that match the
> value(s) you specified. If you're trying to filter the input STAT
> lines, then "-alpha" will allow you to subset the input lines by the
> value in the ALPHA column.
>
> However, I'm guessing you're trying to run an aggregate-stat job on
> matched pair (MPR), contingency table (CTC), or partial sums (SL1L2)
> lines and want to specify an alpha value to be used for the
confidence
> intervals. If that's the case, you should use the "-out_alpha" job
> command option. That specifies the alpha value to be used when
computing
> confidence intervals.
>
> It's is a bit confusing but "-column_name" (like -alpha) type
options
> filter the input column names, and "-out_name" (like -out_alpha)
> defines setting to be used in the output.
>
> Also, an aggregate-stat job can only handle one "-out_alpha" setting
> at a time. If you want to use 2 of them, you'll need to run that
job twice.
>
> Thanks,
> John
>
>
> On Mon, Aug 25, 2014 at 3:13 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Aug 25 15:13:07 2014: Request 68757 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: Question about Stat-Analysis (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68757 >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > I wanted the confidence intervals for two different alpha values
in
> > Stat-Analysis output, so I specified the two values in the command
> > line
> as:
> >
> > -alpha 0.05 -alpha 0.0071.
> >
> > However, when I ran this , it gave a WARNING about "no matching
STAT
> lines
> > found". Is this because I had not run Point-Stat with these two
> > alpha values specified?
> >
> > Thanks.
> >
> > R/
> > John
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
>
>
Classification: UNCLASSIFIED
Caveats: NONE
------------------------------------------------
Subject: Question about Stat-Analysis (UNCLASSIFIED)
From: John Halley Gotway
Time: Tue Aug 26 09:53:29 2014
John,
Unfortunately, changing the precision in that output line requires
recompiling, but the change is very easy. Just edit line 45 of this
file:
METv4.1/src/basic/vx_log/concat_string.h
Old:
static const int concat_string_default_precision = 2;
New:
static const int concat_string_default_precision = 5;
This will use 5 decimal places of precision in the strings written by
MET.
And then recompile. I'll go ahead and make this change for the next
release.
Thanks,
John
On Tue, Aug 26, 2014 at 7:41 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68757 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for the details on how -alpha and -out_alpha work and for
your
> consideration of possible changes for better behavior
>
> I checked my CNT output from Point-Stat and it does specify my
intended
> alpha
> of 0.05.
>
> Yesterday, I confirmed the behavior you mention below when
specifying
> -alpha.
> It stated "no matching STAT lines" and the output .txt file was
created,
> but 0
> contents.
>
> Later, yesterday, I did successfully run after changing the
specification
> to -out_alpha 0.0071. The log file for that run reports -out_alpha
0.01
> confirming the behavior you saw in the job command line below. Would
be
> nice
> to see more decimal places, because it is an issue for the following
> reason:
>
> I would like to generate Stat-Analysis output with two different
alpha
> values.
> One for the 95% CI and another for the CI defined by applying the
> Bonferroni
> Correction so I can compare a family of error statistics to see if
they are
> within the overall Bonferroni limits. These limits are calculated by
taking
> the alpha value of .05 and dividing by the number of members in the
family.
> This will typically (at least as I have learned from my limited
testing)
> result in a much smaller alpha value which becomes important to
establish
> the
> new limits. In my case, the new alpha value is 0.0071 which gives an
> effective
> level of confidence of 99.3% instead of 95%. This widens the error
bars to
> allow for the fact that you are comparing a family of statistical
values
> rather than just one statistic.
>
> So it would be nice to have confirmation that the alpha value used
was
> 0.0071
> rather than just 0.01.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Monday, August 25, 2014 5:01 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #68757] Question about Stat-Analysis
> (UNCLASSIFIED)
>
> John,
>
> If you closely look at the MPR lines you're passing to STAT-
Analysis,
> you'll
> see that "ALPHA" column contains "NA" in all of them. It's NA
because the
> confidence interval alpha value does not apply to a matched pair
line. It
> only applies to line types that contains confidence intervals, like
CNT and
> CTS, for example. The ALPHA column in those lines should be a real
value,
> like 0.05 or whatever you've chosen.
>
> Therefore when defining a STAT-Analysis aggregate_stat job, you
should not
> specify an "-alpha" value at all. In fact, specifying one should
cause
> you to
> match 0 MPR lines because the column value should be NA.
>
> Using the information you sent, your job should look like this:
> stat_analysis \
> -lookin ~/MET_PointStat_Reen_FINAL/results_m3o3_R/20120207 \ -out
>
>
~/MET_StatAnalysis/Summary_byHour/m3o3_Reen_1km/20120207/hr00/m3o3_Reen_1km_SFC_TMP_hr00.txt
> \
> -v 3 \
> -job aggregate_stat -line_type MPR -out_line_type CNT \ -obtype
> ADPSFC -fcst_var TMP -fcst_lev Z2 -fcst_lead 000000 \ -out_alpha
0.0071 \
> -log
>
>
~/MET_StatAnalysis/logs/Summary_by_hour/m3o3_Reen_1km_sfctmp/20120207/hr00_log
>
> Note that I've only used "-out_alpha 0.0071".
>
> When I run a job like this, I do see "-out_alpha 0.01" in the job
command
> line. I ran it through the debugger and see that the value of
0.0071 is
> actually being used, but its being rounded to 2 decimal points in
the
> output
> string. I'll take a look at that to see if we can make that behave
better
> in
> the next release.
>
> Thanks,
> John
>
>
>
>
>
>
> On Mon, Aug 25, 2014 at 4:05 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68757 >
> >
> > John -
> >
> > Thanks for the quick response. I should have specified the way I
ran
> > Stat-Analysis. I used the following command:
> >
> > stat_analysis -lookin
> > ~/MET_PointStat_Reen_FINAL/results_m3o3_R/20120207
> > -out
> >
~/MET_StatAnalysis/Summary_byHour/m3o3_Reen_1km/20120207/hr00/m3o3_Ree
> > n_1km_SFC_TMP_hr00.txt -v 3 -job aggregate_stat -line_type MPR
> > -out_line_type CNT -obtype ADPSFC -fcst_var TMP -fcst_lev Z2
> > -fcst_lead 000000 -alpha 0.05 -alpha 0.0071 -log
> >
~/MET_StatAnalysis/logs/Summary_by_hour/m3o3_Reen_1km_sfctmp/20120207/
> > hr00_log
> >
> > Your guess about the way I ran it (aggregate-stat job) and then
the
> > explanation based on that guess fixed the problem. Just ran with
> > -out_alpha
> > 0.0071 instead of -alpha 0.0071 and it ran with no WARNING and
> > produced the expected output. Will the confidence limits be based
on
> > .0071 for alpha or just .01? The "processing job" dialog shows
.01.
> >
> > R/
> > John
> >
> >
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Monday, August 25, 2014 3:41 PM
> > To: Raby, John W CIV USARMY ARL (US)
> > Subject: Re: [rt.rap.ucar.edu #68757] Question about Stat-Analysis
> > (UNCLASSIFIED)
> >
> > John,
> >
> > Depends on the type of job you're running...
> >
> > In general, the "-column_name" options define filtering criteria
for
> > the input STAT lines. So the "-alpha" option look at the column
named
> > "ALPHA"
> > in the input STAT lines and only keeps the lines that match the
> > value(s) you specified. If you're trying to filter the input STAT
> > lines, then "-alpha" will allow you to subset the input lines by
the
> > value in the ALPHA column.
> >
> > However, I'm guessing you're trying to run an aggregate-stat job
on
> > matched pair (MPR), contingency table (CTC), or partial sums
(SL1L2)
> > lines and want to specify an alpha value to be used for the
confidence
> > intervals. If that's the case, you should use the "-out_alpha"
job
> > command option. That specifies the alpha value to be used when
computing
> > confidence intervals.
> >
> > It's is a bit confusing but "-column_name" (like -alpha) type
options
> > filter the input column names, and "-out_name" (like -out_alpha)
> > defines setting to be used in the output.
> >
> > Also, an aggregate-stat job can only handle one "-out_alpha"
setting
> > at a time. If you want to use 2 of them, you'll need to run that
job
> twice.
> >
> > Thanks,
> > John
> >
> >
> > On Mon, Aug 25, 2014 at 3:13 PM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Mon Aug 25 15:13:07 2014: Request 68757 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > Queue: met_help
> > > Subject: Question about Stat-Analysis (UNCLASSIFIED)
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68757 >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > I wanted the confidence intervals for two different alpha values
in
> > > Stat-Analysis output, so I specified the two values in the
command
> > > line
> > as:
> > >
> > > -alpha 0.05 -alpha 0.0071.
> > >
> > > However, when I ran this , it gave a WARNING about "no matching
STAT
> > lines
> > > found". Is this because I had not run Point-Stat with these two
> > > alpha values specified?
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>
------------------------------------------------
Subject: Question about Stat-Analysis (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Aug 26 10:53:57 2014
Classification: UNCLASSIFIED
Caveats: NONE
John -
Thanks for providing the method to correct the precision now and
making the
change a permanent one. Another thought I had was to allow for an
option for
the user to apply one or multiple -out_alpha job command options in a
single
run to accommodate a situation like I mentioned below. That way the
output
contains CI info for multiple alpha values without having to do
separate runs
for each alpha value. Something to consider.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, August 26, 2014 9:53 AM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #68757] Question about Stat-Analysis
(UNCLASSIFIED)
John,
Unfortunately, changing the precision in that output line requires
recompiling, but the change is very easy. Just edit line 45 of this
file:
METv4.1/src/basic/vx_log/concat_string.h
Old:
static const int concat_string_default_precision = 2;
New:
static const int concat_string_default_precision = 5;
This will use 5 decimal places of precision in the strings written by
MET.
And then recompile. I'll go ahead and make this change for the next
release.
Thanks,
John
On Tue, Aug 26, 2014 at 7:41 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68757 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for the details on how -alpha and -out_alpha work and for
your
> consideration of possible changes for better behavior
>
> I checked my CNT output from Point-Stat and it does specify my
> intended alpha of 0.05.
>
> Yesterday, I confirmed the behavior you mention below when
specifying
> -alpha.
> It stated "no matching STAT lines" and the output .txt file was
> created, but 0 contents.
>
> Later, yesterday, I did successfully run after changing the
> specification to -out_alpha 0.0071. The log file for that run
reports
> -out_alpha 0.01 confirming the behavior you saw in the job command
> line below. Would be nice to see more decimal places, because it is
an
> issue for the following
> reason:
>
> I would like to generate Stat-Analysis output with two different
alpha
> values.
> One for the 95% CI and another for the CI defined by applying the
> Bonferroni Correction so I can compare a family of error statistics
to
> see if they are within the overall Bonferroni limits. These limits
are
> calculated by taking the alpha value of .05 and dividing by the
number
> of members in the family.
> This will typically (at least as I have learned from my limited
> testing) result in a much smaller alpha value which becomes
important
> to establish the new limits. In my case, the new alpha value is
0.0071
> which gives an effective level of confidence of 99.3% instead of
95%.
> This widens the error bars to allow for the fact that you are
> comparing a family of statistical values rather than just one
> statistic.
>
> So it would be nice to have confirmation that the alpha value used
was
> 0.0071
> rather than just 0.01.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Monday, August 25, 2014 5:01 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #68757] Question about Stat-Analysis
> (UNCLASSIFIED)
>
> John,
>
> If you closely look at the MPR lines you're passing to STAT-
Analysis,
> you'll see that "ALPHA" column contains "NA" in all of them. It's
NA
> because the confidence interval alpha value does not apply to a
> matched pair line. It only applies to line types that contains
> confidence intervals, like CNT and CTS, for example. The ALPHA
column
> in those lines should be a real value, like 0.05 or whatever you've
> chosen.
>
> Therefore when defining a STAT-Analysis aggregate_stat job, you
should
> not specify an "-alpha" value at all. In fact, specifying one
should
> cause you to match 0 MPR lines because the column value should be
NA.
>
> Using the information you sent, your job should look like this:
> stat_analysis \
> -lookin ~/MET_PointStat_Reen_FINAL/results_m3o3_R/20120207 \ -out
>
>
~/MET_StatAnalysis/Summary_byHour/m3o3_Reen_1km/20120207/hr00/m3o3_Ree
> n_1km_SFC_TMP_hr00.txt
> \
> -v 3 \
> -job aggregate_stat -line_type MPR -out_line_type CNT \ -obtype
ADPSFC
> -fcst_var TMP -fcst_lev Z2 -fcst_lead 000000 \ -out_alpha 0.0071 \
> -log
>
>
~/MET_StatAnalysis/logs/Summary_by_hour/m3o3_Reen_1km_sfctmp/20120207/
> hr00_log
>
> Note that I've only used "-out_alpha 0.0071".
>
> When I run a job like this, I do see "-out_alpha 0.01" in the job
> command line. I ran it through the debugger and see that the value
of
> 0.0071 is actually being used, but its being rounded to 2 decimal
> points in the output string. I'll take a look at that to see if we
> can make that behave better in the next release.
>
> Thanks,
> John
>
>
>
>
>
>
> On Mon, Aug 25, 2014 at 4:05 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68757 >
> >
> > John -
> >
> > Thanks for the quick response. I should have specified the way I
ran
> > Stat-Analysis. I used the following command:
> >
> > stat_analysis -lookin
> > ~/MET_PointStat_Reen_FINAL/results_m3o3_R/20120207
> > -out
> >
~/MET_StatAnalysis/Summary_byHour/m3o3_Reen_1km/20120207/hr00/m3o3_R
> > ee n_1km_SFC_TMP_hr00.txt -v 3 -job aggregate_stat -line_type MPR
> > -out_line_type CNT -obtype ADPSFC -fcst_var TMP -fcst_lev Z2
> > -fcst_lead 000000 -alpha 0.05 -alpha 0.0071 -log
> >
~/MET_StatAnalysis/logs/Summary_by_hour/m3o3_Reen_1km_sfctmp/2012020
> > 7/
> > hr00_log
> >
> > Your guess about the way I ran it (aggregate-stat job) and then
the
> > explanation based on that guess fixed the problem. Just ran with
> > -out_alpha
> > 0.0071 instead of -alpha 0.0071 and it ran with no WARNING and
> > produced the expected output. Will the confidence limits be based
on
> > .0071 for alpha or just .01? The "processing job" dialog shows
.01.
> >
> > R/
> > John
> >
> >
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Monday, August 25, 2014 3:41 PM
> > To: Raby, John W CIV USARMY ARL (US)
> > Subject: Re: [rt.rap.ucar.edu #68757] Question about Stat-Analysis
> > (UNCLASSIFIED)
> >
> > John,
> >
> > Depends on the type of job you're running...
> >
> > In general, the "-column_name" options define filtering criteria
for
> > the input STAT lines. So the "-alpha" option look at the column
> > named "ALPHA"
> > in the input STAT lines and only keeps the lines that match the
> > value(s) you specified. If you're trying to filter the input STAT
> > lines, then "-alpha" will allow you to subset the input lines by
the
> > value in the ALPHA column.
> >
> > However, I'm guessing you're trying to run an aggregate-stat job
on
> > matched pair (MPR), contingency table (CTC), or partial sums
(SL1L2)
> > lines and want to specify an alpha value to be used for the
> > confidence intervals. If that's the case, you should use the
> > "-out_alpha" job command option. That specifies the alpha value
to
> > be used when computing confidence intervals.
> >
> > It's is a bit confusing but "-column_name" (like -alpha) type
> > options filter the input column names, and "-out_name" (like
> > -out_alpha) defines setting to be used in the output.
> >
> > Also, an aggregate-stat job can only handle one "-out_alpha"
setting
> > at a time. If you want to use 2 of them, you'll need to run that
> > job
> twice.
> >
> > Thanks,
> > John
> >
> >
> > On Mon, Aug 25, 2014 at 3:13 PM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Mon Aug 25 15:13:07 2014: Request 68757 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > Queue: met_help
> > > Subject: Question about Stat-Analysis (UNCLASSIFIED)
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68757 >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > I wanted the confidence intervals for two different alpha values
> > > in Stat-Analysis output, so I specified the two values in the
> > > command line
> > as:
> > >
> > > -alpha 0.05 -alpha 0.0071.
> > >
> > > However, when I ran this , it gave a WARNING about "no matching
> > > STAT
> > lines
> > > found". Is this because I had not run Point-Stat with these two
> > > alpha values specified?
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>
Classification: UNCLASSIFIED
Caveats: NONE
------------------------------------------------
Subject: Question about Stat-Analysis (UNCLASSIFIED)
From: John Halley Gotway
Time: Tue Aug 26 15:32:34 2014
John,
Thanks for the suggestion. Really this is the first time it's come
up.
The same logic would apply if you were running an aggregate_stat job
on
input MPR lines to compute an output contingency table (CTC) and
wanted to
do so using different thresholds. Just like with "-out_alpha", you'd
need
to run the job once for each output threshold using "-out_fcst_thresh"
and
"-out_obs_thresh".
We don't currently have a mechanism to do thist without running one
job for
each set of output criteria you'd like to use.
If you were doing this routinely as part of an automated system, I'd
suggest using a configuration file for STAT-Analysis. You'd set all
the
filtering criteria once at the top of the config file and then the
jobs
section would be pretty simple:
jobs = [
"-job aggregate_stat -line_type MPR -out_line_type CNT -out_alpha
0.05",
"-job aggregate_stat -line_type MPR -out_line_type CNT -out_alpha
0.0071"
];
Just a thought.
John
On Tue, Aug 26, 2014 at 10:53 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68757 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for providing the method to correct the precision now and
making the
> change a permanent one. Another thought I had was to allow for an
option
> for
> the user to apply one or multiple -out_alpha job command options in
a
> single
> run to accommodate a situation like I mentioned below. That way the
output
> contains CI info for multiple alpha values without having to do
separate
> runs
> for each alpha value. Something to consider.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, August 26, 2014 9:53 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #68757] Question about Stat-Analysis
> (UNCLASSIFIED)
>
> John,
>
> Unfortunately, changing the precision in that output line requires
> recompiling, but the change is very easy. Just edit line 45 of this
file:
> METv4.1/src/basic/vx_log/concat_string.h
> Old:
> static const int concat_string_default_precision = 2;
> New:
> static const int concat_string_default_precision = 5;
>
> This will use 5 decimal places of precision in the strings written
by MET.
>
> And then recompile. I'll go ahead and make this change for the next
> release.
>
> Thanks,
> John
>
>
> On Tue, Aug 26, 2014 at 7:41 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68757 >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > John -
> >
> > Thanks for the details on how -alpha and -out_alpha work and for
your
> > consideration of possible changes for better behavior
> >
> > I checked my CNT output from Point-Stat and it does specify my
> > intended alpha of 0.05.
> >
> > Yesterday, I confirmed the behavior you mention below when
specifying
> > -alpha.
> > It stated "no matching STAT lines" and the output .txt file was
> > created, but 0 contents.
> >
> > Later, yesterday, I did successfully run after changing the
> > specification to -out_alpha 0.0071. The log file for that run
reports
> > -out_alpha 0.01 confirming the behavior you saw in the job command
> > line below. Would be nice to see more decimal places, because it
is an
> > issue for the following
> > reason:
> >
> > I would like to generate Stat-Analysis output with two different
alpha
> > values.
> > One for the 95% CI and another for the CI defined by applying the
> > Bonferroni Correction so I can compare a family of error
statistics to
> > see if they are within the overall Bonferroni limits. These limits
are
> > calculated by taking the alpha value of .05 and dividing by the
number
> > of members in the family.
> > This will typically (at least as I have learned from my limited
> > testing) result in a much smaller alpha value which becomes
important
> > to establish the new limits. In my case, the new alpha value is
0.0071
> > which gives an effective level of confidence of 99.3% instead of
95%.
> > This widens the error bars to allow for the fact that you are
> > comparing a family of statistical values rather than just one
> > statistic.
> >
> > So it would be nice to have confirmation that the alpha value used
was
> > 0.0071
> > rather than just 0.01.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Monday, August 25, 2014 5:01 PM
> > To: Raby, John W CIV USARMY ARL (US)
> > Subject: Re: [rt.rap.ucar.edu #68757] Question about Stat-Analysis
> > (UNCLASSIFIED)
> >
> > John,
> >
> > If you closely look at the MPR lines you're passing to STAT-
Analysis,
> > you'll see that "ALPHA" column contains "NA" in all of them. It's
NA
> > because the confidence interval alpha value does not apply to a
> > matched pair line. It only applies to line types that contains
> > confidence intervals, like CNT and CTS, for example. The ALPHA
column
> > in those lines should be a real value, like 0.05 or whatever
you've
> > chosen.
> >
> > Therefore when defining a STAT-Analysis aggregate_stat job, you
should
> > not specify an "-alpha" value at all. In fact, specifying one
should
> > cause you to match 0 MPR lines because the column value should be
NA.
> >
> > Using the information you sent, your job should look like this:
> > stat_analysis \
> > -lookin ~/MET_PointStat_Reen_FINAL/results_m3o3_R/20120207 \ -out
> >
> >
~/MET_StatAnalysis/Summary_byHour/m3o3_Reen_1km/20120207/hr00/m3o3_Ree
> > n_1km_SFC_TMP_hr00.txt
> > \
> > -v 3 \
> > -job aggregate_stat -line_type MPR -out_line_type CNT \ -obtype
ADPSFC
> > -fcst_var TMP -fcst_lev Z2 -fcst_lead 000000 \ -out_alpha 0.0071 \
> > -log
> >
> >
~/MET_StatAnalysis/logs/Summary_by_hour/m3o3_Reen_1km_sfctmp/20120207/
> > hr00_log
> >
> > Note that I've only used "-out_alpha 0.0071".
> >
> > When I run a job like this, I do see "-out_alpha 0.01" in the job
> > command line. I ran it through the debugger and see that the
value of
> > 0.0071 is actually being used, but its being rounded to 2 decimal
> > points in the output string. I'll take a look at that to see if
we
> > can make that behave better in the next release.
> >
> > Thanks,
> > John
> >
> >
> >
> >
> >
> >
> > On Mon, Aug 25, 2014 at 4:05 PM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68757 >
> > >
> > > John -
> > >
> > > Thanks for the quick response. I should have specified the way I
ran
> > > Stat-Analysis. I used the following command:
> > >
> > > stat_analysis -lookin
> > > ~/MET_PointStat_Reen_FINAL/results_m3o3_R/20120207
> > > -out
> > >
~/MET_StatAnalysis/Summary_byHour/m3o3_Reen_1km/20120207/hr00/m3o3_R
> > > ee n_1km_SFC_TMP_hr00.txt -v 3 -job aggregate_stat -line_type
MPR
> > > -out_line_type CNT -obtype ADPSFC -fcst_var TMP -fcst_lev Z2
> > > -fcst_lead 000000 -alpha 0.05 -alpha 0.0071 -log
> > >
~/MET_StatAnalysis/logs/Summary_by_hour/m3o3_Reen_1km_sfctmp/2012020
> > > 7/
> > > hr00_log
> > >
> > > Your guess about the way I ran it (aggregate-stat job) and then
the
> > > explanation based on that guess fixed the problem. Just ran with
> > > -out_alpha
> > > 0.0071 instead of -alpha 0.0071 and it ran with no WARNING and
> > > produced the expected output. Will the confidence limits be
based on
> > > .0071 for alpha or just .01? The "processing job" dialog shows
.01.
> > >
> > > R/
> > > John
> > >
> > >
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Monday, August 25, 2014 3:41 PM
> > > To: Raby, John W CIV USARMY ARL (US)
> > > Subject: Re: [rt.rap.ucar.edu #68757] Question about Stat-
Analysis
> > > (UNCLASSIFIED)
> > >
> > > John,
> > >
> > > Depends on the type of job you're running...
> > >
> > > In general, the "-column_name" options define filtering criteria
for
> > > the input STAT lines. So the "-alpha" option look at the column
> > > named "ALPHA"
> > > in the input STAT lines and only keeps the lines that match the
> > > value(s) you specified. If you're trying to filter the input
STAT
> > > lines, then "-alpha" will allow you to subset the input lines by
the
> > > value in the ALPHA column.
> > >
> > > However, I'm guessing you're trying to run an aggregate-stat job
on
> > > matched pair (MPR), contingency table (CTC), or partial sums
(SL1L2)
> > > lines and want to specify an alpha value to be used for the
> > > confidence intervals. If that's the case, you should use the
> > > "-out_alpha" job command option. That specifies the alpha value
to
> > > be used when computing confidence intervals.
> > >
> > > It's is a bit confusing but "-column_name" (like -alpha) type
> > > options filter the input column names, and "-out_name" (like
> > > -out_alpha) defines setting to be used in the output.
> > >
> > > Also, an aggregate-stat job can only handle one "-out_alpha"
setting
> > > at a time. If you want to use 2 of them, you'll need to run
that
> > > job
> > twice.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Mon, Aug 25, 2014 at 3:13 PM, Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Mon Aug 25 15:13:07 2014: Request 68757 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > Queue: met_help
> > > > Subject: Question about Stat-Analysis (UNCLASSIFIED)
> > > > Owner: Nobody
> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68757 >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > > I wanted the confidence intervals for two different alpha
values
> > > > in Stat-Analysis output, so I specified the two values in the
> > > > command line
> > > as:
> > > >
> > > > -alpha 0.05 -alpha 0.0071.
> > > >
> > > > However, when I ran this , it gave a WARNING about "no
matching
> > > > STAT
> > > lines
> > > > found". Is this because I had not run Point-Stat with these
two
> > > > alpha values specified?
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>
------------------------------------------------
Subject: Question about Stat-Analysis (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Aug 26 15:39:57 2014
Classification: UNCLASSIFIED
Caveats: NONE
John -
I really should just work with config files. I've considered it and
now this
is some motivation . Thanks for the idea and the suggestion for the
job
section.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, August 26, 2014 3:33 PM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #68757] Question about Stat-Analysis
(UNCLASSIFIED)
John,
Thanks for the suggestion. Really this is the first time it's come
up.
The same logic would apply if you were running an aggregate_stat job
on input
MPR lines to compute an output contingency table (CTC) and wanted to
do so
using different thresholds. Just like with "-out_alpha", you'd need
to run
the job once for each output threshold using "-out_fcst_thresh" and
"-out_obs_thresh".
We don't currently have a mechanism to do thist without running one
job for
each set of output criteria you'd like to use.
If you were doing this routinely as part of an automated system, I'd
suggest
using a configuration file for STAT-Analysis. You'd set all the
filtering
criteria once at the top of the config file and then the jobs section
would be
pretty simple:
jobs = [
"-job aggregate_stat -line_type MPR -out_line_type CNT -out_alpha
0.05",
"-job aggregate_stat -line_type MPR -out_line_type CNT -out_alpha
0.0071"
];
Just a thought.
John
On Tue, Aug 26, 2014 at 10:53 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68757 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for providing the method to correct the precision now and
> making the change a permanent one. Another thought I had was to
allow
> for an option for the user to apply one or multiple -out_alpha job
> command options in a single run to accommodate a situation like I
> mentioned below. That way the output contains CI info for multiple
> alpha values without having to do separate runs for each alpha
value.
> Something to consider.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, August 26, 2014 9:53 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #68757] Question about Stat-Analysis
> (UNCLASSIFIED)
>
> John,
>
> Unfortunately, changing the precision in that output line requires
> recompiling, but the change is very easy. Just edit line 45 of this
file:
> METv4.1/src/basic/vx_log/concat_string.h
> Old:
> static const int concat_string_default_precision = 2;
> New:
> static const int concat_string_default_precision = 5;
>
> This will use 5 decimal places of precision in the strings written
by MET.
>
> And then recompile. I'll go ahead and make this change for the next
> release.
>
> Thanks,
> John
>
>
> On Tue, Aug 26, 2014 at 7:41 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68757 >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > John -
> >
> > Thanks for the details on how -alpha and -out_alpha work and for
> > your consideration of possible changes for better behavior
> >
> > I checked my CNT output from Point-Stat and it does specify my
> > intended alpha of 0.05.
> >
> > Yesterday, I confirmed the behavior you mention below when
> > specifying -alpha.
> > It stated "no matching STAT lines" and the output .txt file was
> > created, but 0 contents.
> >
> > Later, yesterday, I did successfully run after changing the
> > specification to -out_alpha 0.0071. The log file for that run
> > reports -out_alpha 0.01 confirming the behavior you saw in the job
> > command line below. Would be nice to see more decimal places,
> > because it is an issue for the following
> > reason:
> >
> > I would like to generate Stat-Analysis output with two different
> > alpha values.
> > One for the 95% CI and another for the CI defined by applying the
> > Bonferroni Correction so I can compare a family of error
statistics
> > to see if they are within the overall Bonferroni limits. These
> > limits are calculated by taking the alpha value of .05 and
dividing
> > by the number of members in the family.
> > This will typically (at least as I have learned from my limited
> > testing) result in a much smaller alpha value which becomes
> > important to establish the new limits. In my case, the new alpha
> > value is 0.0071 which gives an effective level of confidence of
99.3%
> > instead of 95%.
> > This widens the error bars to allow for the fact that you are
> > comparing a family of statistical values rather than just one
> > statistic.
> >
> > So it would be nice to have confirmation that the alpha value used
> > was
> > 0.0071
> > rather than just 0.01.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Monday, August 25, 2014 5:01 PM
> > To: Raby, John W CIV USARMY ARL (US)
> > Subject: Re: [rt.rap.ucar.edu #68757] Question about Stat-Analysis
> > (UNCLASSIFIED)
> >
> > John,
> >
> > If you closely look at the MPR lines you're passing to
> > STAT-Analysis, you'll see that "ALPHA" column contains "NA" in all
> > of them. It's NA because the confidence interval alpha value does
> > not apply to a matched pair line. It only applies to line types
> > that contains confidence intervals, like CNT and CTS, for example.
> > The ALPHA column in those lines should be a real value, like 0.05
or
> > whatever you've chosen.
> >
> > Therefore when defining a STAT-Analysis aggregate_stat job, you
> > should not specify an "-alpha" value at all. In fact, specifying
> > one should cause you to match 0 MPR lines because the column value
should
> > be NA.
> >
> > Using the information you sent, your job should look like this:
> > stat_analysis \
> > -lookin ~/MET_PointStat_Reen_FINAL/results_m3o3_R/20120207 \ -out
> >
> >
~/MET_StatAnalysis/Summary_byHour/m3o3_Reen_1km/20120207/hr00/m3o3_R
> > ee
> > n_1km_SFC_TMP_hr00.txt
> > \
> > -v 3 \
> > -job aggregate_stat -line_type MPR -out_line_type CNT \ -obtype
> > ADPSFC -fcst_var TMP -fcst_lev Z2 -fcst_lead 000000 \ -out_alpha
> > 0.0071 \ -log
> >
> >
~/MET_StatAnalysis/logs/Summary_by_hour/m3o3_Reen_1km_sfctmp/2012020
> > 7/
> > hr00_log
> >
> > Note that I've only used "-out_alpha 0.0071".
> >
> > When I run a job like this, I do see "-out_alpha 0.01" in the job
> > command line. I ran it through the debugger and see that the
value
> > of
> > 0.0071 is actually being used, but its being rounded to 2 decimal
> > points in the output string. I'll take a look at that to see if
we
> > can make that behave better in the next release.
> >
> > Thanks,
> > John
> >
> >
> >
> >
> >
> >
> > On Mon, Aug 25, 2014 at 4:05 PM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68757 >
> > >
> > > John -
> > >
> > > Thanks for the quick response. I should have specified the way I
> > > ran Stat-Analysis. I used the following command:
> > >
> > > stat_analysis -lookin
> > > ~/MET_PointStat_Reen_FINAL/results_m3o3_R/20120207
> > > -out
> > >
~/MET_StatAnalysis/Summary_byHour/m3o3_Reen_1km/20120207/hr00/m3o3
> > > _R ee n_1km_SFC_TMP_hr00.txt -v 3 -job aggregate_stat -line_type
> > > MPR -out_line_type CNT -obtype ADPSFC -fcst_var TMP -fcst_lev Z2
> > > -fcst_lead 000000 -alpha 0.05 -alpha 0.0071 -log
> > >
~/MET_StatAnalysis/logs/Summary_by_hour/m3o3_Reen_1km_sfctmp/20120
> > > 20
> > > 7/
> > > hr00_log
> > >
> > > Your guess about the way I ran it (aggregate-stat job) and then
> > > the explanation based on that guess fixed the problem. Just ran
> > > with -out_alpha
> > > 0.0071 instead of -alpha 0.0071 and it ran with no WARNING and
> > > produced the expected output. Will the confidence limits be
based
> > > on
> > > .0071 for alpha or just .01? The "processing job" dialog shows
.01.
> > >
> > > R/
> > > John
> > >
> > >
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Monday, August 25, 2014 3:41 PM
> > > To: Raby, John W CIV USARMY ARL (US)
> > > Subject: Re: [rt.rap.ucar.edu #68757] Question about Stat-
Analysis
> > > (UNCLASSIFIED)
> > >
> > > John,
> > >
> > > Depends on the type of job you're running...
> > >
> > > In general, the "-column_name" options define filtering criteria
> > > for the input STAT lines. So the "-alpha" option look at the
> > > column named "ALPHA"
> > > in the input STAT lines and only keeps the lines that match the
> > > value(s) you specified. If you're trying to filter the input
STAT
> > > lines, then "-alpha" will allow you to subset the input lines by
> > > the value in the ALPHA column.
> > >
> > > However, I'm guessing you're trying to run an aggregate-stat job
> > > on matched pair (MPR), contingency table (CTC), or partial sums
> > > (SL1L2) lines and want to specify an alpha value to be used for
> > > the confidence intervals. If that's the case, you should use
the
> > > "-out_alpha" job command option. That specifies the alpha value
> > > to be used when computing confidence intervals.
> > >
> > > It's is a bit confusing but "-column_name" (like -alpha) type
> > > options filter the input column names, and "-out_name" (like
> > > -out_alpha) defines setting to be used in the output.
> > >
> > > Also, an aggregate-stat job can only handle one "-out_alpha"
> > > setting at a time. If you want to use 2 of them, you'll need to
> > > run that job
> > twice.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > On Mon, Aug 25, 2014 at 3:13 PM, Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Mon Aug 25 15:13:07 2014: Request 68757 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > Queue: met_help
> > > > Subject: Question about Stat-Analysis (UNCLASSIFIED)
> > > > Owner: Nobody
> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68757 >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > > I wanted the confidence intervals for two different alpha
values
> > > > in Stat-Analysis output, so I specified the two values in the
> > > > command line
> > > as:
> > > >
> > > > -alpha 0.05 -alpha 0.0071.
> > > >
> > > > However, when I ran this , it gave a WARNING about "no
matching
> > > > STAT
> > > lines
> > > > found". Is this because I had not run Point-Stat with these
two
> > > > alpha values specified?
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>
Classification: UNCLASSIFIED
Caveats: NONE
------------------------------------------------
More information about the Met_help
mailing list