[Met_help] [rt.rap.ucar.edu #70293] History for tc_stat for FSP at multiple lead times?
John Halley Gotway via RT
met_help at ucar.edu
Thu Jan 15 10:09:08 MST 2015
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello,
I am having trouble using tc_stat to discern frequency of superior
performance at multiple lead times between a small set of models.
For example, I have a filtered tcst file with 4 different models in it for
a one year period, and I want to calculate the FSP for each particular
model (they should add up to approximately 1).
So I copied over the default TCStat config file from the source tree and
added
-job summary -line_type TCMPR -amodel HWRF -column TK_ERR -dump_row
./tc_summary_job.tcst
to the jobs array. I have done nothing else. (note: the user guide is not
tremendously clear on the addition of these commands -- it appears they
need to be in quotes to define as a string and I still have not been able
to get tc_stat to run multiple filters in order without having to dump
temporary files and re-run tc_stat).
The output I get is:
COL_NAME: COLUMN TOTAL VALID MEAN MEAN_NCL MEAN_NCU STDEV
MIN P10 P25 P50 P75 P90 MAX SUM
TS_INT TS_IND FSP_TOTAL FSP_BEST FSP_TIES FSP
SUMMARY: TK_ERR 8588 1960 117.13681 112.04645 122.22717 114.98159
0.00000 24.65873 45.09268 82.44326 141.14324 260.62057 866.88303
229588.14214 NA NA 0 0 0 NA
which doesn't make any sense. It appears it is aggergating all track errors
at all lead times for that particular model. It also doesn't do any
comparison with the other models in the dataset.
I have also tried adding -amodel HWRF,GFDL. Now my results are
COL_NAME: COLUMN TOTAL VALID MEAN MEAN_NCL MEAN_NCU STDEV
MIN P10 P25 P50 P75 P90 MAX SUM
TS_INT TS_IND FSP_TOTAL FSP_BEST FSP_TIES FSP
SUMMARY: TK_ERR 8588 1960 117.13681 112.04645 122.22717 114.98159
0.00000 24.65873 45.09268 82.44326 141.14324 260.62057 866.88303
229588.14214 NA NA 0 0 0 NA
which again appears to confirm my suspicion that tc_stat is filtering out
the two models (all data points for the two models), mashing them together,
and calculating the statistics.
How do I run the utility such that
A) It is not automatically merging model and lead time?
B) I can get statistics for superior performance (ideally, how many times
at lead time XXX does model A outperform model B?)
In my personal case, I think sample files/data would go a long way to
helping bridge the gap between the document (which is sometimes tough to
decipher) and usage.
I am sorry if this is somewhat confusing/rambling -- please let me know if
I can clarify anything.
Thanks in advance for any assistance,
-Colin
--
Colin M. Zarzycki, ASP postdoctoral fellow
Atmospheric Modeling and Predictability
National Center for Atmospheric Research
http://www.colinzarzycki.com
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: tc_stat for FSP at multiple lead times?
From: John Halley Gotway
Time: Wed Jan 07 13:54:19 2015
Colin,
We actually had a very similar question about this functionality the
other
day. Clearly, we need to provide better examples in the
documentation.
Please try rerunning your job using this option "-by amodel". That
tells
tc_stat to run the same job once for each unique value it encounters
in the
AMODEL column. If your data contains output for 5 models, you should
see 5
output lines - one for each. And the FSP column should contain real
numbers. If you want to compute FSP only over 2 of those models, use
"-amodel HWRF,GFDL" as you already have done. FSP is only computed
when
looking at output for different models.
Lastly, I'll cut-and-paste some instructions I wrote this morning to
another user about running the plot_tcmpr.R plotting script. One of
the
output image types is a plot of the frequency of superior performance,
and
another is the rank. If you're looking at the FSP output from
tc_stat, you
may find these plot types useful.
Hope that helps.
Thanks,
John Halley Gotway
And here are those instructions...
Also, are you aware of the sample Rscript included in the MET release
for
plotting the output of tc_pairs? It does 2 things... It (1) uses the
command line arguments you pass it to call tc_stat and filter the
track
data down to the subset you want and (2) produces a handful of plot
types
using the filtered output of tc_pairs. Here's an example of how I ran
it
on some sample data and the output it creates:
# Must set MET_BUILD_BASE environment variable
setenv MET_BUILD_BASE
/var/autofs/mnt/dakota_d3/projects/MET/MET_releases/met-5.0
Rscript \
/var/autofs/mnt/dakota_d3/projects/MET/MET_releases/met-
5.0/scripts/Rscripts/plot_tcmpr.R
\
-lookin d2014_vx_20141117_reset/al/tc_pairs/tc_pairs_H3WI_* \
-lookin d2014_vx_20141117_reset/al/tc_pairs/tc_pairs_HWFI_* \
-filter "-amodel HWFI,H3WI" -plot
BOXPLOT,SCATTER,MEAN,MEDIAN,RELPERF,RANK
By default, it'll create boxplots of TK_ERR. Run it with no arguments
to
see the usage statement.
Use the "-filter" option to pass filtering options to the call to
tc_stat.
Here, I'm looking only at the HWFI and H3WI models.
Use "-plot" to specify which plot types should be created. I did all
possible types and have attached the results. You may be interested
in the
relative performance, in particular.
Use "-dep" to specify which columns of data you want to plot.
On Wed, Jan 7, 2015 at 12:24 PM, Colin Zarzycki via RT
<met_help at ucar.edu>
wrote:
>
> Wed Jan 07 12:24:36 2015: Request 70293 was acted upon.
> Transaction: Ticket created by zarzycki at ucar.edu
> Queue: met_help
> Subject: tc_stat for FSP at multiple lead times?
> Owner: Nobody
> Requestors: zarzycki at ucar.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70293 >
>
>
> Hello,
>
> I am having trouble using tc_stat to discern frequency of superior
> performance at multiple lead times between a small set of models.
>
> For example, I have a filtered tcst file with 4 different models in
it for
> a one year period, and I want to calculate the FSP for each
particular
> model (they should add up to approximately 1).
>
> So I copied over the default TCStat config file from the source tree
and
> added
>
> -job summary -line_type TCMPR -amodel HWRF -column TK_ERR -dump_row
> ./tc_summary_job.tcst
>
> to the jobs array. I have done nothing else. (note: the user guide
is not
> tremendously clear on the addition of these commands -- it appears
they
> need to be in quotes to define as a string and I still have not been
able
> to get tc_stat to run multiple filters in order without having to
dump
> temporary files and re-run tc_stat).
>
> The output I get is:
>
> COL_NAME: COLUMN TOTAL VALID MEAN MEAN_NCL MEAN_NCU STDEV
> MIN P10 P25 P50 P75 P90 MAX
SUM
> TS_INT TS_IND FSP_TOTAL FSP_BEST FSP_TIES FSP
> SUMMARY: TK_ERR 8588 1960 117.13681 112.04645 122.22717 114.98159
> 0.00000 24.65873 45.09268 82.44326 141.14324 260.62057 866.88303
> 229588.14214 NA NA 0 0 0 NA
>
> which doesn't make any sense. It appears it is aggergating all track
errors
> at all lead times for that particular model. It also doesn't do any
> comparison with the other models in the dataset.
>
> I have also tried adding -amodel HWRF,GFDL. Now my results are
>
> COL_NAME: COLUMN TOTAL VALID MEAN MEAN_NCL MEAN_NCU STDEV
> MIN P10 P25 P50 P75 P90 MAX
SUM
> TS_INT TS_IND FSP_TOTAL FSP_BEST FSP_TIES FSP
> SUMMARY: TK_ERR 8588 1960 117.13681 112.04645 122.22717 114.98159
> 0.00000 24.65873 45.09268 82.44326 141.14324 260.62057 866.88303
> 229588.14214 NA NA 0 0 0 NA
>
> which again appears to confirm my suspicion that tc_stat is
filtering out
> the two models (all data points for the two models), mashing them
together,
> and calculating the statistics.
>
> How do I run the utility such that
> A) It is not automatically merging model and lead time?
> B) I can get statistics for superior performance (ideally, how many
times
> at lead time XXX does model A outperform model B?)
>
> In my personal case, I think sample files/data would go a long way
to
> helping bridge the gap between the document (which is sometimes
tough to
> decipher) and usage.
>
> I am sorry if this is somewhat confusing/rambling -- please let me
know if
> I can clarify anything.
>
> Thanks in advance for any assistance,
>
> -Colin
>
> --
> Colin M. Zarzycki, ASP postdoctoral fellow
> Atmospheric Modeling and Predictability
> National Center for Atmospheric Research
> http://www.colinzarzycki.com
>
>
------------------------------------------------
Subject: tc_stat for FSP at multiple lead times?
From: Colin Zarzycki
Time: Wed Jan 07 14:57:58 2015
Hi John,
Thank you for the quick response! It helped. The functionality is a
little
better now. Another couple questions.
1.) I notice if I put -by amodel,lead it does stratify by both model
and
lead time, but won't do FSP (I assume because it isn't sure how to
compare). Out of curiously, is there a quick way to have tc_stat first
break out by individual model, and then compare the FSP for all models
at
each lead time (as the R script does with the timeseries RELPERF
plot)?
Example: at 12 hours, model X has a FSP of 60%, at 24 hours it is 64%,
etc.
2.) Is there a quick way to dump the raw summary data that is plotted
from plot_tcmpr.R? For example, I am interested in the rank/FSP data
as a
function of lead time. I see that a table is generated using
get_case_data
in plot_tcmpr_util.R, so I could probably just dump that array into a
text
file somehow, but I am not an R guy so if there's something to do it
without me writing something new, that would be preferable.
Cheers,
-Colin
On Wed, Jan 7, 2015 at 1:54 PM, John Halley Gotway via RT
<met_help at ucar.edu
> wrote:
> Colin,
>
> We actually had a very similar question about this functionality the
other
> day. Clearly, we need to provide better examples in the
documentation.
>
> Please try rerunning your job using this option "-by amodel". That
tells
> tc_stat to run the same job once for each unique value it encounters
in the
> AMODEL column. If your data contains output for 5 models, you
should see 5
> output lines - one for each. And the FSP column should contain real
> numbers. If you want to compute FSP only over 2 of those models,
use
> "-amodel HWRF,GFDL" as you already have done. FSP is only computed
when
> looking at output for different models.
>
> Lastly, I'll cut-and-paste some instructions I wrote this morning to
> another user about running the plot_tcmpr.R plotting script. One of
the
> output image types is a plot of the frequency of superior
performance, and
> another is the rank. If you're looking at the FSP output from
tc_stat, you
> may find these plot types useful.
>
> Hope that helps.
>
> Thanks,
> John Halley Gotway
>
> And here are those instructions...
>
> Also, are you aware of the sample Rscript included in the MET
release for
> plotting the output of tc_pairs? It does 2 things... It (1) uses
the
> command line arguments you pass it to call tc_stat and filter the
track
> data down to the subset you want and (2) produces a handful of plot
types
> using the filtered output of tc_pairs. Here's an example of how I
ran it
> on some sample data and the output it creates:
>
> # Must set MET_BUILD_BASE environment variable
> setenv MET_BUILD_BASE
> /var/autofs/mnt/dakota_d3/projects/MET/MET_releases/met-5.0
>
> Rscript \
>
> /var/autofs/mnt/dakota_d3/projects/MET/MET_releases/met-
5.0/scripts/Rscripts/plot_tcmpr.R
> \
> -lookin d2014_vx_20141117_reset/al/tc_pairs/tc_pairs_H3WI_* \
> -lookin d2014_vx_20141117_reset/al/tc_pairs/tc_pairs_HWFI_* \
> -filter "-amodel HWFI,H3WI" -plot
BOXPLOT,SCATTER,MEAN,MEDIAN,RELPERF,RANK
>
> By default, it'll create boxplots of TK_ERR. Run it with no
arguments to
> see the usage statement.
> Use the "-filter" option to pass filtering options to the call to
tc_stat.
> Here, I'm looking only at the HWFI and H3WI models.
> Use "-plot" to specify which plot types should be created. I did
all
> possible types and have attached the results. You may be interested
in the
> relative performance, in particular.
> Use "-dep" to specify which columns of data you want to plot.
>
> On Wed, Jan 7, 2015 at 12:24 PM, Colin Zarzycki via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Wed Jan 07 12:24:36 2015: Request 70293 was acted upon.
> > Transaction: Ticket created by zarzycki at ucar.edu
> > Queue: met_help
> > Subject: tc_stat for FSP at multiple lead times?
> > Owner: Nobody
> > Requestors: zarzycki at ucar.edu
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70293 >
> >
> >
> > Hello,
> >
> > I am having trouble using tc_stat to discern frequency of superior
> > performance at multiple lead times between a small set of models.
> >
> > For example, I have a filtered tcst file with 4 different models
in it
> for
> > a one year period, and I want to calculate the FSP for each
particular
> > model (they should add up to approximately 1).
> >
> > So I copied over the default TCStat config file from the source
tree and
> > added
> >
> > -job summary -line_type TCMPR -amodel HWRF -column TK_ERR
-dump_row
> > ./tc_summary_job.tcst
> >
> > to the jobs array. I have done nothing else. (note: the user guide
is not
> > tremendously clear on the addition of these commands -- it appears
they
> > need to be in quotes to define as a string and I still have not
been able
> > to get tc_stat to run multiple filters in order without having to
dump
> > temporary files and re-run tc_stat).
> >
> > The output I get is:
> >
> > COL_NAME: COLUMN TOTAL VALID MEAN MEAN_NCL MEAN_NCU
STDEV
> > MIN P10 P25 P50 P75 P90 MAX
SUM
> > TS_INT TS_IND FSP_TOTAL FSP_BEST FSP_TIES FSP
> > SUMMARY: TK_ERR 8588 1960 117.13681 112.04645 122.22717
114.98159
> > 0.00000 24.65873 45.09268 82.44326 141.14324 260.62057 866.88303
> > 229588.14214 NA NA 0 0 0 NA
> >
> > which doesn't make any sense. It appears it is aggergating all
track
> errors
> > at all lead times for that particular model. It also doesn't do
any
> > comparison with the other models in the dataset.
> >
> > I have also tried adding -amodel HWRF,GFDL. Now my results are
> >
> > COL_NAME: COLUMN TOTAL VALID MEAN MEAN_NCL MEAN_NCU
STDEV
> > MIN P10 P25 P50 P75 P90 MAX
SUM
> > TS_INT TS_IND FSP_TOTAL FSP_BEST FSP_TIES FSP
> > SUMMARY: TK_ERR 8588 1960 117.13681 112.04645 122.22717
114.98159
> > 0.00000 24.65873 45.09268 82.44326 141.14324 260.62057 866.88303
> > 229588.14214 NA NA 0 0 0 NA
> >
> > which again appears to confirm my suspicion that tc_stat is
filtering out
> > the two models (all data points for the two models), mashing them
> together,
> > and calculating the statistics.
> >
> > How do I run the utility such that
> > A) It is not automatically merging model and lead time?
> > B) I can get statistics for superior performance (ideally, how
many times
> > at lead time XXX does model A outperform model B?)
> >
> > In my personal case, I think sample files/data would go a long way
to
> > helping bridge the gap between the document (which is sometimes
tough to
> > decipher) and usage.
> >
> > I am sorry if this is somewhat confusing/rambling -- please let me
know
> if
> > I can clarify anything.
> >
> > Thanks in advance for any assistance,
> >
> > -Colin
> >
> > --
> > Colin M. Zarzycki, ASP postdoctoral fellow
> > Atmospheric Modeling and Predictability
> > National Center for Atmospheric Research
> > http://www.colinzarzycki.com
> >
> >
>
>
--
Colin M. Zarzycki, ASP postdoctoral fellow
Atmospheric Modeling and Predictability
National Center for Atmospheric Research
http://www.colinzarzycki.com
------------------------------------------------
Subject: tc_stat for FSP at multiple lead times?
From: John Halley Gotway
Time: Wed Jan 07 15:52:59 2015
Colin,
(1) If you want to see the FSP output for each lead time, you'd need
to run
separate jobs to compute them. Run it once for 12-hours, again for
24,
again for 36, and so on. You could group those multiple jobs into a
single
configuration file and run it that way. But there's no way to define
a
single job on the command line (without a config file) and compute FSP
for
different lead times.
Here's why... FSP is only computed when "-by amodel" is used, and it's
only
computed over the "common" cases. For each unique case (i.e. storm,
initialization, lead time), tc_stat picks which model performed best.
When
you stratify by multiple values of AMODEL *and* LEAD time, there are
no
more cases common to all of the subsets since the lead time changes.
I'm
pretty sure I know what you'd like - compute FSP separately for each
permutation of the "-by" options excluding the AMODEL. Theoretically
that
could be done, but the internal logic is already pretty complex. So
that
won't work in the current version of tc_stat.
(2) As for writing the data that's plotted, the script doesn't
currently
support that. But that's a good idea - for each plot, it'd be pretty
slick
to dump the data that's plotted. Here's something you could do...
edit the
function get_case_data() in the file "plot_tcmpr_util.R" with the
following...
# Compute summary info for each case
case_data$MIN = aggregate(series_data$PLOT,
by=list(series_data$CASE), FUN=min)$x;
case_data$MAX = aggregate(series_data$PLOT,
by=list(series_data$CASE), FUN=max)$x;
case_data$WIN = aggregate(series_data$PLOT,
by=list(series_data$CASE), FUN=find_winner)$x;
case_data$DIFF = case_data$MAX - case_data$MIN;
case_data$RP_THRESH = unlist(lapply(case_data$LEAD_HR,
FUN=find_thresh));
case_data$DIFF_TEST = paste(case_data$DIFF, case_data$RP_THRESH,
sep='');
case_data$RESULT = unlist(lapply(case_data$DIFF_TEST,
FUN=eval_exp));
case_data$PLOT = ifelse(case_data$RESULT, case_data$WIN,
"TIE");
case_data$RANK_RANDOM = aggregate(series_data$PLOT,
by=list(series_data$CASE), FUN=rank_random)$x;
case_data$RANK_MIN = aggregate(series_data$PLOT,
by=list(series_data$CASE), FUN=rank_min)$x;
# Write the case data
cat("Writing case_data.txt\n");
write.table(case_data, "case_data.txt", quote=FALSE,
row.names=FALSE);
Each time this function is called, it'll over-write the file named
"case_data.txt". I had to modify 2 lines above that call by adding
"unlist" to get the call to "write.table" to work.
You could give that a shot.
Thanks,
John
On Wed, Jan 7, 2015 at 2:57 PM, Colin Zarzycki via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70293 >
>
> Hi John,
>
> Thank you for the quick response! It helped. The functionality is a
little
> better now. Another couple questions.
>
> 1.) I notice if I put -by amodel,lead it does stratify by both model
and
> lead time, but won't do FSP (I assume because it isn't sure how to
> compare). Out of curiously, is there a quick way to have tc_stat
first
> break out by individual model, and then compare the FSP for all
models at
> each lead time (as the R script does with the timeseries RELPERF
plot)?
> Example: at 12 hours, model X has a FSP of 60%, at 24 hours it is
64%, etc.
>
> 2.) Is there a quick way to dump the raw summary data that is
plotted
> from plot_tcmpr.R? For example, I am interested in the rank/FSP data
as a
> function of lead time. I see that a table is generated using
get_case_data
> in plot_tcmpr_util.R, so I could probably just dump that array into
a text
> file somehow, but I am not an R guy so if there's something to do it
> without me writing something new, that would be preferable.
>
> Cheers,
>
> -Colin
>
>
> On Wed, Jan 7, 2015 at 1:54 PM, John Halley Gotway via RT <
> met_help at ucar.edu
> > wrote:
>
> > Colin,
> >
> > We actually had a very similar question about this functionality
the
> other
> > day. Clearly, we need to provide better examples in the
documentation.
> >
> > Please try rerunning your job using this option "-by amodel".
That tells
> > tc_stat to run the same job once for each unique value it
encounters in
> the
> > AMODEL column. If your data contains output for 5 models, you
should
> see 5
> > output lines - one for each. And the FSP column should contain
real
> > numbers. If you want to compute FSP only over 2 of those models,
use
> > "-amodel HWRF,GFDL" as you already have done. FSP is only
computed when
> > looking at output for different models.
> >
> > Lastly, I'll cut-and-paste some instructions I wrote this morning
to
> > another user about running the plot_tcmpr.R plotting script. One
of the
> > output image types is a plot of the frequency of superior
performance,
> and
> > another is the rank. If you're looking at the FSP output from
tc_stat,
> you
> > may find these plot types useful.
> >
> > Hope that helps.
> >
> > Thanks,
> > John Halley Gotway
> >
> > And here are those instructions...
> >
> > Also, are you aware of the sample Rscript included in the MET
release for
> > plotting the output of tc_pairs? It does 2 things... It (1) uses
the
> > command line arguments you pass it to call tc_stat and filter the
track
> > data down to the subset you want and (2) produces a handful of
plot types
> > using the filtered output of tc_pairs. Here's an example of how I
ran it
> > on some sample data and the output it creates:
> >
> > # Must set MET_BUILD_BASE environment variable
> > setenv MET_BUILD_BASE
> > /var/autofs/mnt/dakota_d3/projects/MET/MET_releases/met-5.0
> >
> > Rscript \
> >
> >
> /var/autofs/mnt/dakota_d3/projects/MET/MET_releases/met-
5.0/scripts/Rscripts/plot_tcmpr.R
> > \
> > -lookin d2014_vx_20141117_reset/al/tc_pairs/tc_pairs_H3WI_* \
> > -lookin d2014_vx_20141117_reset/al/tc_pairs/tc_pairs_HWFI_* \
> > -filter "-amodel HWFI,H3WI" -plot
> BOXPLOT,SCATTER,MEAN,MEDIAN,RELPERF,RANK
> >
> > By default, it'll create boxplots of TK_ERR. Run it with no
arguments to
> > see the usage statement.
> > Use the "-filter" option to pass filtering options to the call to
> tc_stat.
> > Here, I'm looking only at the HWFI and H3WI models.
> > Use "-plot" to specify which plot types should be created. I did
all
> > possible types and have attached the results. You may be
interested in
> the
> > relative performance, in particular.
> > Use "-dep" to specify which columns of data you want to plot.
> >
> > On Wed, Jan 7, 2015 at 12:24 PM, Colin Zarzycki via RT <
> met_help at ucar.edu>
> > wrote:
> >
> > >
> > > Wed Jan 07 12:24:36 2015: Request 70293 was acted upon.
> > > Transaction: Ticket created by zarzycki at ucar.edu
> > > Queue: met_help
> > > Subject: tc_stat for FSP at multiple lead times?
> > > Owner: Nobody
> > > Requestors: zarzycki at ucar.edu
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70293
> >
> > >
> > >
> > > Hello,
> > >
> > > I am having trouble using tc_stat to discern frequency of
superior
> > > performance at multiple lead times between a small set of
models.
> > >
> > > For example, I have a filtered tcst file with 4 different models
in it
> > for
> > > a one year period, and I want to calculate the FSP for each
particular
> > > model (they should add up to approximately 1).
> > >
> > > So I copied over the default TCStat config file from the source
tree
> and
> > > added
> > >
> > > -job summary -line_type TCMPR -amodel HWRF -column TK_ERR
-dump_row
> > > ./tc_summary_job.tcst
> > >
> > > to the jobs array. I have done nothing else. (note: the user
guide is
> not
> > > tremendously clear on the addition of these commands -- it
appears they
> > > need to be in quotes to define as a string and I still have not
been
> able
> > > to get tc_stat to run multiple filters in order without having
to dump
> > > temporary files and re-run tc_stat).
> > >
> > > The output I get is:
> > >
> > > COL_NAME: COLUMN TOTAL VALID MEAN MEAN_NCL MEAN_NCU
STDEV
> > > MIN P10 P25 P50 P75 P90 MAX
> SUM
> > > TS_INT TS_IND FSP_TOTAL FSP_BEST FSP_TIES FSP
> > > SUMMARY: TK_ERR 8588 1960 117.13681 112.04645 122.22717
114.98159
> > > 0.00000 24.65873 45.09268 82.44326 141.14324 260.62057 866.88303
> > > 229588.14214 NA NA 0 0 0 NA
> > >
> > > which doesn't make any sense. It appears it is aggergating all
track
> > errors
> > > at all lead times for that particular model. It also doesn't do
any
> > > comparison with the other models in the dataset.
> > >
> > > I have also tried adding -amodel HWRF,GFDL. Now my results are
> > >
> > > COL_NAME: COLUMN TOTAL VALID MEAN MEAN_NCL MEAN_NCU
STDEV
> > > MIN P10 P25 P50 P75 P90 MAX
> SUM
> > > TS_INT TS_IND FSP_TOTAL FSP_BEST FSP_TIES FSP
> > > SUMMARY: TK_ERR 8588 1960 117.13681 112.04645 122.22717
114.98159
> > > 0.00000 24.65873 45.09268 82.44326 141.14324 260.62057 866.88303
> > > 229588.14214 NA NA 0 0 0 NA
> > >
> > > which again appears to confirm my suspicion that tc_stat is
filtering
> out
> > > the two models (all data points for the two models), mashing
them
> > together,
> > > and calculating the statistics.
> > >
> > > How do I run the utility such that
> > > A) It is not automatically merging model and lead time?
> > > B) I can get statistics for superior performance (ideally, how
many
> times
> > > at lead time XXX does model A outperform model B?)
> > >
> > > In my personal case, I think sample files/data would go a long
way to
> > > helping bridge the gap between the document (which is sometimes
tough
> to
> > > decipher) and usage.
> > >
> > > I am sorry if this is somewhat confusing/rambling -- please let
me know
> > if
> > > I can clarify anything.
> > >
> > > Thanks in advance for any assistance,
> > >
> > > -Colin
> > >
> > > --
> > > Colin M. Zarzycki, ASP postdoctoral fellow
> > > Atmospheric Modeling and Predictability
> > > National Center for Atmospheric Research
> > > http://www.colinzarzycki.com
> > >
> > >
> >
> >
>
>
> --
> Colin M. Zarzycki, ASP postdoctoral fellow
> Atmospheric Modeling and Predictability
> National Center for Atmospheric Research
> http://www.colinzarzycki.com
>
>
------------------------------------------------
More information about the Met_help
mailing list