[Met_help] [rt.rap.ucar.edu #70270] History for Frequency of Superior Performance
John Halley Gotway via RT
met_help at ucar.edu
Thu Jan 15 10:09:28 MST 2015
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Dear MET Help,
I am trying to analyze the frequency of superior performance (FSP) using
Version 5 of MET-TC. I can successfully use "tc_pairs" and "tc_stat" to
compute error statistics for a set of A-Deck models. However, although the
summary output from "tc_stat" creates columns for FSP analysis, it is "0"
for all rows that contain non-zero valid entries. Furthermore, the time
series independence analysis is "NA" for all rows that contain non-zero
valid entries.
How do I turn on the FSP analysis? What am I doing wrong? I can not find
options in the TCStatConfig file to change this. Please find a snapshot of
my "tc_stat" summary file. You can see the job options that I used in the
first line of this snapshot.
Thank you for your help!
Best,
Gus Alaka
[image: Inline image 1]
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Frequency of Superior Performance
From: John Halley Gotway
Time: Tue Jan 06 13:49:15 2015
Hello Gus,
I'd be happy to help you with this issue, but we can only provide
direct
support to registered users of MET. We do this to ensure that we
remain in
compliance with U.S. export control laws. You can fill out the
registration form here:
http://www.dtcenter.org/met/users/downloads/index.php
Just enter your email address and hit enter. That should take you to
the
registration page.
Please write me back once you've filled that out, and I should be able
to
offer more help.
Thanks,
John Halley Gotway
met_help at ucar.edu
On Tue, Jan 6, 2015 at 10:57 AM, Ghassan Alaka - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> Tue Jan 06 10:57:18 2015: Request 70270 was acted upon.
> Transaction: Ticket created by ghassan.alaka at noaa.gov
> Queue: met_help
> Subject: Frequency of Superior Performance
> Owner: Nobody
> Requestors: ghassan.alaka at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70270 >
>
>
> Dear MET Help,
>
> I am trying to analyze the frequency of superior performance (FSP)
using
> Version 5 of MET-TC. I can successfully use "tc_pairs" and "tc_stat"
to
> compute error statistics for a set of A-Deck models. However,
although the
> summary output from "tc_stat" creates columns for FSP analysis, it
is "0"
> for all rows that contain non-zero valid entries. Furthermore, the
time
> series independence analysis is "NA" for all rows that contain non-
zero
> valid entries.
>
> How do I turn on the FSP analysis? What am I doing wrong? I can not
find
> options in the TCStatConfig file to change this. Please find a
snapshot of
> my "tc_stat" summary file. You can see the job options that I used
in the
> first line of this snapshot.
>
> Thank you for your help!
>
> Best,
> Gus Alaka
>
> [image: Inline image 1]
>
>
------------------------------------------------
Subject: Frequency of Superior Performance
From: Ghassan Alaka - NOAA Affiliate
Time: Tue Jan 06 14:12:02 2015
Hi John,
I filled out the registration form and submitted. Looking forward to
your
assistance.
Best,
Gus
On Tue, Jan 6, 2015 at 3:49 PM, John Halley Gotway via RT
<met_help at ucar.edu
> wrote:
> Hello Gus,
>
> I'd be happy to help you with this issue, but we can only provide
direct
> support to registered users of MET. We do this to ensure that we
remain in
> compliance with U.S. export control laws. You can fill out the
> registration form here:
> http://www.dtcenter.org/met/users/downloads/index.php
> Just enter your email address and hit enter. That should take you
to the
> registration page.
>
> Please write me back once you've filled that out, and I should be
able to
> offer more help.
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu
>
> On Tue, Jan 6, 2015 at 10:57 AM, Ghassan Alaka - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Jan 06 10:57:18 2015: Request 70270 was acted upon.
> > Transaction: Ticket created by ghassan.alaka at noaa.gov
> > Queue: met_help
> > Subject: Frequency of Superior Performance
> > Owner: Nobody
> > Requestors: ghassan.alaka at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70270 >
> >
> >
> > Dear MET Help,
> >
> > I am trying to analyze the frequency of superior performance (FSP)
using
> > Version 5 of MET-TC. I can successfully use "tc_pairs" and
"tc_stat" to
> > compute error statistics for a set of A-Deck models. However,
although
> the
> > summary output from "tc_stat" creates columns for FSP analysis, it
is "0"
> > for all rows that contain non-zero valid entries. Furthermore, the
time
> > series independence analysis is "NA" for all rows that contain
non-zero
> > valid entries.
> >
> > How do I turn on the FSP analysis? What am I doing wrong? I can
not find
> > options in the TCStatConfig file to change this. Please find a
snapshot
> of
> > my "tc_stat" summary file. You can see the job options that I used
in the
> > first line of this snapshot.
> >
> > Thank you for your help!
> >
> > Best,
> > Gus Alaka
> >
> > [image: Inline image 1]
> >
> >
>
>
------------------------------------------------
Subject: Frequency of Superior Performance
From: John Halley Gotway
Time: Tue Jan 06 14:24:02 2015
Gus,
Great, thanks for doing that. If you were to rerun that same job you
ran
using a verbosity level of 4 (-v 4), you'd see the following line
printed
to the screen:
DEBUG 4: Skipping frequency of superior performance since the case
information does not contain "AMODEL".
The tc_stat job you ran looked at the 48 hour lead time for the HWRF
and
H3HW models but grouped together all the output. FSP is only computed
when
you have separate output for each model, and you can request that
using the
"-by AMODEL" option. I've attached the output created when running
the
following job on some sample data (note that I've used -by AMODEL):
met-5.0/bin/tc_stat \
-lookin d2014_vx_20141117_reset/al/tc_pairs/tc_pairs_H3WI_* \
-lookin d2014_vx_20141117_reset/al/tc_pairs/tc_pairs_HWFI_* \
-job summary -lead 480000 -column TRACK -amodel HWFI,H3WI -by
AMODEL
-out sample.out
Try adding "-by AMODEL" and see if that does the trick. If not, try
using
debug level 4 (-v 4) to see if there's any more useful log messages.
Hope that helps.
Thanks,
John
On Tue, Jan 6, 2015 at 2:12 PM, Ghassan Alaka - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70270 >
>
> Hi John,
>
> I filled out the registration form and submitted. Looking forward to
your
> assistance.
>
> Best,
> Gus
>
> On Tue, Jan 6, 2015 at 3:49 PM, John Halley Gotway via RT <
> met_help at ucar.edu
> > wrote:
>
> > Hello Gus,
> >
> > I'd be happy to help you with this issue, but we can only provide
direct
> > support to registered users of MET. We do this to ensure that we
remain
> in
> > compliance with U.S. export control laws. You can fill out the
> > registration form here:
> > http://www.dtcenter.org/met/users/downloads/index.php
> > Just enter your email address and hit enter. That should take
you to
> the
> > registration page.
> >
> > Please write me back once you've filled that out, and I should be
able to
> > offer more help.
> >
> > Thanks,
> > John Halley Gotway
> > met_help at ucar.edu
> >
> > On Tue, Jan 6, 2015 at 10:57 AM, Ghassan Alaka - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Jan 06 10:57:18 2015: Request 70270 was acted upon.
> > > Transaction: Ticket created by ghassan.alaka at noaa.gov
> > > Queue: met_help
> > > Subject: Frequency of Superior Performance
> > > Owner: Nobody
> > > Requestors: ghassan.alaka at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70270
> >
> > >
> > >
> > > Dear MET Help,
> > >
> > > I am trying to analyze the frequency of superior performance
(FSP)
> using
> > > Version 5 of MET-TC. I can successfully use "tc_pairs" and
"tc_stat" to
> > > compute error statistics for a set of A-Deck models. However,
although
> > the
> > > summary output from "tc_stat" creates columns for FSP analysis,
it is
> "0"
> > > for all rows that contain non-zero valid entries. Furthermore,
the time
> > > series independence analysis is "NA" for all rows that contain
non-zero
> > > valid entries.
> > >
> > > How do I turn on the FSP analysis? What am I doing wrong? I can
not
> find
> > > options in the TCStatConfig file to change this. Please find a
snapshot
> > of
> > > my "tc_stat" summary file. You can see the job options that I
used in
> the
> > > first line of this snapshot.
> > >
> > > Thank you for your help!
> > >
> > > Best,
> > > Gus Alaka
> > >
> > > [image: Inline image 1]
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Frequency of Superior Performance
From: Ghassan Alaka - NOAA Affiliate
Time: Tue Jan 06 15:58:43 2015
John,
That did the trick! Thank you so much. The syntax makes sense.
Now, do you have any idea why the time series independence columns
read
"NA"? Just curious about how to get those statistics. I'm hoping to
compute
confidence intervals conservatively, so I'm looking for the number of
independent samples rather than the total number of time stamps.
Thank you,
Gus
On Tue, Jan 6, 2015 at 4:24 PM, John Halley Gotway via RT
<met_help at ucar.edu
> wrote:
> Gus,
>
> Great, thanks for doing that. If you were to rerun that same job
you ran
> using a verbosity level of 4 (-v 4), you'd see the following line
printed
> to the screen:
> DEBUG 4: Skipping frequency of superior performance since the
case
> information does not contain "AMODEL".
>
> The tc_stat job you ran looked at the 48 hour lead time for the HWRF
and
> H3HW models but grouped together all the output. FSP is only
computed when
> you have separate output for each model, and you can request that
using the
> "-by AMODEL" option. I've attached the output created when running
the
> following job on some sample data (note that I've used -by AMODEL):
>
> met-5.0/bin/tc_stat \
> -lookin d2014_vx_20141117_reset/al/tc_pairs/tc_pairs_H3WI_* \
> -lookin d2014_vx_20141117_reset/al/tc_pairs/tc_pairs_HWFI_* \
> -job summary -lead 480000 -column TRACK -amodel HWFI,H3WI -by
AMODEL
> -out sample.out
>
> Try adding "-by AMODEL" and see if that does the trick. If not, try
using
> debug level 4 (-v 4) to see if there's any more useful log messages.
>
> Hope that helps.
>
> Thanks,
> John
>
>
>
>
> On Tue, Jan 6, 2015 at 2:12 PM, Ghassan Alaka - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70270 >
> >
> > Hi John,
> >
> > I filled out the registration form and submitted. Looking forward
to your
> > assistance.
> >
> > Best,
> > Gus
> >
> > On Tue, Jan 6, 2015 at 3:49 PM, John Halley Gotway via RT <
> > met_help at ucar.edu
> > > wrote:
> >
> > > Hello Gus,
> > >
> > > I'd be happy to help you with this issue, but we can only
provide
> direct
> > > support to registered users of MET. We do this to ensure that
we
> remain
> > in
> > > compliance with U.S. export control laws. You can fill out the
> > > registration form here:
> > > http://www.dtcenter.org/met/users/downloads/index.php
> > > Just enter your email address and hit enter. That should take
you to
> > the
> > > registration page.
> > >
> > > Please write me back once you've filled that out, and I should
be able
> to
> > > offer more help.
> > >
> > > Thanks,
> > > John Halley Gotway
> > > met_help at ucar.edu
> > >
> > > On Tue, Jan 6, 2015 at 10:57 AM, Ghassan Alaka - NOAA Affiliate
via RT
> <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Tue Jan 06 10:57:18 2015: Request 70270 was acted upon.
> > > > Transaction: Ticket created by ghassan.alaka at noaa.gov
> > > > Queue: met_help
> > > > Subject: Frequency of Superior Performance
> > > > Owner: Nobody
> > > > Requestors: ghassan.alaka at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70270
> > >
> > > >
> > > >
> > > > Dear MET Help,
> > > >
> > > > I am trying to analyze the frequency of superior performance
(FSP)
> > using
> > > > Version 5 of MET-TC. I can successfully use "tc_pairs" and
"tc_stat"
> to
> > > > compute error statistics for a set of A-Deck models. However,
> although
> > > the
> > > > summary output from "tc_stat" creates columns for FSP
analysis, it is
> > "0"
> > > > for all rows that contain non-zero valid entries. Furthermore,
the
> time
> > > > series independence analysis is "NA" for all rows that contain
> non-zero
> > > > valid entries.
> > > >
> > > > How do I turn on the FSP analysis? What am I doing wrong? I
can not
> > find
> > > > options in the TCStatConfig file to change this. Please find a
> snapshot
> > > of
> > > > my "tc_stat" summary file. You can see the job options that I
used in
> > the
> > > > first line of this snapshot.
> > > >
> > > > Thank you for your help!
> > > >
> > > > Best,
> > > > Gus Alaka
> > > >
> > > > [image: Inline image 1]
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Frequency of Superior Performance
From: John Halley Gotway
Time: Wed Jan 07 10:06:29 2015
Gus,
The code only computes time series independence when the data passed
to it
is a true time-series, meaning that the time spacing between points
remains
constant. If you run the job at verbosity level 4, you'll probably
see
messages like this:
DEBUG 4: Skipping time-series computations since the initialization
time
spacing changed: 21600 != 1274400
The time unit are seconds.
The tc_stat job you ran was looking only at 48 hour lead times, so the
time
spacing is basically the frequency of model initializations - likely
every
6 hours or 12 hours. But if you pass it data for multiple storms, the
time
steps will almost certainly not remain constant since there will be a
gap
in the initialization times between storms. Try running your job for
a
single storm (i.e. -storm_name BERTHA, or some other storm name). If
the
initialization times are equally spaced, you should see time series
info.
Or you could try fixing a single initialization time and look at all
lead
times. That's another type of time series. Did that do the trick?
I can provide you info about the details of running the tools and the
logic
they're using. But if you'd like more info about interpreting the
statistical output from MET, I could refer you to a statistician on
our
team. Just let me know.
Also, are you aware of the sample Rscript included in the MET release
for
plotting the output of tc_pairs? It does 2 things... It (1) uses the
command line arguments you pass it to call tc_stat and filter the
track
data down to the subset you want and (2) produces a handful of plot
types
using the filtered output of tc_pairs. Here's an example of how I ran
it
on some sample data and the output it creates:
# Must set MET_BUILD_BASE environment variable
setenv MET_BUILD_BASE
/var/autofs/mnt/dakota_d3/projects/MET/MET_releases/met-5.0
Rscript \
/var/autofs/mnt/dakota_d3/projects/MET/MET_releases/met-
5.0/scripts/Rscripts/plot_tcmpr.R
\
-lookin d2014_vx_20141117_reset/al/tc_pairs/tc_pairs_H3WI_* \
-lookin d2014_vx_20141117_reset/al/tc_pairs/tc_pairs_HWFI_* \
-filter "-amodel HWFI,H3WI" -plot
BOXPLOT,SCATTER,MEAN,MEDIAN,RELPERF,RANK
By default, it'll create boxplots of TK_ERR. Run it with no arguments
to
see the usage statement.
Use the "-filter" option to pass filtering options to the call to
tc_stat.
Here, I'm looking only at the HWFI and H3WI models.
Use "-plot" to specify which plot types should be created. I did all
possible types and have attached the results. You may be interested
in the
relative performance, in particular.
Use "-dep" to specify which columns of data you want to plot.
Hope that helps.
Thanks,
John
On Tue, Jan 6, 2015 at 3:58 PM, Ghassan Alaka - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70270 >
>
> John,
>
> That did the trick! Thank you so much. The syntax makes sense.
>
> Now, do you have any idea why the time series independence columns
read
> "NA"? Just curious about how to get those statistics. I'm hoping to
compute
> confidence intervals conservatively, so I'm looking for the number
of
> independent samples rather than the total number of time stamps.
>
> Thank you,
> Gus
>
> On Tue, Jan 6, 2015 at 4:24 PM, John Halley Gotway via RT <
> met_help at ucar.edu
> > wrote:
>
> > Gus,
> >
> > Great, thanks for doing that. If you were to rerun that same job
you ran
> > using a verbosity level of 4 (-v 4), you'd see the following line
printed
> > to the screen:
> > DEBUG 4: Skipping frequency of superior performance since the
case
> > information does not contain "AMODEL".
> >
> > The tc_stat job you ran looked at the 48 hour lead time for the
HWRF and
> > H3HW models but grouped together all the output. FSP is only
computed
> when
> > you have separate output for each model, and you can request that
using
> the
> > "-by AMODEL" option. I've attached the output created when
running the
> > following job on some sample data (note that I've used -by
AMODEL):
> >
> > met-5.0/bin/tc_stat \
> > -lookin d2014_vx_20141117_reset/al/tc_pairs/tc_pairs_H3WI_* \
> > -lookin d2014_vx_20141117_reset/al/tc_pairs/tc_pairs_HWFI_* \
> > -job summary -lead 480000 -column TRACK -amodel HWFI,H3WI -by
AMODEL
> > -out sample.out
> >
> > Try adding "-by AMODEL" and see if that does the trick. If not,
try
> using
> > debug level 4 (-v 4) to see if there's any more useful log
messages.
> >
> > Hope that helps.
> >
> > Thanks,
> > John
> >
> >
> >
> >
> > On Tue, Jan 6, 2015 at 2:12 PM, Ghassan Alaka - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70270 >
> > >
> > > Hi John,
> > >
> > > I filled out the registration form and submitted. Looking
forward to
> your
> > > assistance.
> > >
> > > Best,
> > > Gus
> > >
> > > On Tue, Jan 6, 2015 at 3:49 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu
> > > > wrote:
> > >
> > > > Hello Gus,
> > > >
> > > > I'd be happy to help you with this issue, but we can only
provide
> > direct
> > > > support to registered users of MET. We do this to ensure that
we
> > remain
> > > in
> > > > compliance with U.S. export control laws. You can fill out
the
> > > > registration form here:
> > > > http://www.dtcenter.org/met/users/downloads/index.php
> > > > Just enter your email address and hit enter. That should
take you
> to
> > > the
> > > > registration page.
> > > >
> > > > Please write me back once you've filled that out, and I should
be
> able
> > to
> > > > offer more help.
> > > >
> > > > Thanks,
> > > > John Halley Gotway
> > > > met_help at ucar.edu
> > > >
> > > > On Tue, Jan 6, 2015 at 10:57 AM, Ghassan Alaka - NOAA
Affiliate via
> RT
> > <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Tue Jan 06 10:57:18 2015: Request 70270 was acted upon.
> > > > > Transaction: Ticket created by ghassan.alaka at noaa.gov
> > > > > Queue: met_help
> > > > > Subject: Frequency of Superior Performance
> > > > > Owner: Nobody
> > > > > Requestors: ghassan.alaka at noaa.gov
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70270
> > > >
> > > > >
> > > > >
> > > > > Dear MET Help,
> > > > >
> > > > > I am trying to analyze the frequency of superior performance
(FSP)
> > > using
> > > > > Version 5 of MET-TC. I can successfully use "tc_pairs" and
> "tc_stat"
> > to
> > > > > compute error statistics for a set of A-Deck models.
However,
> > although
> > > > the
> > > > > summary output from "tc_stat" creates columns for FSP
analysis, it
> is
> > > "0"
> > > > > for all rows that contain non-zero valid entries.
Furthermore, the
> > time
> > > > > series independence analysis is "NA" for all rows that
contain
> > non-zero
> > > > > valid entries.
> > > > >
> > > > > How do I turn on the FSP analysis? What am I doing wrong? I
can not
> > > find
> > > > > options in the TCStatConfig file to change this. Please find
a
> > snapshot
> > > > of
> > > > > my "tc_stat" summary file. You can see the job options that
I used
> in
> > > the
> > > > > first line of this snapshot.
> > > > >
> > > > > Thank you for your help!
> > > > >
> > > > > Best,
> > > > > Gus Alaka
> > > > >
> > > > > [image: Inline image 1]
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list