[Met_help] [rt.rap.ucar.edu #70055] History for TC rapid intensification verification
John Halley Gotway via RT
met_help at ucar.edu
Tue Jan 20 09:34:27 MST 2015
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello. I thought I heard somewhere that MET can perform verifications of TC
rapid intensification (to produce output such as Brier score and ROC
diagrams). All I found so far is that MET can filter the out_tcmpr.tcst
file to only include times of RI/RW. Are RI verification metrics produced
by MET yet? Or is that planned for a future release?
Thanks,
Matt
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: TC rapid intensification verification
From: John Halley Gotway
Time: Wed Dec 10 11:54:21 2014
Matt,
met-5.0 only has the ability to filter out the ri/rw events, as you've
described. But we've developed and are testing internally a new
tc_stat
job type called "rirw" which identifies rirw events in both the ADECK
and
BDECK and uses them to populate a 2x2 contingency table and dump out
the
corresponding statistics. We've included a lot of flexibility in the
definition of rirw events in the either deck and how close the events
need
to be in time to be considered a "hit". I've listed below a
description of
this job type from the README_TC file in the development version of
the
code.
If you'd be willing/interested in testing out this capability and
providing
us feedback, we might be able to provide you a snapshot of the latest
development version.
Thanks,
John
// "rirw"
// To define rapid intensification/weakening contingency
table
using
// the ADECK and BDECK RI/RW settings and the matching time
window
// and output contingency table counts and statistics.
//
// Optional Args:
// -rapid_inten_window HHMMSS to override default of 0,
requiring exact match
// -rapid_inten_time or -rapid_inten_time_adeck and
-rapid_inten_time_bdeck to override defaults
// -rapid_inten_exact or -rapid_inten_exact_adeck and
-rapid_inten_exact_bdeck to override defaults
// -rapid_inten_thresh or -rapid_inten_thresh_adeck and
-rapid_inten_thresh_bdeck to override defaults
// -by column_name to specify case information
// -out_alpha to override default alpha value
// -out_line_type to specify which output types (CTC, CTS,
and
MPR) should be written
//
// Note that the "-dump_row path" option results in 4 files
being
created:
// path_FY_OY.tcst, path_FY_ON.tcst, path_FN_OY.tcst, and
path_FN_ON.tcst
// containing the TCST lines that were hits, false alarms,
misses, and correct negatives,
// respectively. These files may be used as input for
additional TC-Stat analysis.
On Wed, Dec 10, 2014 at 11:11 AM, Matthew Morin - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> Wed Dec 10 11:11:37 2014: Request 70055 was acted upon.
> Transaction: Ticket created by matthew.morin at noaa.gov
> Queue: met_help
> Subject: TC rapid intensification verification
> Owner: Nobody
> Requestors: matthew.morin at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055 >
>
>
> Hello. I thought I heard somewhere that MET can perform
verifications of TC
> rapid intensification (to produce output such as Brier score and ROC
> diagrams). All I found so far is that MET can filter the
out_tcmpr.tcst
> file to only include times of RI/RW. Are RI verification metrics
produced
> by MET yet? Or is that planned for a future release?
>
> Thanks,
> Matt
>
>
------------------------------------------------
Subject: TC rapid intensification verification
From: Matthew Morin - NOAA Affiliate
Time: Wed Dec 10 12:19:30 2014
Hi John,
Thanks for the information. Sure, I can be a MET RI/RW verification
tester.
I developed an RI verification system (not very elaborate) for the
GFDL
ensemble, so I am familiar with the 2x2 contingency table and
verification
metrics.
-Matt
On Wed, Dec 10, 2014 at 1:54 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Matt,
>
> met-5.0 only has the ability to filter out the ri/rw events, as
you've
> described. But we've developed and are testing internally a new
tc_stat
> job type called "rirw" which identifies rirw events in both the
ADECK and
> BDECK and uses them to populate a 2x2 contingency table and dump out
the
> corresponding statistics. We've included a lot of flexibility in
the
> definition of rirw events in the either deck and how close the
events need
> to be in time to be considered a "hit". I've listed below a
description of
> this job type from the README_TC file in the development version of
the
> code.
>
> If you'd be willing/interested in testing out this capability and
providing
> us feedback, we might be able to provide you a snapshot of the
latest
> development version.
>
> Thanks,
> John
>
> // "rirw"
>
> // To define rapid intensification/weakening contingency
table
> using
>
> // the ADECK and BDECK RI/RW settings and the matching time
window
>
> // and output contingency table counts and statistics.
>
> //
>
> // Optional Args:
>
> // -rapid_inten_window HHMMSS to override default of 0,
> requiring exact match
>
> // -rapid_inten_time or -rapid_inten_time_adeck and
> -rapid_inten_time_bdeck to override defaults
>
> // -rapid_inten_exact or -rapid_inten_exact_adeck and
> -rapid_inten_exact_bdeck to override defaults
>
> // -rapid_inten_thresh or -rapid_inten_thresh_adeck and
> -rapid_inten_thresh_bdeck to override defaults
>
> // -by column_name to specify case information
>
> // -out_alpha to override default alpha value
>
> // -out_line_type to specify which output types (CTC,
CTS, and
> MPR) should be written
>
> //
>
> // Note that the "-dump_row path" option results in 4 files
being
> created:
>
> // path_FY_OY.tcst, path_FY_ON.tcst, path_FN_OY.tcst,
and
> path_FN_ON.tcst
>
> // containing the TCST lines that were hits, false
alarms,
> misses, and correct negatives,
> // respectively. These files may be used as input for
> additional TC-Stat analysis.
>
> On Wed, Dec 10, 2014 at 11:11 AM, Matthew Morin - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Dec 10 11:11:37 2014: Request 70055 was acted upon.
> > Transaction: Ticket created by matthew.morin at noaa.gov
> > Queue: met_help
> > Subject: TC rapid intensification verification
> > Owner: Nobody
> > Requestors: matthew.morin at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055 >
> >
> >
> > Hello. I thought I heard somewhere that MET can perform
verifications of
> TC
> > rapid intensification (to produce output such as Brier score and
ROC
> > diagrams). All I found so far is that MET can filter the
out_tcmpr.tcst
> > file to only include times of RI/RW. Are RI verification metrics
produced
> > by MET yet? Or is that planned for a future release?
> >
> > Thanks,
> > Matt
>
------------------------------------------------
Subject: TC rapid intensification verification
From: John Halley Gotway
Time: Wed Dec 10 13:20:56 2014
Matt,
I see that you were running MET on zeus in the past. Is that still
the
case? If so, I could just build the development version there.
Would that suffice?
Thanks,
John
On Wed, Dec 10, 2014 at 12:19 PM, Matthew Morin - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055 >
>
> Hi John,
>
> Thanks for the information. Sure, I can be a MET RI/RW verification
tester.
> I developed an RI verification system (not very elaborate) for the
GFDL
> ensemble, so I am familiar with the 2x2 contingency table and
verification
> metrics.
>
> -Matt
>
> On Wed, Dec 10, 2014 at 1:54 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Matt,
> >
> > met-5.0 only has the ability to filter out the ri/rw events, as
you've
> > described. But we've developed and are testing internally a new
tc_stat
> > job type called "rirw" which identifies rirw events in both the
ADECK and
> > BDECK and uses them to populate a 2x2 contingency table and dump
out the
> > corresponding statistics. We've included a lot of flexibility in
the
> > definition of rirw events in the either deck and how close the
events
> need
> > to be in time to be considered a "hit". I've listed below a
description
> of
> > this job type from the README_TC file in the development version
of the
> > code.
> >
> > If you'd be willing/interested in testing out this capability and
> providing
> > us feedback, we might be able to provide you a snapshot of the
latest
> > development version.
> >
> > Thanks,
> > John
> >
> > // "rirw"
> >
> > // To define rapid intensification/weakening contingency
table
> > using
> >
> > // the ADECK and BDECK RI/RW settings and the matching
time
> window
> >
> > // and output contingency table counts and statistics.
> >
> > //
> >
> > // Optional Args:
> >
> > // -rapid_inten_window HHMMSS to override default of
0,
> > requiring exact match
> >
> > // -rapid_inten_time or -rapid_inten_time_adeck and
> > -rapid_inten_time_bdeck to override defaults
> >
> > // -rapid_inten_exact or -rapid_inten_exact_adeck and
> > -rapid_inten_exact_bdeck to override defaults
> >
> > // -rapid_inten_thresh or -rapid_inten_thresh_adeck
and
> > -rapid_inten_thresh_bdeck to override defaults
> >
> > // -by column_name to specify case information
> >
> > // -out_alpha to override default alpha value
> >
> > // -out_line_type to specify which output types (CTC,
CTS,
> and
> > MPR) should be written
> >
> > //
> >
> > // Note that the "-dump_row path" option results in 4
files
> being
> > created:
> >
> > // path_FY_OY.tcst, path_FY_ON.tcst, path_FN_OY.tcst,
and
> > path_FN_ON.tcst
> >
> > // containing the TCST lines that were hits, false
alarms,
> > misses, and correct negatives,
> > // respectively. These files may be used as input for
> > additional TC-Stat analysis.
> >
> > On Wed, Dec 10, 2014 at 11:11 AM, Matthew Morin - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Dec 10 11:11:37 2014: Request 70055 was acted upon.
> > > Transaction: Ticket created by matthew.morin at noaa.gov
> > > Queue: met_help
> > > Subject: TC rapid intensification verification
> > > Owner: Nobody
> > > Requestors: matthew.morin at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055
> >
> > >
> > >
> > > Hello. I thought I heard somewhere that MET can perform
verifications
> of
> > TC
> > > rapid intensification (to produce output such as Brier score and
ROC
> > > diagrams). All I found so far is that MET can filter the
out_tcmpr.tcst
> > > file to only include times of RI/RW. Are RI verification metrics
> produced
> > > by MET yet? Or is that planned for a future release?
> > >
> > > Thanks,
> > > Matt
> >
>
>
------------------------------------------------
Subject: TC rapid intensification verification
From: Matthew Morin - NOAA Affiliate
Time: Wed Dec 10 13:33:48 2014
John,
Yes, I'm running MET on Zeus now. If you could build it there, that
would
be great.
Thanks,
-Matt
On Wed, Dec 10, 2014 at 3:20 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Matt,
>
> I see that you were running MET on zeus in the past. Is that still
the
> case? If so, I could just build the development version there.
>
> Would that suffice?
>
> Thanks,
> John
>
> On Wed, Dec 10, 2014 at 12:19 PM, Matthew Morin - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055 >
> >
> > Hi John,
> >
> > Thanks for the information. Sure, I can be a MET RI/RW
verification
> tester.
> > I developed an RI verification system (not very elaborate) for the
GFDL
> > ensemble, so I am familiar with the 2x2 contingency table and
> verification
> > metrics.
> >
> > -Matt
> >
> > On Wed, Dec 10, 2014 at 1:54 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Matt,
> > >
> > > met-5.0 only has the ability to filter out the ri/rw events, as
you've
> > > described. But we've developed and are testing internally a new
> tc_stat
> > > job type called "rirw" which identifies rirw events in both the
ADECK
> and
> > > BDECK and uses them to populate a 2x2 contingency table and dump
out
> the
> > > corresponding statistics. We've included a lot of flexibility
in the
> > > definition of rirw events in the either deck and how close the
events
> > need
> > > to be in time to be considered a "hit". I've listed below a
> description
> > of
> > > this job type from the README_TC file in the development version
of the
> > > code.
> > >
> > > If you'd be willing/interested in testing out this capability
and
> > providing
> > > us feedback, we might be able to provide you a snapshot of the
latest
> > > development version.
> > >
> > > Thanks,
> > > John
> > >
> > > // "rirw"
> > >
> > > // To define rapid intensification/weakening
contingency table
> > > using
> > >
> > > // the ADECK and BDECK RI/RW settings and the matching
time
> > window
> > >
> > > // and output contingency table counts and statistics.
> > >
> > > //
> > >
> > > // Optional Args:
> > >
> > > // -rapid_inten_window HHMMSS to override default of
0,
> > > requiring exact match
> > >
> > > // -rapid_inten_time or -rapid_inten_time_adeck and
> > > -rapid_inten_time_bdeck to override defaults
> > >
> > > // -rapid_inten_exact or -rapid_inten_exact_adeck
and
> > > -rapid_inten_exact_bdeck to override defaults
> > >
> > > // -rapid_inten_thresh or -rapid_inten_thresh_adeck
and
> > > -rapid_inten_thresh_bdeck to override defaults
> > >
> > > // -by column_name to specify case information
> > >
> > > // -out_alpha to override default alpha value
> > >
> > > // -out_line_type to specify which output types
(CTC, CTS,
> > and
> > > MPR) should be written
> > >
> > > //
> > >
> > > // Note that the "-dump_row path" option results in 4
files
> > being
> > > created:
> > >
> > > // path_FY_OY.tcst, path_FY_ON.tcst,
path_FN_OY.tcst, and
> > > path_FN_ON.tcst
> > >
> > > // containing the TCST lines that were hits, false
alarms,
> > > misses, and correct negatives,
> > > // respectively. These files may be used as input
for
> > > additional TC-Stat analysis.
> > >
> > > On Wed, Dec 10, 2014 at 11:11 AM, Matthew Morin - NOAA Affiliate
via
> RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Wed Dec 10 11:11:37 2014: Request 70055 was acted upon.
> > > > Transaction: Ticket created by matthew.morin at noaa.gov
> > > > Queue: met_help
> > > > Subject: TC rapid intensification verification
> > > > Owner: Nobody
> > > > Requestors: matthew.morin at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055
> > >
> > > >
> > > >
> > > > Hello. I thought I heard somewhere that MET can perform
verifications
> > of
> > > TC
> > > > rapid intensification (to produce output such as Brier score
and ROC
> > > > diagrams). All I found so far is that MET can filter the
> out_tcmpr.tcst
> > > > file to only include times of RI/RW. Are RI verification
metrics
> > produced
> > > > by MET yet? Or is that planned for a future release?
> > > >
> > > > Thanks,
> > > > Matt
>
------------------------------------------------
Subject: TC rapid intensification verification
From: John Halley Gotway
Time: Wed Dec 10 14:13:46 2014
Done. You can find it in:
/scratch1/portfolios/BMC/dtc/MET/met-rev2577
2577 is just the revision number of the MET subversion repository.
To get the most output, you'd run something like this...
/scratch1/portfolios/BMC/dtc/MET/met-rev2577/bin/tc_stat \
-lookin path/to/tc_pairs/output \
-job rirw -dump_row test \
-out_line_type CTC,CTS,MPR
Please search for "rirw" in here:
/scratch1/portfolios/BMC/dtc/MET/met-rev2577/data/config/README_TC
For a brief description of the job options.
By default, rapid intensification is defined as a 24-hour exact change
exceeding 30kts. You can modify that definition using for the ADECK,
BDECK, or both using -rapid_inten_time, -rapid_inten_exact, and
-rapid_inten_thresh options. Set -rapid_inten_window to something
larger
than 0 to enable false alarms to be considered hits when they were
"close
enough" in time.
Evaluate rapid weakening by setting "-rapid_inten_thresh <=-30".
Once you have the job working as you'd like, you can easily run the
same
job for different lead time using the "-by LEAD" option.
Please let me know how it all goes and if you have questions.
Thanks,
John
On Wed, Dec 10, 2014 at 1:33 PM, Matthew Morin - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055 >
>
> John,
> Yes, I'm running MET on Zeus now. If you could build it there, that
would
> be great.
> Thanks,
> -Matt
>
> On Wed, Dec 10, 2014 at 3:20 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Matt,
> >
> > I see that you were running MET on zeus in the past. Is that
still the
> > case? If so, I could just build the development version there.
> >
> > Would that suffice?
> >
> > Thanks,
> > John
> >
> > On Wed, Dec 10, 2014 at 12:19 PM, Matthew Morin - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055 >
> > >
> > > Hi John,
> > >
> > > Thanks for the information. Sure, I can be a MET RI/RW
verification
> > tester.
> > > I developed an RI verification system (not very elaborate) for
the GFDL
> > > ensemble, so I am familiar with the 2x2 contingency table and
> > verification
> > > metrics.
> > >
> > > -Matt
> > >
> > > On Wed, Dec 10, 2014 at 1:54 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Matt,
> > > >
> > > > met-5.0 only has the ability to filter out the ri/rw events,
as
> you've
> > > > described. But we've developed and are testing internally a
new
> > tc_stat
> > > > job type called "rirw" which identifies rirw events in both
the ADECK
> > and
> > > > BDECK and uses them to populate a 2x2 contingency table and
dump out
> > the
> > > > corresponding statistics. We've included a lot of flexibility
in the
> > > > definition of rirw events in the either deck and how close the
events
> > > need
> > > > to be in time to be considered a "hit". I've listed below a
> > description
> > > of
> > > > this job type from the README_TC file in the development
version of
> the
> > > > code.
> > > >
> > > > If you'd be willing/interested in testing out this capability
and
> > > providing
> > > > us feedback, we might be able to provide you a snapshot of the
latest
> > > > development version.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > // "rirw"
> > > >
> > > > // To define rapid intensification/weakening
contingency
> table
> > > > using
> > > >
> > > > // the ADECK and BDECK RI/RW settings and the
matching time
> > > window
> > > >
> > > > // and output contingency table counts and
statistics.
> > > >
> > > > //
> > > >
> > > > // Optional Args:
> > > >
> > > > // -rapid_inten_window HHMMSS to override default
of 0,
> > > > requiring exact match
> > > >
> > > > // -rapid_inten_time or -rapid_inten_time_adeck
and
> > > > -rapid_inten_time_bdeck to override defaults
> > > >
> > > > // -rapid_inten_exact or -rapid_inten_exact_adeck
and
> > > > -rapid_inten_exact_bdeck to override defaults
> > > >
> > > > // -rapid_inten_thresh or
-rapid_inten_thresh_adeck and
> > > > -rapid_inten_thresh_bdeck to override defaults
> > > >
> > > > // -by column_name to specify case information
> > > >
> > > > // -out_alpha to override default alpha value
> > > >
> > > > // -out_line_type to specify which output types
(CTC,
> CTS,
> > > and
> > > > MPR) should be written
> > > >
> > > > //
> > > >
> > > > // Note that the "-dump_row path" option results in 4
files
> > > being
> > > > created:
> > > >
> > > > // path_FY_OY.tcst, path_FY_ON.tcst,
path_FN_OY.tcst, and
> > > > path_FN_ON.tcst
> > > >
> > > > // containing the TCST lines that were hits, false
> alarms,
> > > > misses, and correct negatives,
> > > > // respectively. These files may be used as input
for
> > > > additional TC-Stat analysis.
> > > >
> > > > On Wed, Dec 10, 2014 at 11:11 AM, Matthew Morin - NOAA
Affiliate via
> > RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Wed Dec 10 11:11:37 2014: Request 70055 was acted upon.
> > > > > Transaction: Ticket created by matthew.morin at noaa.gov
> > > > > Queue: met_help
> > > > > Subject: TC rapid intensification verification
> > > > > Owner: Nobody
> > > > > Requestors: matthew.morin at noaa.gov
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055
> > > >
> > > > >
> > > > >
> > > > > Hello. I thought I heard somewhere that MET can perform
> verifications
> > > of
> > > > TC
> > > > > rapid intensification (to produce output such as Brier score
and
> ROC
> > > > > diagrams). All I found so far is that MET can filter the
> > out_tcmpr.tcst
> > > > > file to only include times of RI/RW. Are RI verification
metrics
> > > produced
> > > > > by MET yet? Or is that planned for a future release?
> > > > >
> > > > > Thanks,
> > > > > Matt
> >
>
>
------------------------------------------------
Subject: TC rapid intensification verification
From: Matthew Morin - NOAA Affiliate
Time: Wed Dec 10 14:26:54 2014
John,
Thanks very much. I just quickly ran the command you listed and I got
good
output. I will learn the format of (and sort through) the output
tomorrow.
Thanks again,
Matt
On Wed, Dec 10, 2014 at 4:13 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Done. You can find it in:
> /scratch1/portfolios/BMC/dtc/MET/met-rev2577
>
> 2577 is just the revision number of the MET subversion repository.
>
> To get the most output, you'd run something like this...
> /scratch1/portfolios/BMC/dtc/MET/met-rev2577/bin/tc_stat \
> -lookin path/to/tc_pairs/output \
> -job rirw -dump_row test \
> -out_line_type CTC,CTS,MPR
>
> Please search for "rirw" in here:
> /scratch1/portfolios/BMC/dtc/MET/met-
rev2577/data/config/README_TC
> For a brief description of the job options.
>
> By default, rapid intensification is defined as a 24-hour exact
change
> exceeding 30kts. You can modify that definition using for the
ADECK,
> BDECK, or both using -rapid_inten_time, -rapid_inten_exact, and
> -rapid_inten_thresh options. Set -rapid_inten_window to something
larger
> than 0 to enable false alarms to be considered hits when they were
"close
> enough" in time.
>
> Evaluate rapid weakening by setting "-rapid_inten_thresh <=-30".
>
> Once you have the job working as you'd like, you can easily run the
same
> job for different lead time using the "-by LEAD" option.
>
> Please let me know how it all goes and if you have questions.
>
> Thanks,
> John
>
> On Wed, Dec 10, 2014 at 1:33 PM, Matthew Morin - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055 >
> >
> > John,
> > Yes, I'm running MET on Zeus now. If you could build it there,
that would
> > be great.
> > Thanks,
> > -Matt
> >
> > On Wed, Dec 10, 2014 at 3:20 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Matt,
> > >
> > > I see that you were running MET on zeus in the past. Is that
still the
> > > case? If so, I could just build the development version there.
> > >
> > > Would that suffice?
> > >
> > > Thanks,
> > > John
> > >
> > > On Wed, Dec 10, 2014 at 12:19 PM, Matthew Morin - NOAA Affiliate
via
> RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055
>
> > > >
> > > > Hi John,
> > > >
> > > > Thanks for the information. Sure, I can be a MET RI/RW
verification
> > > tester.
> > > > I developed an RI verification system (not very elaborate) for
the
> GFDL
> > > > ensemble, so I am familiar with the 2x2 contingency table and
> > > verification
> > > > metrics.
> > > >
> > > > -Matt
> > > >
> > > > On Wed, Dec 10, 2014 at 1:54 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Matt,
> > > > >
> > > > > met-5.0 only has the ability to filter out the ri/rw events,
as
> > you've
> > > > > described. But we've developed and are testing internally a
new
> > > tc_stat
> > > > > job type called "rirw" which identifies rirw events in both
the
> ADECK
> > > and
> > > > > BDECK and uses them to populate a 2x2 contingency table and
dump
> out
> > > the
> > > > > corresponding statistics. We've included a lot of
flexibility in
> the
> > > > > definition of rirw events in the either deck and how close
the
> events
> > > > need
> > > > > to be in time to be considered a "hit". I've listed below a
> > > description
> > > > of
> > > > > this job type from the README_TC file in the development
version of
> > the
> > > > > code.
> > > > >
> > > > > If you'd be willing/interested in testing out this
capability and
> > > > providing
> > > > > us feedback, we might be able to provide you a snapshot of
the
> latest
> > > > > development version.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > // "rirw"
> > > > >
> > > > > // To define rapid intensification/weakening
contingency
> > table
> > > > > using
> > > > >
> > > > > // the ADECK and BDECK RI/RW settings and the
matching
> time
> > > > window
> > > > >
> > > > > // and output contingency table counts and
statistics.
> > > > >
> > > > > //
> > > > >
> > > > > // Optional Args:
> > > > >
> > > > > // -rapid_inten_window HHMMSS to override
default of 0,
> > > > > requiring exact match
> > > > >
> > > > > // -rapid_inten_time or -rapid_inten_time_adeck
and
> > > > > -rapid_inten_time_bdeck to override defaults
> > > > >
> > > > > // -rapid_inten_exact or
-rapid_inten_exact_adeck and
> > > > > -rapid_inten_exact_bdeck to override defaults
> > > > >
> > > > > // -rapid_inten_thresh or
-rapid_inten_thresh_adeck and
> > > > > -rapid_inten_thresh_bdeck to override defaults
> > > > >
> > > > > // -by column_name to specify case information
> > > > >
> > > > > // -out_alpha to override default alpha value
> > > > >
> > > > > // -out_line_type to specify which output types
(CTC,
> > CTS,
> > > > and
> > > > > MPR) should be written
> > > > >
> > > > > //
> > > > >
> > > > > // Note that the "-dump_row path" option results in
4
> files
> > > > being
> > > > > created:
> > > > >
> > > > > // path_FY_OY.tcst, path_FY_ON.tcst,
path_FN_OY.tcst,
> and
> > > > > path_FN_ON.tcst
> > > > >
> > > > > // containing the TCST lines that were hits,
false
> > alarms,
> > > > > misses, and correct negatives,
> > > > > // respectively. These files may be used as
input for
> > > > > additional TC-Stat analysis.
> > > > >
> > > > > On Wed, Dec 10, 2014 at 11:11 AM, Matthew Morin - NOAA
Affiliate
> via
> > > RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Wed Dec 10 11:11:37 2014: Request 70055 was acted upon.
> > > > > > Transaction: Ticket created by matthew.morin at noaa.gov
> > > > > > Queue: met_help
> > > > > > Subject: TC rapid intensification verification
> > > > > > Owner: Nobody
> > > > > > Requestors: matthew.morin at noaa.gov
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055
> > > > >
> > > > > >
> > > > > >
> > > > > > Hello. I thought I heard somewhere that MET can perform
> > verifications
> > > > of
> > > > > TC
> > > > > > rapid intensification (to produce output such as Brier
score and
> > ROC
> > > > > > diagrams). All I found so far is that MET can filter the
> > > out_tcmpr.tcst
> > > > > > file to only include times of RI/RW. Are RI verification
metrics
> > > > produced
> > > > > > by MET yet? Or is that planned for a future release?
> > > > > >
> > > > > > Thanks,
> > > > > > Matt
>
------------------------------------------------
Subject: TC rapid intensification verification
From: Matthew Morin - NOAA Affiliate
Time: Fri Dec 12 11:43:29 2014
Hi John,
I just finished using this developmental version of MET to calculate
probability of detection and false alarm ratio for my hurricane
ensemble.
Attached are the plots.
Also attached is a small write-up of what I observed and suggest.
Was there any other feedback or tests you want from me?
Thanks,
Matt
On Wed, Dec 10, 2014 at 4:26 PM, Matthew Morin - NOAA Affiliate <
matthew.morin at noaa.gov> wrote:
> John,
>
> Thanks very much. I just quickly ran the command you listed and I
got good
> output. I will learn the format of (and sort through) the output
tomorrow.
>
> Thanks again,
> Matt
>
> On Wed, Dec 10, 2014 at 4:13 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Done. You can find it in:
>> /scratch1/portfolios/BMC/dtc/MET/met-rev2577
>>
>> 2577 is just the revision number of the MET subversion repository.
>>
>> To get the most output, you'd run something like this...
>> /scratch1/portfolios/BMC/dtc/MET/met-rev2577/bin/tc_stat \
>> -lookin path/to/tc_pairs/output \
>> -job rirw -dump_row test \
>> -out_line_type CTC,CTS,MPR
>>
>> Please search for "rirw" in here:
>> /scratch1/portfolios/BMC/dtc/MET/met-
rev2577/data/config/README_TC
>> For a brief description of the job options.
>>
>> By default, rapid intensification is defined as a 24-hour exact
change
>> exceeding 30kts. You can modify that definition using for the
ADECK,
>> BDECK, or both using -rapid_inten_time, -rapid_inten_exact, and
>> -rapid_inten_thresh options. Set -rapid_inten_window to something
larger
>> than 0 to enable false alarms to be considered hits when they were
"close
>> enough" in time.
>>
>> Evaluate rapid weakening by setting "-rapid_inten_thresh <=-30".
>>
>> Once you have the job working as you'd like, you can easily run the
same
>> job for different lead time using the "-by LEAD" option.
>>
>> Please let me know how it all goes and if you have questions.
>>
>> Thanks,
>> John
>>
>> On Wed, Dec 10, 2014 at 1:33 PM, Matthew Morin - NOAA Affiliate via
RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055 >
>> >
>> > John,
>> > Yes, I'm running MET on Zeus now. If you could build it there,
that
>> would
>> > be great.
>> > Thanks,
>> > -Matt
>> >
>> > On Wed, Dec 10, 2014 at 3:20 PM, John Halley Gotway via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > > Matt,
>> > >
>> > > I see that you were running MET on zeus in the past. Is that
still
>> the
>> > > case? If so, I could just build the development version there.
>> > >
>> > > Would that suffice?
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > > On Wed, Dec 10, 2014 at 12:19 PM, Matthew Morin - NOAA
Affiliate via
>> RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055
>
>> > > >
>> > > > Hi John,
>> > > >
>> > > > Thanks for the information. Sure, I can be a MET RI/RW
verification
>> > > tester.
>> > > > I developed an RI verification system (not very elaborate)
for the
>> GFDL
>> > > > ensemble, so I am familiar with the 2x2 contingency table and
>> > > verification
>> > > > metrics.
>> > > >
>> > > > -Matt
>> > > >
>> > > > On Wed, Dec 10, 2014 at 1:54 PM, John Halley Gotway via RT <
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > > > Matt,
>> > > > >
>> > > > > met-5.0 only has the ability to filter out the ri/rw
events, as
>> > you've
>> > > > > described. But we've developed and are testing internally
a new
>> > > tc_stat
>> > > > > job type called "rirw" which identifies rirw events in both
the
>> ADECK
>> > > and
>> > > > > BDECK and uses them to populate a 2x2 contingency table and
dump
>> out
>> > > the
>> > > > > corresponding statistics. We've included a lot of
flexibility in
>> the
>> > > > > definition of rirw events in the either deck and how close
the
>> events
>> > > > need
>> > > > > to be in time to be considered a "hit". I've listed below
a
>> > > description
>> > > > of
>> > > > > this job type from the README_TC file in the development
version
>> of
>> > the
>> > > > > code.
>> > > > >
>> > > > > If you'd be willing/interested in testing out this
capability and
>> > > > providing
>> > > > > us feedback, we might be able to provide you a snapshot of
the
>> latest
>> > > > > development version.
>> > > > >
>> > > > > Thanks,
>> > > > > John
>> > > > >
>> > > > > // "rirw"
>> > > > >
>> > > > > // To define rapid intensification/weakening
contingency
>> > table
>> > > > > using
>> > > > >
>> > > > > // the ADECK and BDECK RI/RW settings and the
matching
>> time
>> > > > window
>> > > > >
>> > > > > // and output contingency table counts and
statistics.
>> > > > >
>> > > > > //
>> > > > >
>> > > > > // Optional Args:
>> > > > >
>> > > > > // -rapid_inten_window HHMMSS to override
default of
>> 0,
>> > > > > requiring exact match
>> > > > >
>> > > > > // -rapid_inten_time or -rapid_inten_time_adeck
and
>> > > > > -rapid_inten_time_bdeck to override defaults
>> > > > >
>> > > > > // -rapid_inten_exact or
-rapid_inten_exact_adeck and
>> > > > > -rapid_inten_exact_bdeck to override defaults
>> > > > >
>> > > > > // -rapid_inten_thresh or
-rapid_inten_thresh_adeck
>> and
>> > > > > -rapid_inten_thresh_bdeck to override defaults
>> > > > >
>> > > > > // -by column_name to specify case information
>> > > > >
>> > > > > // -out_alpha to override default alpha value
>> > > > >
>> > > > > // -out_line_type to specify which output types
(CTC,
>> > CTS,
>> > > > and
>> > > > > MPR) should be written
>> > > > >
>> > > > > //
>> > > > >
>> > > > > // Note that the "-dump_row path" option results
in 4
>> files
>> > > > being
>> > > > > created:
>> > > > >
>> > > > > // path_FY_OY.tcst, path_FY_ON.tcst,
path_FN_OY.tcst,
>> and
>> > > > > path_FN_ON.tcst
>> > > > >
>> > > > > // containing the TCST lines that were hits,
false
>> > alarms,
>> > > > > misses, and correct negatives,
>> > > > > // respectively. These files may be used as
input for
>> > > > > additional TC-Stat analysis.
>> > > > >
>> > > > > On Wed, Dec 10, 2014 at 11:11 AM, Matthew Morin - NOAA
Affiliate
>> via
>> > > RT <
>> > > > > met_help at ucar.edu> wrote:
>> > > > >
>> > > > > >
>> > > > > > Wed Dec 10 11:11:37 2014: Request 70055 was acted upon.
>> > > > > > Transaction: Ticket created by matthew.morin at noaa.gov
>> > > > > > Queue: met_help
>> > > > > > Subject: TC rapid intensification verification
>> > > > > > Owner: Nobody
>> > > > > > Requestors: matthew.morin at noaa.gov
>> > > > > > Status: new
>> > > > > > Ticket <URL:
>> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055
>> > > > >
>> > > > > >
>> > > > > >
>> > > > > > Hello. I thought I heard somewhere that MET can perform
>> > verifications
>> > > > of
>> > > > > TC
>> > > > > > rapid intensification (to produce output such as Brier
score and
>> > ROC
>> > > > > > diagrams). All I found so far is that MET can filter the
>> > > out_tcmpr.tcst
>> > > > > > file to only include times of RI/RW. Are RI verification
metrics
>> > > > produced
>> > > > > > by MET yet? Or is that planned for a future release?
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Matt
>>
>
--
-------------------------------------------------------
Matthew Morin
Engility
High Technology Services Group
GFDL
Princeton University Forrestal Campus
201 Forrestal Road
Princeton, NJ 08540
phone: 609-452-5381
email: Matthew.Morin at noaa.gov
-------------------------------------------------------
------------------------------------------------
Subject: TC rapid intensification verification
From: John Halley Gotway
Time: Fri Dec 12 13:02:41 2014
Matt,
Great, thanks for sending me that feedback.
The *_F?_O?.tcst files are just subsets of the input .tcst lines.
They are
the lines falling into those categories. As such, I'm reticent to
make any
changes to the format of those lines. Dividing the LEAD hours by
10,000
would work well for most data, but not sub-hourly data, which
admittedly is
uncommon. The format of that column is HHMMSS, where HH expands to
the
number of digits necessary to store the hours. The "VALID" column
contains
the timestamp for the valid time of that track point.
I'm aware of the columns being misaligned. This tools reads track
data
from multiple input files in which the inter-column spacing will vary
based
on the length of the strings in those files. When the tool reads each
input line, it decides whether or not it should be written to the
output
file, instead of buffering the data in memory. This makes it run
faster.
Presumably, we could fix this to align all the output columns, but it
hasn't seemed worth the extra effort yet.
Regarding breaking results down by model, that actually is pretty easy
to
do. Just rerun your job, but add in the "-by AMODEL" command line
option.
That will run the exact same job for each unique value found in the
AMODEL
column. You can also string multiple ones together. For example, "-
by
BASIN -by AMODEL -by LEAD" will compute stats for each unique
combination
of basin, model name, and lead time encountered.
That's an interesting suggestion about verifying ensemble
probabilities of
RI/RW. I'll bring that up with some other folks here.
Thanks,
John
On Fri, Dec 12, 2014 at 11:43 AM, Matthew Morin - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055 >
>
> Hi John,
>
> I just finished using this developmental version of MET to calculate
> probability of detection and false alarm ratio for my hurricane
ensemble.
> Attached are the plots.
>
> Also attached is a small write-up of what I observed and suggest.
>
> Was there any other feedback or tests you want from me?
>
> Thanks,
> Matt
>
> On Wed, Dec 10, 2014 at 4:26 PM, Matthew Morin - NOAA Affiliate <
> matthew.morin at noaa.gov> wrote:
>
> > John,
> >
> > Thanks very much. I just quickly ran the command you listed and I
got
> good
> > output. I will learn the format of (and sort through) the output
> tomorrow.
> >
> > Thanks again,
> > Matt
> >
> > On Wed, Dec 10, 2014 at 4:13 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Done. You can find it in:
> >> /scratch1/portfolios/BMC/dtc/MET/met-rev2577
> >>
> >> 2577 is just the revision number of the MET subversion
repository.
> >>
> >> To get the most output, you'd run something like this...
> >> /scratch1/portfolios/BMC/dtc/MET/met-rev2577/bin/tc_stat \
> >> -lookin path/to/tc_pairs/output \
> >> -job rirw -dump_row test \
> >> -out_line_type CTC,CTS,MPR
> >>
> >> Please search for "rirw" in here:
> >> /scratch1/portfolios/BMC/dtc/MET/met-
rev2577/data/config/README_TC
> >> For a brief description of the job options.
> >>
> >> By default, rapid intensification is defined as a 24-hour exact
change
> >> exceeding 30kts. You can modify that definition using for the
ADECK,
> >> BDECK, or both using -rapid_inten_time, -rapid_inten_exact, and
> >> -rapid_inten_thresh options. Set -rapid_inten_window to
something
> larger
> >> than 0 to enable false alarms to be considered hits when they
were
> "close
> >> enough" in time.
> >>
> >> Evaluate rapid weakening by setting "-rapid_inten_thresh <=-30".
> >>
> >> Once you have the job working as you'd like, you can easily run
the same
> >> job for different lead time using the "-by LEAD" option.
> >>
> >> Please let me know how it all goes and if you have questions.
> >>
> >> Thanks,
> >> John
> >>
> >> On Wed, Dec 10, 2014 at 1:33 PM, Matthew Morin - NOAA Affiliate
via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055 >
> >> >
> >> > John,
> >> > Yes, I'm running MET on Zeus now. If you could build it there,
that
> >> would
> >> > be great.
> >> > Thanks,
> >> > -Matt
> >> >
> >> > On Wed, Dec 10, 2014 at 3:20 PM, John Halley Gotway via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > > Matt,
> >> > >
> >> > > I see that you were running MET on zeus in the past. Is that
still
> >> the
> >> > > case? If so, I could just build the development version
there.
> >> > >
> >> > > Would that suffice?
> >> > >
> >> > > Thanks,
> >> > > John
> >> > >
> >> > > On Wed, Dec 10, 2014 at 12:19 PM, Matthew Morin - NOAA
Affiliate via
> >> RT <
> >> > > met_help at ucar.edu> wrote:
> >> > >
> >> > > >
> >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055 >
> >> > > >
> >> > > > Hi John,
> >> > > >
> >> > > > Thanks for the information. Sure, I can be a MET RI/RW
> verification
> >> > > tester.
> >> > > > I developed an RI verification system (not very elaborate)
for the
> >> GFDL
> >> > > > ensemble, so I am familiar with the 2x2 contingency table
and
> >> > > verification
> >> > > > metrics.
> >> > > >
> >> > > > -Matt
> >> > > >
> >> > > > On Wed, Dec 10, 2014 at 1:54 PM, John Halley Gotway via RT
<
> >> > > > met_help at ucar.edu> wrote:
> >> > > >
> >> > > > > Matt,
> >> > > > >
> >> > > > > met-5.0 only has the ability to filter out the ri/rw
events, as
> >> > you've
> >> > > > > described. But we've developed and are testing
internally a new
> >> > > tc_stat
> >> > > > > job type called "rirw" which identifies rirw events in
both the
> >> ADECK
> >> > > and
> >> > > > > BDECK and uses them to populate a 2x2 contingency table
and dump
> >> out
> >> > > the
> >> > > > > corresponding statistics. We've included a lot of
flexibility
> in
> >> the
> >> > > > > definition of rirw events in the either deck and how
close the
> >> events
> >> > > > need
> >> > > > > to be in time to be considered a "hit". I've listed
below a
> >> > > description
> >> > > > of
> >> > > > > this job type from the README_TC file in the development
version
> >> of
> >> > the
> >> > > > > code.
> >> > > > >
> >> > > > > If you'd be willing/interested in testing out this
capability
> and
> >> > > > providing
> >> > > > > us feedback, we might be able to provide you a snapshot
of the
> >> latest
> >> > > > > development version.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > John
> >> > > > >
> >> > > > > // "rirw"
> >> > > > >
> >> > > > > // To define rapid intensification/weakening
> contingency
> >> > table
> >> > > > > using
> >> > > > >
> >> > > > > // the ADECK and BDECK RI/RW settings and the
matching
> >> time
> >> > > > window
> >> > > > >
> >> > > > > // and output contingency table counts and
statistics.
> >> > > > >
> >> > > > > //
> >> > > > >
> >> > > > > // Optional Args:
> >> > > > >
> >> > > > > // -rapid_inten_window HHMMSS to override
default of
> >> 0,
> >> > > > > requiring exact match
> >> > > > >
> >> > > > > // -rapid_inten_time or
-rapid_inten_time_adeck and
> >> > > > > -rapid_inten_time_bdeck to override defaults
> >> > > > >
> >> > > > > // -rapid_inten_exact or
-rapid_inten_exact_adeck
> and
> >> > > > > -rapid_inten_exact_bdeck to override defaults
> >> > > > >
> >> > > > > // -rapid_inten_thresh or
-rapid_inten_thresh_adeck
> >> and
> >> > > > > -rapid_inten_thresh_bdeck to override defaults
> >> > > > >
> >> > > > > // -by column_name to specify case
information
> >> > > > >
> >> > > > > // -out_alpha to override default alpha value
> >> > > > >
> >> > > > > // -out_line_type to specify which output
types
> (CTC,
> >> > CTS,
> >> > > > and
> >> > > > > MPR) should be written
> >> > > > >
> >> > > > > //
> >> > > > >
> >> > > > > // Note that the "-dump_row path" option results
in 4
> >> files
> >> > > > being
> >> > > > > created:
> >> > > > >
> >> > > > > // path_FY_OY.tcst, path_FY_ON.tcst,
> path_FN_OY.tcst,
> >> and
> >> > > > > path_FN_ON.tcst
> >> > > > >
> >> > > > > // containing the TCST lines that were hits,
false
> >> > alarms,
> >> > > > > misses, and correct negatives,
> >> > > > > // respectively. These files may be used as
input
> for
> >> > > > > additional TC-Stat analysis.
> >> > > > >
> >> > > > > On Wed, Dec 10, 2014 at 11:11 AM, Matthew Morin - NOAA
Affiliate
> >> via
> >> > > RT <
> >> > > > > met_help at ucar.edu> wrote:
> >> > > > >
> >> > > > > >
> >> > > > > > Wed Dec 10 11:11:37 2014: Request 70055 was acted upon.
> >> > > > > > Transaction: Ticket created by matthew.morin at noaa.gov
> >> > > > > > Queue: met_help
> >> > > > > > Subject: TC rapid intensification verification
> >> > > > > > Owner: Nobody
> >> > > > > > Requestors: matthew.morin at noaa.gov
> >> > > > > > Status: new
> >> > > > > > Ticket <URL:
> >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055
> >> > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > Hello. I thought I heard somewhere that MET can perform
> >> > verifications
> >> > > > of
> >> > > > > TC
> >> > > > > > rapid intensification (to produce output such as Brier
score
> and
> >> > ROC
> >> > > > > > diagrams). All I found so far is that MET can filter
the
> >> > > out_tcmpr.tcst
> >> > > > > > file to only include times of RI/RW. Are RI
verification
> metrics
> >> > > > produced
> >> > > > > > by MET yet? Or is that planned for a future release?
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Matt
> >>
> >
>
>
> --
> -------------------------------------------------------
> Matthew Morin
> Engility
> High Technology Services Group
> GFDL
> Princeton University Forrestal Campus
> 201 Forrestal Road
> Princeton, NJ 08540
> phone: 609-452-5381
> email: Matthew.Morin at noaa.gov
> -------------------------------------------------------
>
>
------------------------------------------------
Subject: TC rapid intensification verification
From: John Halley Gotway
Time: Fri Jan 16 16:46:50 2015
Hello Matt,
We made some tweaks to the logic of the time window matching when
populating an RI/RW contingency table.
The earlier version of the code that you have looks for false alarms
(ADECK
yes, BDECK no) and searches the BDECK to see if the event actually did
occur close enough in time. The downside to this is that a single
event in
the BDECK could be counted as a hit multiple times.
The updated logic is slightly different and keeps the total number of
BDECK
events and non-events constant. When the ADECK and BDECK disagree,
the
code now searches the ADECK in that time window for agreement with the
BDECK. If found, it switches false alarms to correct negatives and
misses
to hits.
This logic is only applied when you set the matching time window
greater
than 0.
Also note that we switched all instances of "rapid_inten" to "rirw" to
make
things easier to type. For example, the "-rapid_inten_thresh" is now
named
"-rirw_thresh". And the "rapid_inten" section of the tc-stat config
file
is now named "rirw".
If you'd like to use the updated development version of MET, the code
can
be found here:
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/met-
rev2603.20150116.tar.gz
Thanks,
John
On Fri, Dec 12, 2014 at 11:43 AM, Matthew Morin - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055 >
>
> Hi John,
>
> I just finished using this developmental version of MET to calculate
> probability of detection and false alarm ratio for my hurricane
ensemble.
> Attached are the plots.
>
> Also attached is a small write-up of what I observed and suggest.
>
> Was there any other feedback or tests you want from me?
>
> Thanks,
> Matt
>
> On Wed, Dec 10, 2014 at 4:26 PM, Matthew Morin - NOAA Affiliate <
> matthew.morin at noaa.gov> wrote:
>
> > John,
> >
> > Thanks very much. I just quickly ran the command you listed and I
got
> good
> > output. I will learn the format of (and sort through) the output
> tomorrow.
> >
> > Thanks again,
> > Matt
> >
> > On Wed, Dec 10, 2014 at 4:13 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Done. You can find it in:
> >> /scratch1/portfolios/BMC/dtc/MET/met-rev2577
> >>
> >> 2577 is just the revision number of the MET subversion
repository.
> >>
> >> To get the most output, you'd run something like this...
> >> /scratch1/portfolios/BMC/dtc/MET/met-rev2577/bin/tc_stat \
> >> -lookin path/to/tc_pairs/output \
> >> -job rirw -dump_row test \
> >> -out_line_type CTC,CTS,MPR
> >>
> >> Please search for "rirw" in here:
> >> /scratch1/portfolios/BMC/dtc/MET/met-
rev2577/data/config/README_TC
> >> For a brief description of the job options.
> >>
> >> By default, rapid intensification is defined as a 24-hour exact
change
> >> exceeding 30kts. You can modify that definition using for the
ADECK,
> >> BDECK, or both using -rapid_inten_time, -rapid_inten_exact, and
> >> -rapid_inten_thresh options. Set -rapid_inten_window to
something
> larger
> >> than 0 to enable false alarms to be considered hits when they
were
> "close
> >> enough" in time.
> >>
> >> Evaluate rapid weakening by setting "-rapid_inten_thresh <=-30".
> >>
> >> Once you have the job working as you'd like, you can easily run
the same
> >> job for different lead time using the "-by LEAD" option.
> >>
> >> Please let me know how it all goes and if you have questions.
> >>
> >> Thanks,
> >> John
> >>
> >> On Wed, Dec 10, 2014 at 1:33 PM, Matthew Morin - NOAA Affiliate
via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055 >
> >> >
> >> > John,
> >> > Yes, I'm running MET on Zeus now. If you could build it there,
that
> >> would
> >> > be great.
> >> > Thanks,
> >> > -Matt
> >> >
> >> > On Wed, Dec 10, 2014 at 3:20 PM, John Halley Gotway via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > > Matt,
> >> > >
> >> > > I see that you were running MET on zeus in the past. Is that
still
> >> the
> >> > > case? If so, I could just build the development version
there.
> >> > >
> >> > > Would that suffice?
> >> > >
> >> > > Thanks,
> >> > > John
> >> > >
> >> > > On Wed, Dec 10, 2014 at 12:19 PM, Matthew Morin - NOAA
Affiliate via
> >> RT <
> >> > > met_help at ucar.edu> wrote:
> >> > >
> >> > > >
> >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055 >
> >> > > >
> >> > > > Hi John,
> >> > > >
> >> > > > Thanks for the information. Sure, I can be a MET RI/RW
> verification
> >> > > tester.
> >> > > > I developed an RI verification system (not very elaborate)
for the
> >> GFDL
> >> > > > ensemble, so I am familiar with the 2x2 contingency table
and
> >> > > verification
> >> > > > metrics.
> >> > > >
> >> > > > -Matt
> >> > > >
> >> > > > On Wed, Dec 10, 2014 at 1:54 PM, John Halley Gotway via RT
<
> >> > > > met_help at ucar.edu> wrote:
> >> > > >
> >> > > > > Matt,
> >> > > > >
> >> > > > > met-5.0 only has the ability to filter out the ri/rw
events, as
> >> > you've
> >> > > > > described. But we've developed and are testing
internally a new
> >> > > tc_stat
> >> > > > > job type called "rirw" which identifies rirw events in
both the
> >> ADECK
> >> > > and
> >> > > > > BDECK and uses them to populate a 2x2 contingency table
and dump
> >> out
> >> > > the
> >> > > > > corresponding statistics. We've included a lot of
flexibility
> in
> >> the
> >> > > > > definition of rirw events in the either deck and how
close the
> >> events
> >> > > > need
> >> > > > > to be in time to be considered a "hit". I've listed
below a
> >> > > description
> >> > > > of
> >> > > > > this job type from the README_TC file in the development
version
> >> of
> >> > the
> >> > > > > code.
> >> > > > >
> >> > > > > If you'd be willing/interested in testing out this
capability
> and
> >> > > > providing
> >> > > > > us feedback, we might be able to provide you a snapshot
of the
> >> latest
> >> > > > > development version.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > John
> >> > > > >
> >> > > > > // "rirw"
> >> > > > >
> >> > > > > // To define rapid intensification/weakening
> contingency
> >> > table
> >> > > > > using
> >> > > > >
> >> > > > > // the ADECK and BDECK RI/RW settings and the
matching
> >> time
> >> > > > window
> >> > > > >
> >> > > > > // and output contingency table counts and
statistics.
> >> > > > >
> >> > > > > //
> >> > > > >
> >> > > > > // Optional Args:
> >> > > > >
> >> > > > > // -rapid_inten_window HHMMSS to override
default of
> >> 0,
> >> > > > > requiring exact match
> >> > > > >
> >> > > > > // -rapid_inten_time or
-rapid_inten_time_adeck and
> >> > > > > -rapid_inten_time_bdeck to override defaults
> >> > > > >
> >> > > > > // -rapid_inten_exact or
-rapid_inten_exact_adeck
> and
> >> > > > > -rapid_inten_exact_bdeck to override defaults
> >> > > > >
> >> > > > > // -rapid_inten_thresh or
-rapid_inten_thresh_adeck
> >> and
> >> > > > > -rapid_inten_thresh_bdeck to override defaults
> >> > > > >
> >> > > > > // -by column_name to specify case
information
> >> > > > >
> >> > > > > // -out_alpha to override default alpha value
> >> > > > >
> >> > > > > // -out_line_type to specify which output
types
> (CTC,
> >> > CTS,
> >> > > > and
> >> > > > > MPR) should be written
> >> > > > >
> >> > > > > //
> >> > > > >
> >> > > > > // Note that the "-dump_row path" option results
in 4
> >> files
> >> > > > being
> >> > > > > created:
> >> > > > >
> >> > > > > // path_FY_OY.tcst, path_FY_ON.tcst,
> path_FN_OY.tcst,
> >> and
> >> > > > > path_FN_ON.tcst
> >> > > > >
> >> > > > > // containing the TCST lines that were hits,
false
> >> > alarms,
> >> > > > > misses, and correct negatives,
> >> > > > > // respectively. These files may be used as
input
> for
> >> > > > > additional TC-Stat analysis.
> >> > > > >
> >> > > > > On Wed, Dec 10, 2014 at 11:11 AM, Matthew Morin - NOAA
Affiliate
> >> via
> >> > > RT <
> >> > > > > met_help at ucar.edu> wrote:
> >> > > > >
> >> > > > > >
> >> > > > > > Wed Dec 10 11:11:37 2014: Request 70055 was acted upon.
> >> > > > > > Transaction: Ticket created by matthew.morin at noaa.gov
> >> > > > > > Queue: met_help
> >> > > > > > Subject: TC rapid intensification verification
> >> > > > > > Owner: Nobody
> >> > > > > > Requestors: matthew.morin at noaa.gov
> >> > > > > > Status: new
> >> > > > > > Ticket <URL:
> >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055
> >> > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > Hello. I thought I heard somewhere that MET can perform
> >> > verifications
> >> > > > of
> >> > > > > TC
> >> > > > > > rapid intensification (to produce output such as Brier
score
> and
> >> > ROC
> >> > > > > > diagrams). All I found so far is that MET can filter
the
> >> > > out_tcmpr.tcst
> >> > > > > > file to only include times of RI/RW. Are RI
verification
> metrics
> >> > > > produced
> >> > > > > > by MET yet? Or is that planned for a future release?
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Matt
> >>
> >
>
>
> --
> -------------------------------------------------------
> Matthew Morin
> Engility
> High Technology Services Group
> GFDL
> Princeton University Forrestal Campus
> 201 Forrestal Road
> Princeton, NJ 08540
> phone: 609-452-5381
> email: Matthew.Morin at noaa.gov
> -------------------------------------------------------
>
>
------------------------------------------------
Subject: TC rapid intensification verification
From: Matthew Morin - NOAA Affiliate
Time: Tue Jan 20 07:46:21 2015
Hi John,
Thanks for the update.
-Matt
On Fri, Jan 16, 2015 at 6:46 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Hello Matt,
>
> We made some tweaks to the logic of the time window matching when
> populating an RI/RW contingency table.
>
> The earlier version of the code that you have looks for false alarms
(ADECK
> yes, BDECK no) and searches the BDECK to see if the event actually
did
> occur close enough in time. The downside to this is that a single
event in
> the BDECK could be counted as a hit multiple times.
>
> The updated logic is slightly different and keeps the total number
of BDECK
> events and non-events constant. When the ADECK and BDECK disagree,
the
> code now searches the ADECK in that time window for agreement with
the
> BDECK. If found, it switches false alarms to correct negatives and
misses
> to hits.
>
> This logic is only applied when you set the matching time window
greater
> than 0.
>
> Also note that we switched all instances of "rapid_inten" to "rirw"
to make
> things easier to type. For example, the "-rapid_inten_thresh" is
now named
> "-rirw_thresh". And the "rapid_inten" section of the tc-stat config
file
> is now named "rirw".
>
> If you'd like to use the updated development version of MET, the
code can
> be found here:
>
> ftp://ftp.rap.ucar.edu/incoming/irap/met_help/met-
rev2603.20150116.tar.gz
>
> Thanks,
> John
>
> On Fri, Dec 12, 2014 at 11:43 AM, Matthew Morin - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055 >
> >
> > Hi John,
> >
> > I just finished using this developmental version of MET to
calculate
> > probability of detection and false alarm ratio for my hurricane
ensemble.
> > Attached are the plots.
> >
> > Also attached is a small write-up of what I observed and suggest.
> >
> > Was there any other feedback or tests you want from me?
> >
> > Thanks,
> > Matt
> >
> > On Wed, Dec 10, 2014 at 4:26 PM, Matthew Morin - NOAA Affiliate <
> > matthew.morin at noaa.gov> wrote:
> >
> > > John,
> > >
> > > Thanks very much. I just quickly ran the command you listed and
I got
> > good
> > > output. I will learn the format of (and sort through) the output
> > tomorrow.
> > >
> > > Thanks again,
> > > Matt
> > >
> > > On Wed, Dec 10, 2014 at 4:13 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >> Done. You can find it in:
> > >> /scratch1/portfolios/BMC/dtc/MET/met-rev2577
> > >>
> > >> 2577 is just the revision number of the MET subversion
repository.
> > >>
> > >> To get the most output, you'd run something like this...
> > >> /scratch1/portfolios/BMC/dtc/MET/met-rev2577/bin/tc_stat \
> > >> -lookin path/to/tc_pairs/output \
> > >> -job rirw -dump_row test \
> > >> -out_line_type CTC,CTS,MPR
> > >>
> > >> Please search for "rirw" in here:
> > >> /scratch1/portfolios/BMC/dtc/MET/met-
rev2577/data/config/README_TC
> > >> For a brief description of the job options.
> > >>
> > >> By default, rapid intensification is defined as a 24-hour exact
change
> > >> exceeding 30kts. You can modify that definition using for the
ADECK,
> > >> BDECK, or both using -rapid_inten_time, -rapid_inten_exact, and
> > >> -rapid_inten_thresh options. Set -rapid_inten_window to
something
> > larger
> > >> than 0 to enable false alarms to be considered hits when they
were
> > "close
> > >> enough" in time.
> > >>
> > >> Evaluate rapid weakening by setting "-rapid_inten_thresh <=-
30".
> > >>
> > >> Once you have the job working as you'd like, you can easily run
the
> same
> > >> job for different lead time using the "-by LEAD" option.
> > >>
> > >> Please let me know how it all goes and if you have questions.
> > >>
> > >> Thanks,
> > >> John
> > >>
> > >> On Wed, Dec 10, 2014 at 1:33 PM, Matthew Morin - NOAA Affiliate
via
> RT <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >> >
> > >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055
>
> > >> >
> > >> > John,
> > >> > Yes, I'm running MET on Zeus now. If you could build it
there, that
> > >> would
> > >> > be great.
> > >> > Thanks,
> > >> > -Matt
> > >> >
> > >> > On Wed, Dec 10, 2014 at 3:20 PM, John Halley Gotway via RT <
> > >> > met_help at ucar.edu> wrote:
> > >> >
> > >> > > Matt,
> > >> > >
> > >> > > I see that you were running MET on zeus in the past. Is
that
> still
> > >> the
> > >> > > case? If so, I could just build the development version
there.
> > >> > >
> > >> > > Would that suffice?
> > >> > >
> > >> > > Thanks,
> > >> > > John
> > >> > >
> > >> > > On Wed, Dec 10, 2014 at 12:19 PM, Matthew Morin - NOAA
Affiliate
> via
> > >> RT <
> > >> > > met_help at ucar.edu> wrote:
> > >> > >
> > >> > > >
> > >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055 >
> > >> > > >
> > >> > > > Hi John,
> > >> > > >
> > >> > > > Thanks for the information. Sure, I can be a MET RI/RW
> > verification
> > >> > > tester.
> > >> > > > I developed an RI verification system (not very
elaborate) for
> the
> > >> GFDL
> > >> > > > ensemble, so I am familiar with the 2x2 contingency table
and
> > >> > > verification
> > >> > > > metrics.
> > >> > > >
> > >> > > > -Matt
> > >> > > >
> > >> > > > On Wed, Dec 10, 2014 at 1:54 PM, John Halley Gotway via
RT <
> > >> > > > met_help at ucar.edu> wrote:
> > >> > > >
> > >> > > > > Matt,
> > >> > > > >
> > >> > > > > met-5.0 only has the ability to filter out the ri/rw
events,
> as
> > >> > you've
> > >> > > > > described. But we've developed and are testing
internally a
> new
> > >> > > tc_stat
> > >> > > > > job type called "rirw" which identifies rirw events in
both
> the
> > >> ADECK
> > >> > > and
> > >> > > > > BDECK and uses them to populate a 2x2 contingency table
and
> dump
> > >> out
> > >> > > the
> > >> > > > > corresponding statistics. We've included a lot of
flexibility
> > in
> > >> the
> > >> > > > > definition of rirw events in the either deck and how
close the
> > >> events
> > >> > > > need
> > >> > > > > to be in time to be considered a "hit". I've listed
below a
> > >> > > description
> > >> > > > of
> > >> > > > > this job type from the README_TC file in the
development
> version
> > >> of
> > >> > the
> > >> > > > > code.
> > >> > > > >
> > >> > > > > If you'd be willing/interested in testing out this
capability
> > and
> > >> > > > providing
> > >> > > > > us feedback, we might be able to provide you a snapshot
of the
> > >> latest
> > >> > > > > development version.
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > John
> > >> > > > >
> > >> > > > > // "rirw"
> > >> > > > >
> > >> > > > > // To define rapid intensification/weakening
> > contingency
> > >> > table
> > >> > > > > using
> > >> > > > >
> > >> > > > > // the ADECK and BDECK RI/RW settings and the
> matching
> > >> time
> > >> > > > window
> > >> > > > >
> > >> > > > > // and output contingency table counts and
> statistics.
> > >> > > > >
> > >> > > > > //
> > >> > > > >
> > >> > > > > // Optional Args:
> > >> > > > >
> > >> > > > > // -rapid_inten_window HHMMSS to override
default
> of
> > >> 0,
> > >> > > > > requiring exact match
> > >> > > > >
> > >> > > > > // -rapid_inten_time or
-rapid_inten_time_adeck
> and
> > >> > > > > -rapid_inten_time_bdeck to override defaults
> > >> > > > >
> > >> > > > > // -rapid_inten_exact or
-rapid_inten_exact_adeck
> > and
> > >> > > > > -rapid_inten_exact_bdeck to override defaults
> > >> > > > >
> > >> > > > > // -rapid_inten_thresh or
> -rapid_inten_thresh_adeck
> > >> and
> > >> > > > > -rapid_inten_thresh_bdeck to override defaults
> > >> > > > >
> > >> > > > > // -by column_name to specify case
information
> > >> > > > >
> > >> > > > > // -out_alpha to override default alpha
value
> > >> > > > >
> > >> > > > > // -out_line_type to specify which output
types
> > (CTC,
> > >> > CTS,
> > >> > > > and
> > >> > > > > MPR) should be written
> > >> > > > >
> > >> > > > > //
> > >> > > > >
> > >> > > > > // Note that the "-dump_row path" option
results in 4
> > >> files
> > >> > > > being
> > >> > > > > created:
> > >> > > > >
> > >> > > > > // path_FY_OY.tcst, path_FY_ON.tcst,
> > path_FN_OY.tcst,
> > >> and
> > >> > > > > path_FN_ON.tcst
> > >> > > > >
> > >> > > > > // containing the TCST lines that were
hits, false
> > >> > alarms,
> > >> > > > > misses, and correct negatives,
> > >> > > > > // respectively. These files may be used
as input
> > for
> > >> > > > > additional TC-Stat analysis.
> > >> > > > >
> > >> > > > > On Wed, Dec 10, 2014 at 11:11 AM, Matthew Morin - NOAA
> Affiliate
> > >> via
> > >> > > RT <
> > >> > > > > met_help at ucar.edu> wrote:
> > >> > > > >
> > >> > > > > >
> > >> > > > > > Wed Dec 10 11:11:37 2014: Request 70055 was acted
upon.
> > >> > > > > > Transaction: Ticket created by matthew.morin at noaa.gov
> > >> > > > > > Queue: met_help
> > >> > > > > > Subject: TC rapid intensification verification
> > >> > > > > > Owner: Nobody
> > >> > > > > > Requestors: matthew.morin at noaa.gov
> > >> > > > > > Status: new
> > >> > > > > > Ticket <URL:
> > >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=70055
> > >> > > > >
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Hello. I thought I heard somewhere that MET can
perform
> > >> > verifications
> > >> > > > of
> > >> > > > > TC
> > >> > > > > > rapid intensification (to produce output such as
Brier score
> > and
> > >> > ROC
> > >> > > > > > diagrams). All I found so far is that MET can filter
the
> > >> > > out_tcmpr.tcst
> > >> > > > > > file to only include times of RI/RW. Are RI
verification
> > metrics
> > >> > > > produced
> > >> > > > > > by MET yet? Or is that planned for a future release?
> > >> > > > > >
> > >> > > > > > Thanks,
> > >> > > > > > Matt
> > >>
> > >
> >
> >
> > --
> > -------------------------------------------------------
> > Matthew Morin
> > Engility
> > High Technology Services Group
> > GFDL
> > Princeton University Forrestal Campus
> > 201 Forrestal Road
> > Princeton, NJ 08540
> > phone: 609-452-5381
> > email: Matthew.Morin at noaa.gov
> > -------------------------------------------------------
> >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list