[Met_help] [rt.rap.ucar.edu #87885] History for Help with using TC_pairs
John Halley Gotway via RT
met_help at ucar.edu
Thu Dec 6 10:20:55 MST 2018
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi,
I'm trying to use TC_pairs for extra-tropical cyclones (basin=ML). I think
the format of the ATCF data is the issue, but, I'm not sure. I have an
Adeck and a Bdeck file, where a sample of one of the lines is:
ML, 0003, 2018090100_F000_687N_0159W_FOF, 2018090100, 03, GFSO, 000, 687N,
159W, 32, 987, XX, 34, NEQ, 0301, 0338, 0000, 0000, 988, 52, -99,
330, 86, -99, -9999, -9999, 175, 202, 137, 135, 0081
This is slightly different from the NRL ATCF format on their website, such
as the second column. Anyway, I configured the config file very simply,
(model=GFS, basin=ML), and ran tc_pairs, and got the output to the screen
below.
Is there a way to work around the diferent ATCF format? I've attached a
sample of my Adeck and Bdeck files.
Thanks,
Roz
Output to the screen:
/opt/MET7/bin/tc_pairs -adeck
~/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003
-bdeck
~/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003
-config
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
-out
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt
-log ~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
-v 3
DEBUG 1: [Source 1 of 1] ADECK Source:
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003,
Model Suffix: (nul)
DEBUG 1: [Source 1 of 1] BDECK Source:
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003,
Model Suffix: (nul)
DEBUG 1: Config File Default:
/opt/MET7/share/met/config/TCPairsConfig_default
DEBUG 1: Config File User:
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
DEBUG 1: Distance to land file: /opt/MET7/share/met/tc_data/
dland_global_tenth_degree.nc
DEBUG 2: Processing 1 BDECK file(s).
WARNING:
WARNING: int parse_lat(const char *) -> bad latitude ... "000"
WARNING:
WARNING:
WARNING: int parse_lon(const char *) -> bad longitude ... "687N"
WARNING:
WARNING:
WARNING: int parse_lat(const char *) -> bad latitude ... "000"
WARNING:
WARNING:
WARNING: int parse_lon(const char *) -> bad longitude ... "701N"
WARNING:
WARNING:
WARNING: int parse_lat(const char *) -> bad latitude ... "000"
WARNING:
WARNING:
WARNING: int parse_lon(const char *) -> bad longitude ... "711N"
WARNING:
WARNING:
WARNING: int parse_lat(const char *) -> bad latitude ... "000"
WARNING:
WARNING:
WARNING: int parse_lon(const char *) -> bad longitude ... "720N"
WARNING:
WARNING:
WARNING: int parse_lat(const char *) -> bad latitude ... "000"
WARNING:
WARNING:
WARNING: int parse_lon(const char *) -> bad longitude ... "730N"
WARNING:
WARNING:
WARNING: int parse_lat(const char *) -> bad latitude ... "000"
WARNING:
WARNING:
WARNING: int parse_lon(const char *) -> bad longitude ... "741N"
WARNING:
WARNING:
WARNING: int parse_lat(const char *) -> bad latitude ... "000"
WARNING:
WARNING:
WARNING: int parse_lon(const char *) -> bad longitude ... "751N"
WARNING:
WARNING:
WARNING: int parse_lat(const char *) -> bad latitude ... "000"
WARNING:
WARNING:
WARNING: int parse_lon(const char *) -> bad longitude ... "760N"
WARNING:
WARNING:
WARNING: int parse_lat(const char *) -> bad latitude ... "000"
WARNING:
WARNING:
WARNING: int parse_lon(const char *) -> bad longitude ... "764N"
WARNING:
DEBUG 3: Used 9 of 9 lines read from 1 file(s).
DEBUG 3: Identified 9 track(s).
DEBUG 2: Found 9 BDECK track(s).
DEBUG 2: Processing 1 ADECK file(s).
DEBUG 3: Used 0 of 22 lines read from 1 file(s).
DEBUG 3: Identified 0 track(s).
DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
DEBUG 2: Finished adding 0 and replacing 0 Interp12 track(s).
DEBUG 2: Deriving 0 ADECK consensus model(s).
DEBUG 2: Added 0 ADECK consensus tracks(s).
DEBUG 2: Deriving 0 ADECK lag model(s).
DEBUG 2: Added 0 ADECK lag tracks(s).
DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
DEBUG 2: Filtering 0 ADECK tracks based on config file settings.
DEBUG 3: Total tracks read = 0
DEBUG 3: Total tracks kept = 0
DEBUG 3: Rejected for storm name = 0
DEBUG 3: Rejected for valid time = 0
DEBUG 3: Rejected for required lead times = 0
DEBUG 3: Rejected for init mask = 0
DEBUG 3: Rejected for valid mask = 0
DEBUG 2: Matching 0 ADECK tracks to 9 BDECK tracks.
DEBUG 1: Watch/Warning file: /opt/MET7/share/met/tc_data/wwpts_us.txt
DEBUG 1: Output file:
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt.tcst
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Help with using TC_pairs
From: John Halley Gotway
Time: Tue Nov 20 17:53:40 2018
Hi Roz,
Yes, I see you’re using the modified ATCF format that Guang Ping uses.
You’re correct that tc_pairs doesn’t parse this directly since it
differs
slightly from the ATCF format.
When we’ve processed this data in the past, we first pre-processed it
in a
shell script to:
1. Replace the -99 value with -9999 for missing data.
2. Remove the 3rd column
3. Updating the 2nd column to include the year and cyclone number.
And that enable tc_pairs to read it.
This is actually Done in python by one of the use cases in METplus.
Take a look at this file:
https://github.com/NCAR/METplus/blob/master/ush/tc_pairs_wrapper.py
At the function named:
process_non_atcf
I haven’t run this using METplus scripts myself, but all the pieces of
logic are there, and it should be possible.
Make sense?
Thanks
John
On Tue, Nov 20, 2018 at 1:13 PM Rosalyn MacCracken - NOAA Affiliate
via RT <
met_help at ucar.edu> wrote:
>
> Tue Nov 20 13:13:05 2018: Request 87885 was acted upon.
> Transaction: Ticket created by rosalyn.maccracken at noaa.gov
> Queue: met_help
> Subject: Help with using TC_pairs
> Owner: Nobody
> Requestors: rosalyn.maccracken at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
>
>
> Hi,
>
> I'm trying to use TC_pairs for extra-tropical cyclones (basin=ML).
I think
> the format of the ATCF data is the issue, but, I'm not sure. I have
an
> Adeck and a Bdeck file, where a sample of one of the lines is:
> ML, 0003, 2018090100_F000_687N_0159W_FOF, 2018090100, 03, GFSO, 000,
687N,
> 159W, 32, 987, XX, 34, NEQ, 0301, 0338, 0000, 0000, 988, 52,
-99,
> 330, 86, -99, -9999, -9999, 175, 202, 137, 135, 0081
>
> This is slightly different from the NRL ATCF format on their
website, such
> as the second column. Anyway, I configured the config file very
simply,
> (model=GFS, basin=ML), and ran tc_pairs, and got the output to the
screen
> below.
>
> Is there a way to work around the diferent ATCF format? I've
attached a
> sample of my Adeck and Bdeck files.
>
> Thanks,
>
> Roz
>
> Output to the screen:
> /opt/MET7/bin/tc_pairs -adeck
>
~/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003
> -bdeck
>
~/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003
> -config
>
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> -out
>
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt
> -log
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> -v 3
> DEBUG 1: [Source 1 of 1] ADECK Source:
>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003,
> Model Suffix: (nul)
> DEBUG 1: [Source 1 of 1] BDECK Source:
>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003,
> Model Suffix: (nul)
> DEBUG 1: Config File Default:
> /opt/MET7/share/met/config/TCPairsConfig_default
> DEBUG 1: Config File User:
>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> DEBUG 1: Distance to land file: /opt/MET7/share/met/tc_data/
> dland_global_tenth_degree.nc
> DEBUG 2: Processing 1 BDECK file(s).
> WARNING:
> WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> WARNING:
> WARNING:
> WARNING: int parse_lon(const char *) -> bad longitude ... "687N"
> WARNING:
> WARNING:
> WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> WARNING:
> WARNING:
> WARNING: int parse_lon(const char *) -> bad longitude ... "701N"
> WARNING:
> WARNING:
> WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> WARNING:
> WARNING:
> WARNING: int parse_lon(const char *) -> bad longitude ... "711N"
> WARNING:
> WARNING:
> WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> WARNING:
> WARNING:
> WARNING: int parse_lon(const char *) -> bad longitude ... "720N"
> WARNING:
> WARNING:
> WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> WARNING:
> WARNING:
> WARNING: int parse_lon(const char *) -> bad longitude ... "730N"
> WARNING:
> WARNING:
> WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> WARNING:
> WARNING:
> WARNING: int parse_lon(const char *) -> bad longitude ... "741N"
> WARNING:
> WARNING:
> WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> WARNING:
> WARNING:
> WARNING: int parse_lon(const char *) -> bad longitude ... "751N"
> WARNING:
> WARNING:
> WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> WARNING:
> WARNING:
> WARNING: int parse_lon(const char *) -> bad longitude ... "760N"
> WARNING:
> WARNING:
> WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> WARNING:
> WARNING:
> WARNING: int parse_lon(const char *) -> bad longitude ... "764N"
> WARNING:
> DEBUG 3: Used 9 of 9 lines read from 1 file(s).
> DEBUG 3: Identified 9 track(s).
> DEBUG 2: Found 9 BDECK track(s).
> DEBUG 2: Processing 1 ADECK file(s).
> DEBUG 3: Used 0 of 22 lines read from 1 file(s).
> DEBUG 3: Identified 0 track(s).
> DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> DEBUG 2: Finished adding 0 and replacing 0 Interp12 track(s).
> DEBUG 2: Deriving 0 ADECK consensus model(s).
> DEBUG 2: Added 0 ADECK consensus tracks(s).
> DEBUG 2: Deriving 0 ADECK lag model(s).
> DEBUG 2: Added 0 ADECK lag tracks(s).
> DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> DEBUG 2: Filtering 0 ADECK tracks based on config file settings.
> DEBUG 3: Total tracks read = 0
> DEBUG 3: Total tracks kept = 0
> DEBUG 3: Rejected for storm name = 0
> DEBUG 3: Rejected for valid time = 0
> DEBUG 3: Rejected for required lead times = 0
> DEBUG 3: Rejected for init mask = 0
> DEBUG 3: Rejected for valid mask = 0
> DEBUG 2: Matching 0 ADECK tracks to 9 BDECK tracks.
> DEBUG 1: Watch/Warning file:
/opt/MET7/share/met/tc_data/wwpts_us.txt
> DEBUG 1: Output file:
>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt.tcst
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
>
------------------------------------------------
Subject: Help with using TC_pairs
From: Rosalyn MacCracken - NOAA Affiliate
Time: Wed Nov 21 07:16:18 2018
Hi John,
Yes, that makes sense, and yes, I am using Guang Pings' modified
files.
I'm not sure how to run tc_pairs_wrapper.py, though. I think I need
some
type of *.conf file to pass in the files I need, or something like
that.
I'm in the process of reading through the code now. I wonder if I
could
just pull out the function process_non_atcf, to run in a stand-alone
mode.
I wish there was some sort of documentation with this code...
And, so, once I can read these files, I can use tc_pairs to create the
forecast error statistics, and then use tc_stats to aggregate the
files and
get the mean, etc, for a period of time, correct? Was there some sort
of
plotting scripts for the output files?
So, with tc_pairs, I can define a region, like I did with point_stat,
correct? Can I use the same *.nc files, or do I use the original
*.poly
files?
Roz
On Tue, Nov 20, 2018 at 7:53 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Hi Roz,
>
> Yes, I see you’re using the modified ATCF format that Guang Ping
uses.
> You’re correct that tc_pairs doesn’t parse this directly since it
differs
> slightly from the ATCF format.
>
> When we’ve processed this data in the past, we first pre-processed
it in a
> shell script to:
>
> 1. Replace the -99 value with -9999 for missing data.
> 2. Remove the 3rd column
> 3. Updating the 2nd column to include the year and cyclone number.
>
> And that enable tc_pairs to read it.
>
> This is actually Done in python by one of the use cases in METplus.
>
> Take a look at this file:
> https://github.com/NCAR/METplus/blob/master/ush/tc_pairs_wrapper.py
>
> At the function named:
> process_non_atcf
>
> I haven’t run this using METplus scripts myself, but all the pieces
of
> logic are there, and it should be possible.
>
> Make sense?
>
> Thanks
> John
>
> On Tue, Nov 20, 2018 at 1:13 PM Rosalyn MacCracken - NOAA Affiliate
via RT
> <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Nov 20 13:13:05 2018: Request 87885 was acted upon.
> > Transaction: Ticket created by rosalyn.maccracken at noaa.gov
> > Queue: met_help
> > Subject: Help with using TC_pairs
> > Owner: Nobody
> > Requestors: rosalyn.maccracken at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
> >
> >
> > Hi,
> >
> > I'm trying to use TC_pairs for extra-tropical cyclones (basin=ML).
I
> think
> > the format of the ATCF data is the issue, but, I'm not sure. I
have an
> > Adeck and a Bdeck file, where a sample of one of the lines is:
> > ML, 0003, 2018090100_F000_687N_0159W_FOF, 2018090100, 03, GFSO,
000,
> 687N,
> > 159W, 32, 987, XX, 34, NEQ, 0301, 0338, 0000, 0000, 988, 52,
-99,
> > 330, 86, -99, -9999, -9999, 175, 202, 137, 135, 0081
> >
> > This is slightly different from the NRL ATCF format on their
website,
> such
> > as the second column. Anyway, I configured the config file very
simply,
> > (model=GFS, basin=ML), and ran tc_pairs, and got the output to the
screen
> > below.
> >
> > Is there a way to work around the diferent ATCF format? I've
attached a
> > sample of my Adeck and Bdeck files.
> >
> > Thanks,
> >
> > Roz
> >
> > Output to the screen:
> > /opt/MET7/bin/tc_pairs -adeck
> >
>
~/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003
> > -bdeck
> >
>
~/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003
> > -config
> >
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> > -out
> >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt
> > -log
>
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> > -v 3
> > DEBUG 1: [Source 1 of 1] ADECK Source:
> >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003,
> > Model Suffix: (nul)
> > DEBUG 1: [Source 1 of 1] BDECK Source:
> >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003,
> > Model Suffix: (nul)
> > DEBUG 1: Config File Default:
> > /opt/MET7/share/met/config/TCPairsConfig_default
> > DEBUG 1: Config File User:
> >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> > DEBUG 1: Distance to land file: /opt/MET7/share/met/tc_data/
> > dland_global_tenth_degree.nc
> > DEBUG 2: Processing 1 BDECK file(s).
> > WARNING:
> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> > WARNING:
> > WARNING:
> > WARNING: int parse_lon(const char *) -> bad longitude ... "687N"
> > WARNING:
> > WARNING:
> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> > WARNING:
> > WARNING:
> > WARNING: int parse_lon(const char *) -> bad longitude ... "701N"
> > WARNING:
> > WARNING:
> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> > WARNING:
> > WARNING:
> > WARNING: int parse_lon(const char *) -> bad longitude ... "711N"
> > WARNING:
> > WARNING:
> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> > WARNING:
> > WARNING:
> > WARNING: int parse_lon(const char *) -> bad longitude ... "720N"
> > WARNING:
> > WARNING:
> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> > WARNING:
> > WARNING:
> > WARNING: int parse_lon(const char *) -> bad longitude ... "730N"
> > WARNING:
> > WARNING:
> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> > WARNING:
> > WARNING:
> > WARNING: int parse_lon(const char *) -> bad longitude ... "741N"
> > WARNING:
> > WARNING:
> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> > WARNING:
> > WARNING:
> > WARNING: int parse_lon(const char *) -> bad longitude ... "751N"
> > WARNING:
> > WARNING:
> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> > WARNING:
> > WARNING:
> > WARNING: int parse_lon(const char *) -> bad longitude ... "760N"
> > WARNING:
> > WARNING:
> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> > WARNING:
> > WARNING:
> > WARNING: int parse_lon(const char *) -> bad longitude ... "764N"
> > WARNING:
> > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
> > DEBUG 3: Identified 9 track(s).
> > DEBUG 2: Found 9 BDECK track(s).
> > DEBUG 2: Processing 1 ADECK file(s).
> > DEBUG 3: Used 0 of 22 lines read from 1 file(s).
> > DEBUG 3: Identified 0 track(s).
> > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> > DEBUG 2: Finished adding 0 and replacing 0 Interp12 track(s).
> > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > DEBUG 2: Deriving 0 ADECK lag model(s).
> > DEBUG 2: Added 0 ADECK lag tracks(s).
> > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> > DEBUG 2: Filtering 0 ADECK tracks based on config file settings.
> > DEBUG 3: Total tracks read = 0
> > DEBUG 3: Total tracks kept = 0
> > DEBUG 3: Rejected for storm name = 0
> > DEBUG 3: Rejected for valid time = 0
> > DEBUG 3: Rejected for required lead times = 0
> > DEBUG 3: Rejected for init mask = 0
> > DEBUG 3: Rejected for valid mask = 0
> > DEBUG 2: Matching 0 ADECK tracks to 9 BDECK tracks.
> > DEBUG 1: Watch/Warning file:
/opt/MET7/share/met/tc_data/wwpts_us.txt
> > DEBUG 1: Output file:
> >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt.tcst
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applications Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
> >
>
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Help with using TC_pairs
From: Rosalyn MacCracken - NOAA Affiliate
Time: Mon Nov 26 08:44:06 2018
Hi John,
I figured out an easy way to read in and write out Guang Ping's
modified
ATCF file, so that it:
1) Removes the 3rd column
2) Replaces -99 with -9999 in column 21 and 24 (after the 3rd column
is
removed)
3) Updates column 2 to include month+cyclone number. (You had said
to do
year+cyclone #, but the tc_pairs_wrapper.py said to do month+cyclone#)
I changed Adeck and Bdeck.
Just a couple lines of my file:
ML, 090002, 2018090100, 03, GFSO, 000, 511S, 897W, 40, 976, XX,
34,
NEQ, 0625, 0630, 0000, 0226, 978, 68, -9999, 123, 155, -9999,
-9999,
-9999, -297, -394, -287, -458, 0073
ML, 090002, 2018090106, 03, GFSO, 000, 537S, 854W, 44, 964, XX,
34,
NEQ, 0640, 0645, 0000, 0297, 969, 108, -9999, 119, 147, -9999,
-9999,
-9999, -403, -648, -305, -394, 0055
So, I've run tc_pairs again and my output is below. I don't
understand the
Model Suffix: (nul). Is the input format still wrong?
Roz
Output:
/opt/MET7/bin/tc_pairs -adeck
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat
-bdeck
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat
-config
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
-out
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt
-log
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
-v 4
DEBUG 1: [Source 1 of 1] ADECK Source:
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat,
Model Suffix: (nul)
DEBUG 1: [Source 1 of 1] BDECK Source:
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat,
Model Suffix: (nul)
DEBUG 1: Config File Default:
/opt/MET7/share/met/config/TCPairsConfig_default
DEBUG 1: Config File User:
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
DEBUG 1: Distance to land file: /opt/MET7/share/met/tc_data/
dland_global_tenth_degree.nc
DEBUG 4:
DEBUG 4: Latitude/Longitude Grid Data:
DEBUG 4: lat_ll: -90
DEBUG 4: lon_ll: 180
DEBUG 4: delta_lat: 0.1
DEBUG 4: delta_lon: 0.1
DEBUG 4: Nlat: 1801
DEBUG 4: Nlon: 3601
DEBUG 4:
DEBUG 2: Processing 1 BDECK file(s).
DEBUG 4: Updating storm name from "-287" to "-305" for ML0900022018.
DEBUG 4: Updating storm name from "-305" to "-255" for ML0900022018.
DEBUG 4: Updating storm name from "-255" to "-203" for ML0900022018.
DEBUG 4: Updating storm name from "-203" to "-205" for ML0900022018.
DEBUG 4: Updating storm name from "-205" to "-217" for ML0900022018.
DEBUG 4: Updating storm name from "-217" to "-211" for ML0900022018.
DEBUG 4: Updating storm name from "-211" to "-182" for ML0900022018.
DEBUG 4: Updating storm name from "-182" to "-138" for ML0900022018.
DEBUG 4: [File 1 of 1] Used 9 of 9 lines read from file
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat"
DEBUG 3: Used 9 of 9 lines read from 1 file(s).
DEBUG 3: Identified 1 track(s).
DEBUG 4: [Track 1 of 1] TrackInfo: StormId = "ML0900022018", IsBest =
false, IsOper = false, CheckAnly = true, IsAnly = true, Basin = "ML",
Cyclone = "090002", StormName = "-138", TechniqueNumber = 3, Technique
=
"GFSO", Initials = "-9999", InitTime = NA, MinValidTime =
20180901_000000,
MaxValidTime = 20180903_000000, NPoints = 9, NAlloc = 100, NTrackLines
= 0
DEBUG 2: Found 1 BDECK track(s).
DEBUG 2: Processing 1 ADECK file(s).
DEBUG 4: [File 1 of 1] Used 0 of 21 lines read from file
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat"
DEBUG 3: Used 0 of 21 lines read from 1 file(s).
DEBUG 3: Identified 0 track(s).
DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
DEBUG 2: Finished adding 0 and replacing 0 Interp12 track(s).
DEBUG 2: Deriving 0 ADECK consensus model(s).
DEBUG 2: Added 0 ADECK consensus tracks(s).
DEBUG 2: Deriving 0 ADECK lag model(s).
DEBUG 2: Added 0 ADECK lag tracks(s).
DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
DEBUG 2: Filtering 0 ADECK tracks based on config file settings.
DEBUG 3: Total tracks read = 0
DEBUG 3: Total tracks kept = 0
DEBUG 3: Rejected for storm name = 0
DEBUG 3: Rejected for valid time = 0
DEBUG 3: Rejected for required lead times = 0
DEBUG 3: Rejected for init mask = 0
DEBUG 3: Rejected for valid mask = 0
DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
DEBUG 1: Watch/Warning file: /opt/MET7/share/met/tc_data/wwpts_us.txt
DEBUG 1: Output file:
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt.tcst
On Wed, Nov 21, 2018 at 9:16 AM Rosalyn MacCracken - NOAA Affiliate <
rosalyn.maccracken at noaa.gov> wrote:
> Hi John,
>
> Yes, that makes sense, and yes, I am using Guang Pings' modified
files.
> I'm not sure how to run tc_pairs_wrapper.py, though. I think I need
some
> type of *.conf file to pass in the files I need, or something like
that.
> I'm in the process of reading through the code now. I wonder if I
could
> just pull out the function process_non_atcf, to run in a stand-alone
mode.
> I wish there was some sort of documentation with this code...
>
> And, so, once I can read these files, I can use tc_pairs to create
the
> forecast error statistics, and then use tc_stats to aggregate the
files and
> get the mean, etc, for a period of time, correct? Was there some
sort of
> plotting scripts for the output files?
>
> So, with tc_pairs, I can define a region, like I did with
point_stat,
> correct? Can I use the same *.nc files, or do I use the original
*.poly
> files?
>
> Roz
>
> On Tue, Nov 20, 2018 at 7:53 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Hi Roz,
>>
>> Yes, I see you’re using the modified ATCF format that Guang Ping
uses.
>> You’re correct that tc_pairs doesn’t parse this directly since it
differs
>> slightly from the ATCF format.
>>
>> When we’ve processed this data in the past, we first pre-processed
it in a
>> shell script to:
>>
>> 1. Replace the -99 value with -9999 for missing data.
>> 2. Remove the 3rd column
>> 3. Updating the 2nd column to include the year and cyclone number.
>>
>> And that enable tc_pairs to read it.
>>
>> This is actually Done in python by one of the use cases in METplus.
>>
>> Take a look at this file:
>> https://github.com/NCAR/METplus/blob/master/ush/tc_pairs_wrapper.py
>>
>> At the function named:
>> process_non_atcf
>>
>> I haven’t run this using METplus scripts myself, but all the pieces
of
>> logic are there, and it should be possible.
>>
>> Make sense?
>>
>> Thanks
>> John
>>
>> On Tue, Nov 20, 2018 at 1:13 PM Rosalyn MacCracken - NOAA Affiliate
via
>> RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > Tue Nov 20 13:13:05 2018: Request 87885 was acted upon.
>> > Transaction: Ticket created by rosalyn.maccracken at noaa.gov
>> > Queue: met_help
>> > Subject: Help with using TC_pairs
>> > Owner: Nobody
>> > Requestors: rosalyn.maccracken at noaa.gov
>> > Status: new
>> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
>> >
>> >
>> > Hi,
>> >
>> > I'm trying to use TC_pairs for extra-tropical cyclones
(basin=ML). I
>> think
>> > the format of the ATCF data is the issue, but, I'm not sure. I
have an
>> > Adeck and a Bdeck file, where a sample of one of the lines is:
>> > ML, 0003, 2018090100_F000_687N_0159W_FOF, 2018090100, 03, GFSO,
000,
>> 687N,
>> > 159W, 32, 987, XX, 34, NEQ, 0301, 0338, 0000, 0000, 988,
52, -99,
>> > 330, 86, -99, -9999, -9999, 175, 202, 137, 135, 0081
>> >
>> > This is slightly different from the NRL ATCF format on their
website,
>> such
>> > as the second column. Anyway, I configured the config file very
simply,
>> > (model=GFS, basin=ML), and ran tc_pairs, and got the output to
the
>> screen
>> > below.
>> >
>> > Is there a way to work around the diferent ATCF format? I've
attached a
>> > sample of my Adeck and Bdeck files.
>> >
>> > Thanks,
>> >
>> > Roz
>> >
>> > Output to the screen:
>> > /opt/MET7/bin/tc_pairs -adeck
>> >
>>
~/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003
>> > -bdeck
>> >
>>
~/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003
>> > -config
>> >
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
>> > -out
>> >
>> >
>>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt
>> > -log
>>
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
>> > -v 3
>> > DEBUG 1: [Source 1 of 1] ADECK Source:
>> >
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003,
>> > Model Suffix: (nul)
>> > DEBUG 1: [Source 1 of 1] BDECK Source:
>> >
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003,
>> > Model Suffix: (nul)
>> > DEBUG 1: Config File Default:
>> > /opt/MET7/share/met/config/TCPairsConfig_default
>> > DEBUG 1: Config File User:
>> >
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
>> > DEBUG 1: Distance to land file: /opt/MET7/share/met/tc_data/
>> > dland_global_tenth_degree.nc
>> > DEBUG 2: Processing 1 BDECK file(s).
>> > WARNING:
>> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
>> > WARNING:
>> > WARNING:
>> > WARNING: int parse_lon(const char *) -> bad longitude ... "687N"
>> > WARNING:
>> > WARNING:
>> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
>> > WARNING:
>> > WARNING:
>> > WARNING: int parse_lon(const char *) -> bad longitude ... "701N"
>> > WARNING:
>> > WARNING:
>> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
>> > WARNING:
>> > WARNING:
>> > WARNING: int parse_lon(const char *) -> bad longitude ... "711N"
>> > WARNING:
>> > WARNING:
>> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
>> > WARNING:
>> > WARNING:
>> > WARNING: int parse_lon(const char *) -> bad longitude ... "720N"
>> > WARNING:
>> > WARNING:
>> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
>> > WARNING:
>> > WARNING:
>> > WARNING: int parse_lon(const char *) -> bad longitude ... "730N"
>> > WARNING:
>> > WARNING:
>> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
>> > WARNING:
>> > WARNING:
>> > WARNING: int parse_lon(const char *) -> bad longitude ... "741N"
>> > WARNING:
>> > WARNING:
>> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
>> > WARNING:
>> > WARNING:
>> > WARNING: int parse_lon(const char *) -> bad longitude ... "751N"
>> > WARNING:
>> > WARNING:
>> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
>> > WARNING:
>> > WARNING:
>> > WARNING: int parse_lon(const char *) -> bad longitude ... "760N"
>> > WARNING:
>> > WARNING:
>> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
>> > WARNING:
>> > WARNING:
>> > WARNING: int parse_lon(const char *) -> bad longitude ... "764N"
>> > WARNING:
>> > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
>> > DEBUG 3: Identified 9 track(s).
>> > DEBUG 2: Found 9 BDECK track(s).
>> > DEBUG 2: Processing 1 ADECK file(s).
>> > DEBUG 3: Used 0 of 22 lines read from 1 file(s).
>> > DEBUG 3: Identified 0 track(s).
>> > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
>> > DEBUG 2: Finished adding 0 and replacing 0 Interp12 track(s).
>> > DEBUG 2: Deriving 0 ADECK consensus model(s).
>> > DEBUG 2: Added 0 ADECK consensus tracks(s).
>> > DEBUG 2: Deriving 0 ADECK lag model(s).
>> > DEBUG 2: Added 0 ADECK lag tracks(s).
>> > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
>> > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
>> > DEBUG 2: Filtering 0 ADECK tracks based on config file settings.
>> > DEBUG 3: Total tracks read = 0
>> > DEBUG 3: Total tracks kept = 0
>> > DEBUG 3: Rejected for storm name = 0
>> > DEBUG 3: Rejected for valid time = 0
>> > DEBUG 3: Rejected for required lead times = 0
>> > DEBUG 3: Rejected for init mask = 0
>> > DEBUG 3: Rejected for valid mask = 0
>> > DEBUG 2: Matching 0 ADECK tracks to 9 BDECK tracks.
>> > DEBUG 1: Watch/Warning file:
/opt/MET7/share/met/tc_data/wwpts_us.txt
>> > DEBUG 1: Output file:
>> >
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt.tcst
>> >
>> > --
>> > Rosalyn MacCracken
>> > Support Scientist
>> >
>> > Ocean Applications Branch
>> > NOAA/NWS Ocean Prediction Center
>> > NCWCP
>> > 5830 University Research Ct
>> > College Park, MD 20740-3818
>> >
>> > (p) 301-683-1551
>> > rosalyn.maccracken at noaa.gov
>> >
>> >
>>
>>
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Help with using TC_pairs
From: John Halley Gotway
Time: Mon Nov 26 11:14:57 2018
Roz,
I see that you're wondering about this log message:
*Model Suffix: (nul)*
Everything is fine, you do not need to be concerned about this.
Here's a
line taken from the usage statement for the tc_pairs tool:
*Note: The "-adeck", "-edeck", and "-bdeck" options may include
"suffix=string" to modify the model names from that source.*
Since you didn't specify "suffix=string" on the command line, the
suffix
remains the default value of an empty string, i.e. "(nul)". So why
would
you ever want to use a suffix? Sometimes data is messy. We were
running
tc_pairs to evaluate track data from multiple sources and we found
that
some model names we reused in both sources. Let's say the OFCL
forecast
showed up in both stream 1.0 and stream 1.5 data. Specifying a suffix
such
as "suffix=_STREAM1.5" changes the model name from "OFCL" to
"OFCL_STREAM1.5". And that enables us to distinguish between those
tracks
in the output.
But uou probably don't have any need for this option in your current
work.
However, I am concerned about these log messages:
DEBUG 4: Updating storm name from "-287" to "-305" for ML0900022018.
DEBUG 4: Updating storm name from "-305" to "-255" for ML0900022018.
DEBUG 4: Updating storm name from "-255" to "-203" for ML0900022018.
DEBUG 4: Updating storm name from "-203" to "-205" for ML0900022018.
DEBUG 4: Updating storm name from "-205" to "-217" for ML0900022018.
DEBUG 4: Updating storm name from "-217" to "-211" for ML0900022018.
DEBUG 4: Updating storm name from "-211" to "-182" for ML0900022018.
DEBUG 4: Updating storm name from "-182" to "-138" for ML0900022018.
You likely have numeric values showing up in the "STORMNAME" ATCF
column:
https://www.nrlmry.navy.mil/atcf_web/docs/database/new/abdeck.txt
I'd recommend looking into that.
Thanks,
John
On Mon, Nov 26, 2018 at 8:44 AM Rosalyn MacCracken - NOAA Affiliate
via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
>
> Hi John,
>
> I figured out an easy way to read in and write out Guang Ping's
modified
> ATCF file, so that it:
> 1) Removes the 3rd column
> 2) Replaces -99 with -9999 in column 21 and 24 (after the 3rd
column is
> removed)
> 3) Updates column 2 to include month+cyclone number. (You had said
to do
> year+cyclone #, but the tc_pairs_wrapper.py said to do
month+cyclone#)
>
> I changed Adeck and Bdeck.
> Just a couple lines of my file:
>
> ML, 090002, 2018090100, 03, GFSO, 000, 511S, 897W, 40, 976, XX,
34,
> NEQ, 0625, 0630, 0000, 0226, 978, 68, -9999, 123, 155, -9999,
-9999,
> -9999, -297, -394, -287, -458, 0073
> ML, 090002, 2018090106, 03, GFSO, 000, 537S, 854W, 44, 964, XX,
34,
> NEQ, 0640, 0645, 0000, 0297, 969, 108, -9999, 119, 147, -9999,
-9999,
> -9999, -403, -648, -305, -394, 0055
>
> So, I've run tc_pairs again and my output is below. I don't
understand the
> Model Suffix: (nul). Is the input format still wrong?
>
> Roz
>
> Output:
> /opt/MET7/bin/tc_pairs -adeck
>
>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat
> -bdeck
>
>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat
> -config
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> -out
>
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt
> -log
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> -v 4
> DEBUG 1: [Source 1 of 1] ADECK Source:
>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat,
> Model Suffix: (nul)
> DEBUG 1: [Source 1 of 1] BDECK Source:
>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat,
> Model Suffix: (nul)
> DEBUG 1: Config File Default:
> /opt/MET7/share/met/config/TCPairsConfig_default
> DEBUG 1: Config File User:
>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> DEBUG 1: Distance to land file: /opt/MET7/share/met/tc_data/
> dland_global_tenth_degree.nc
> DEBUG 4:
> DEBUG 4: Latitude/Longitude Grid Data:
> DEBUG 4: lat_ll: -90
> DEBUG 4: lon_ll: 180
> DEBUG 4: delta_lat: 0.1
> DEBUG 4: delta_lon: 0.1
> DEBUG 4: Nlat: 1801
> DEBUG 4: Nlon: 3601
> DEBUG 4:
> DEBUG 2: Processing 1 BDECK file(s).
> DEBUG 4: Updating storm name from "-287" to "-305" for ML0900022018.
> DEBUG 4: Updating storm name from "-305" to "-255" for ML0900022018.
> DEBUG 4: Updating storm name from "-255" to "-203" for ML0900022018.
> DEBUG 4: Updating storm name from "-203" to "-205" for ML0900022018.
> DEBUG 4: Updating storm name from "-205" to "-217" for ML0900022018.
> DEBUG 4: Updating storm name from "-217" to "-211" for ML0900022018.
> DEBUG 4: Updating storm name from "-211" to "-182" for ML0900022018.
> DEBUG 4: Updating storm name from "-182" to "-138" for ML0900022018.
> DEBUG 4: [File 1 of 1] Used 9 of 9 lines read from file
>
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat"
> DEBUG 3: Used 9 of 9 lines read from 1 file(s).
> DEBUG 3: Identified 1 track(s).
> DEBUG 4: [Track 1 of 1] TrackInfo: StormId = "ML0900022018", IsBest
=
> false, IsOper = false, CheckAnly = true, IsAnly = true, Basin =
"ML",
> Cyclone = "090002", StormName = "-138", TechniqueNumber = 3,
Technique =
> "GFSO", Initials = "-9999", InitTime = NA, MinValidTime =
20180901_000000,
> MaxValidTime = 20180903_000000, NPoints = 9, NAlloc = 100,
NTrackLines = 0
> DEBUG 2: Found 1 BDECK track(s).
> DEBUG 2: Processing 1 ADECK file(s).
> DEBUG 4: [File 1 of 1] Used 0 of 21 lines read from file
>
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat"
> DEBUG 3: Used 0 of 21 lines read from 1 file(s).
> DEBUG 3: Identified 0 track(s).
> DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> DEBUG 2: Finished adding 0 and replacing 0 Interp12 track(s).
> DEBUG 2: Deriving 0 ADECK consensus model(s).
> DEBUG 2: Added 0 ADECK consensus tracks(s).
> DEBUG 2: Deriving 0 ADECK lag model(s).
> DEBUG 2: Added 0 ADECK lag tracks(s).
> DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> DEBUG 2: Filtering 0 ADECK tracks based on config file settings.
> DEBUG 3: Total tracks read = 0
> DEBUG 3: Total tracks kept = 0
> DEBUG 3: Rejected for storm name = 0
> DEBUG 3: Rejected for valid time = 0
> DEBUG 3: Rejected for required lead times = 0
> DEBUG 3: Rejected for init mask = 0
> DEBUG 3: Rejected for valid mask = 0
> DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
> DEBUG 1: Watch/Warning file:
/opt/MET7/share/met/tc_data/wwpts_us.txt
> DEBUG 1: Output file:
>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt.tcst
>
> On Wed, Nov 21, 2018 at 9:16 AM Rosalyn MacCracken - NOAA Affiliate
<
> rosalyn.maccracken at noaa.gov> wrote:
>
> > Hi John,
> >
> > Yes, that makes sense, and yes, I am using Guang Pings' modified
files.
> > I'm not sure how to run tc_pairs_wrapper.py, though. I think I
need some
> > type of *.conf file to pass in the files I need, or something like
that.
> > I'm in the process of reading through the code now. I wonder if I
could
> > just pull out the function process_non_atcf, to run in a stand-
alone
> mode.
> > I wish there was some sort of documentation with this code...
> >
> > And, so, once I can read these files, I can use tc_pairs to create
the
> > forecast error statistics, and then use tc_stats to aggregate the
files
> and
> > get the mean, etc, for a period of time, correct? Was there some
sort of
> > plotting scripts for the output files?
> >
> > So, with tc_pairs, I can define a region, like I did with
point_stat,
> > correct? Can I use the same *.nc files, or do I use the original
*.poly
> > files?
> >
> > Roz
> >
> > On Tue, Nov 20, 2018 at 7:53 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Hi Roz,
> >>
> >> Yes, I see you’re using the modified ATCF format that Guang Ping
uses.
> >> You’re correct that tc_pairs doesn’t parse this directly since it
> differs
> >> slightly from the ATCF format.
> >>
> >> When we’ve processed this data in the past, we first pre-
processed it
> in a
> >> shell script to:
> >>
> >> 1. Replace the -99 value with -9999 for missing data.
> >> 2. Remove the 3rd column
> >> 3. Updating the 2nd column to include the year and cyclone
number.
> >>
> >> And that enable tc_pairs to read it.
> >>
> >> This is actually Done in python by one of the use cases in
METplus.
> >>
> >> Take a look at this file:
> >>
https://github.com/NCAR/METplus/blob/master/ush/tc_pairs_wrapper.py
> >>
> >> At the function named:
> >> process_non_atcf
> >>
> >> I haven’t run this using METplus scripts myself, but all the
pieces of
> >> logic are there, and it should be possible.
> >>
> >> Make sense?
> >>
> >> Thanks
> >> John
> >>
> >> On Tue, Nov 20, 2018 at 1:13 PM Rosalyn MacCracken - NOAA
Affiliate via
> >> RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > Tue Nov 20 13:13:05 2018: Request 87885 was acted upon.
> >> > Transaction: Ticket created by rosalyn.maccracken at noaa.gov
> >> > Queue: met_help
> >> > Subject: Help with using TC_pairs
> >> > Owner: Nobody
> >> > Requestors: rosalyn.maccracken at noaa.gov
> >> > Status: new
> >> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
> >
> >> >
> >> >
> >> > Hi,
> >> >
> >> > I'm trying to use TC_pairs for extra-tropical cyclones
(basin=ML). I
> >> think
> >> > the format of the ATCF data is the issue, but, I'm not sure. I
have
> an
> >> > Adeck and a Bdeck file, where a sample of one of the lines is:
> >> > ML, 0003, 2018090100_F000_687N_0159W_FOF, 2018090100, 03, GFSO,
000,
> >> 687N,
> >> > 159W, 32, 987, XX, 34, NEQ, 0301, 0338, 0000, 0000, 988,
52,
> -99,
> >> > 330, 86, -99, -9999, -9999, 175, 202, 137, 135, 0081
> >> >
> >> > This is slightly different from the NRL ATCF format on their
website,
> >> such
> >> > as the second column. Anyway, I configured the config file
very
> simply,
> >> > (model=GFS, basin=ML), and ran tc_pairs, and got the output to
the
> >> screen
> >> > below.
> >> >
> >> > Is there a way to work around the diferent ATCF format? I've
> attached a
> >> > sample of my Adeck and Bdeck files.
> >> >
> >> > Thanks,
> >> >
> >> > Roz
> >> >
> >> > Output to the screen:
> >> > /opt/MET7/bin/tc_pairs -adeck
> >> >
> >>
>
~/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003
> >> > -bdeck
> >> >
> >>
>
~/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003
> >> > -config
> >> >
>
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> >> > -out
> >> >
> >> >
> >>
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt
> >> > -log
> >>
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> >> > -v 3
> >> > DEBUG 1: [Source 1 of 1] ADECK Source:
> >> >
> >> >
> >>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003,
> >> > Model Suffix: (nul)
> >> > DEBUG 1: [Source 1 of 1] BDECK Source:
> >> >
> >> >
> >>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003,
> >> > Model Suffix: (nul)
> >> > DEBUG 1: Config File Default:
> >> > /opt/MET7/share/met/config/TCPairsConfig_default
> >> > DEBUG 1: Config File User:
> >> >
> >> >
> >>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> >> > DEBUG 1: Distance to land file: /opt/MET7/share/met/tc_data/
> >> > dland_global_tenth_degree.nc
> >> > DEBUG 2: Processing 1 BDECK file(s).
> >> > WARNING:
> >> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> >> > WARNING:
> >> > WARNING:
> >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"687N"
> >> > WARNING:
> >> > WARNING:
> >> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> >> > WARNING:
> >> > WARNING:
> >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"701N"
> >> > WARNING:
> >> > WARNING:
> >> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> >> > WARNING:
> >> > WARNING:
> >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"711N"
> >> > WARNING:
> >> > WARNING:
> >> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> >> > WARNING:
> >> > WARNING:
> >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"720N"
> >> > WARNING:
> >> > WARNING:
> >> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> >> > WARNING:
> >> > WARNING:
> >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"730N"
> >> > WARNING:
> >> > WARNING:
> >> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> >> > WARNING:
> >> > WARNING:
> >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"741N"
> >> > WARNING:
> >> > WARNING:
> >> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> >> > WARNING:
> >> > WARNING:
> >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"751N"
> >> > WARNING:
> >> > WARNING:
> >> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> >> > WARNING:
> >> > WARNING:
> >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"760N"
> >> > WARNING:
> >> > WARNING:
> >> > WARNING: int parse_lat(const char *) -> bad latitude ... "000"
> >> > WARNING:
> >> > WARNING:
> >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"764N"
> >> > WARNING:
> >> > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
> >> > DEBUG 3: Identified 9 track(s).
> >> > DEBUG 2: Found 9 BDECK track(s).
> >> > DEBUG 2: Processing 1 ADECK file(s).
> >> > DEBUG 3: Used 0 of 22 lines read from 1 file(s).
> >> > DEBUG 3: Identified 0 track(s).
> >> > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> >> > DEBUG 2: Finished adding 0 and replacing 0 Interp12 track(s).
> >> > DEBUG 2: Deriving 0 ADECK consensus model(s).
> >> > DEBUG 2: Added 0 ADECK consensus tracks(s).
> >> > DEBUG 2: Deriving 0 ADECK lag model(s).
> >> > DEBUG 2: Added 0 ADECK lag tracks(s).
> >> > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> >> > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> >> > DEBUG 2: Filtering 0 ADECK tracks based on config file
settings.
> >> > DEBUG 3: Total tracks read = 0
> >> > DEBUG 3: Total tracks kept = 0
> >> > DEBUG 3: Rejected for storm name = 0
> >> > DEBUG 3: Rejected for valid time = 0
> >> > DEBUG 3: Rejected for required lead times = 0
> >> > DEBUG 3: Rejected for init mask = 0
> >> > DEBUG 3: Rejected for valid mask = 0
> >> > DEBUG 2: Matching 0 ADECK tracks to 9 BDECK tracks.
> >> > DEBUG 1: Watch/Warning file:
/opt/MET7/share/met/tc_data/wwpts_us.txt
> >> > DEBUG 1: Output file:
> >> >
> >> >
> >>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt.tcst
> >> >
> >> > --
> >> > Rosalyn MacCracken
> >> > Support Scientist
> >> >
> >> > Ocean Applications Branch
> >> > NOAA/NWS Ocean Prediction Center
> >> > NCWCP
> >> > 5830 University Research Ct
> >> > College Park, MD 20740-3818
> >> >
> >> > (p) 301-683-1551
> >> > rosalyn.maccracken at noaa.gov
> >> >
> >> >
> >>
> >>
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applications Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
>
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
>
------------------------------------------------
Subject: Help with using TC_pairs
From: Rosalyn MacCracken - NOAA Affiliate
Time: Mon Nov 26 11:50:33 2018
Hi John,
Ok, I will ignore the suffix error. I do have numeric values in
STORMNAME. I wonder if in the modified files I have, this column is
not
actually STORMNAME, but, some other variable. Now, I don't have the
number
of columns that the NRL website has. Will that be a problem? Should
I
create dummy columns?
Roz
On Mon, Nov 26, 2018 at 1:14 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Roz,
>
> I see that you're wondering about this log message:
> *Model Suffix: (nul)*
>
> Everything is fine, you do not need to be concerned about this.
Here's a
> line taken from the usage statement for the tc_pairs tool:
> *Note: The "-adeck", "-edeck", and "-bdeck" options may include
> "suffix=string" to modify the model names from that source.*
>
> Since you didn't specify "suffix=string" on the command line, the
suffix
> remains the default value of an empty string, i.e. "(nul)". So why
would
> you ever want to use a suffix? Sometimes data is messy. We were
running
> tc_pairs to evaluate track data from multiple sources and we found
that
> some model names we reused in both sources. Let's say the OFCL
forecast
> showed up in both stream 1.0 and stream 1.5 data. Specifying a
suffix such
> as "suffix=_STREAM1.5" changes the model name from "OFCL" to
> "OFCL_STREAM1.5". And that enables us to distinguish between those
tracks
> in the output.
>
> But uou probably don't have any need for this option in your current
work.
>
> However, I am concerned about these log messages:
> DEBUG 4: Updating storm name from "-287" to "-305" for ML0900022018.
> DEBUG 4: Updating storm name from "-305" to "-255" for ML0900022018.
> DEBUG 4: Updating storm name from "-255" to "-203" for ML0900022018.
> DEBUG 4: Updating storm name from "-203" to "-205" for ML0900022018.
> DEBUG 4: Updating storm name from "-205" to "-217" for ML0900022018.
> DEBUG 4: Updating storm name from "-217" to "-211" for ML0900022018.
> DEBUG 4: Updating storm name from "-211" to "-182" for ML0900022018.
> DEBUG 4: Updating storm name from "-182" to "-138" for ML0900022018.
>
> You likely have numeric values showing up in the "STORMNAME" ATCF
column:
> https://www.nrlmry.navy.mil/atcf_web/docs/database/new/abdeck.txt
>
> I'd recommend looking into that.
>
> Thanks,
> John
>
> On Mon, Nov 26, 2018 at 8:44 AM Rosalyn MacCracken - NOAA Affiliate
via RT
> <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
> >
> > Hi John,
> >
> > I figured out an easy way to read in and write out Guang Ping's
modified
> > ATCF file, so that it:
> > 1) Removes the 3rd column
> > 2) Replaces -99 with -9999 in column 21 and 24 (after the 3rd
column is
> > removed)
> > 3) Updates column 2 to include month+cyclone number. (You had
said to
> do
> > year+cyclone #, but the tc_pairs_wrapper.py said to do
month+cyclone#)
> >
> > I changed Adeck and Bdeck.
> > Just a couple lines of my file:
> >
> > ML, 090002, 2018090100, 03, GFSO, 000, 511S, 897W, 40, 976, XX,
34,
> > NEQ, 0625, 0630, 0000, 0226, 978, 68, -9999, 123, 155, -9999,
-9999,
> > -9999, -297, -394, -287, -458, 0073
> > ML, 090002, 2018090106, 03, GFSO, 000, 537S, 854W, 44, 964, XX,
34,
> > NEQ, 0640, 0645, 0000, 0297, 969, 108, -9999, 119, 147, -9999,
-9999,
> > -9999, -403, -648, -305, -394, 0055
> >
> > So, I've run tc_pairs again and my output is below. I don't
understand
> the
> > Model Suffix: (nul). Is the input format still wrong?
> >
> > Roz
> >
> > Output:
> > /opt/MET7/bin/tc_pairs -adeck
> >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat
> > -bdeck
> >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat
> > -config
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > -out
> >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt
> > -log
>
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> > -v 4
> > DEBUG 1: [Source 1 of 1] ADECK Source:
> >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat,
> > Model Suffix: (nul)
> > DEBUG 1: [Source 1 of 1] BDECK Source:
> >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat,
> > Model Suffix: (nul)
> > DEBUG 1: Config File Default:
> > /opt/MET7/share/met/config/TCPairsConfig_default
> > DEBUG 1: Config File User:
> >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > DEBUG 1: Distance to land file: /opt/MET7/share/met/tc_data/
> > dland_global_tenth_degree.nc
> > DEBUG 4:
> > DEBUG 4: Latitude/Longitude Grid Data:
> > DEBUG 4: lat_ll: -90
> > DEBUG 4: lon_ll: 180
> > DEBUG 4: delta_lat: 0.1
> > DEBUG 4: delta_lon: 0.1
> > DEBUG 4: Nlat: 1801
> > DEBUG 4: Nlon: 3601
> > DEBUG 4:
> > DEBUG 2: Processing 1 BDECK file(s).
> > DEBUG 4: Updating storm name from "-287" to "-305" for
ML0900022018.
> > DEBUG 4: Updating storm name from "-305" to "-255" for
ML0900022018.
> > DEBUG 4: Updating storm name from "-255" to "-203" for
ML0900022018.
> > DEBUG 4: Updating storm name from "-203" to "-205" for
ML0900022018.
> > DEBUG 4: Updating storm name from "-205" to "-217" for
ML0900022018.
> > DEBUG 4: Updating storm name from "-217" to "-211" for
ML0900022018.
> > DEBUG 4: Updating storm name from "-211" to "-182" for
ML0900022018.
> > DEBUG 4: Updating storm name from "-182" to "-138" for
ML0900022018.
> > DEBUG 4: [File 1 of 1] Used 9 of 9 lines read from file
> >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat"
> > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
> > DEBUG 3: Identified 1 track(s).
> > DEBUG 4: [Track 1 of 1] TrackInfo: StormId = "ML0900022018",
IsBest =
> > false, IsOper = false, CheckAnly = true, IsAnly = true, Basin =
"ML",
> > Cyclone = "090002", StormName = "-138", TechniqueNumber = 3,
Technique =
> > "GFSO", Initials = "-9999", InitTime = NA, MinValidTime =
> 20180901_000000,
> > MaxValidTime = 20180903_000000, NPoints = 9, NAlloc = 100,
NTrackLines =
> 0
> > DEBUG 2: Found 1 BDECK track(s).
> > DEBUG 2: Processing 1 ADECK file(s).
> > DEBUG 4: [File 1 of 1] Used 0 of 21 lines read from file
> >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat"
> > DEBUG 3: Used 0 of 21 lines read from 1 file(s).
> > DEBUG 3: Identified 0 track(s).
> > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> > DEBUG 2: Finished adding 0 and replacing 0 Interp12 track(s).
> > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > DEBUG 2: Deriving 0 ADECK lag model(s).
> > DEBUG 2: Added 0 ADECK lag tracks(s).
> > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> > DEBUG 2: Filtering 0 ADECK tracks based on config file settings.
> > DEBUG 3: Total tracks read = 0
> > DEBUG 3: Total tracks kept = 0
> > DEBUG 3: Rejected for storm name = 0
> > DEBUG 3: Rejected for valid time = 0
> > DEBUG 3: Rejected for required lead times = 0
> > DEBUG 3: Rejected for init mask = 0
> > DEBUG 3: Rejected for valid mask = 0
> > DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
> > DEBUG 1: Watch/Warning file:
/opt/MET7/share/met/tc_data/wwpts_us.txt
> > DEBUG 1: Output file:
> >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt.tcst
> >
> > On Wed, Nov 21, 2018 at 9:16 AM Rosalyn MacCracken - NOAA
Affiliate <
> > rosalyn.maccracken at noaa.gov> wrote:
> >
> > > Hi John,
> > >
> > > Yes, that makes sense, and yes, I am using Guang Pings' modified
files.
> > > I'm not sure how to run tc_pairs_wrapper.py, though. I think I
need
> some
> > > type of *.conf file to pass in the files I need, or something
like
> that.
> > > I'm in the process of reading through the code now. I wonder if
I
> could
> > > just pull out the function process_non_atcf, to run in a stand-
alone
> > mode.
> > > I wish there was some sort of documentation with this code...
> > >
> > > And, so, once I can read these files, I can use tc_pairs to
create the
> > > forecast error statistics, and then use tc_stats to aggregate
the files
> > and
> > > get the mean, etc, for a period of time, correct? Was there
some sort
> of
> > > plotting scripts for the output files?
> > >
> > > So, with tc_pairs, I can define a region, like I did with
point_stat,
> > > correct? Can I use the same *.nc files, or do I use the
original
> *.poly
> > > files?
> > >
> > > Roz
> > >
> > > On Tue, Nov 20, 2018 at 7:53 PM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >> Hi Roz,
> > >>
> > >> Yes, I see you’re using the modified ATCF format that Guang
Ping uses.
> > >> You’re correct that tc_pairs doesn’t parse this directly since
it
> > differs
> > >> slightly from the ATCF format.
> > >>
> > >> When we’ve processed this data in the past, we first pre-
processed it
> > in a
> > >> shell script to:
> > >>
> > >> 1. Replace the -99 value with -9999 for missing data.
> > >> 2. Remove the 3rd column
> > >> 3. Updating the 2nd column to include the year and cyclone
number.
> > >>
> > >> And that enable tc_pairs to read it.
> > >>
> > >> This is actually Done in python by one of the use cases in
METplus.
> > >>
> > >> Take a look at this file:
> > >>
https://github.com/NCAR/METplus/blob/master/ush/tc_pairs_wrapper.py
> > >>
> > >> At the function named:
> > >> process_non_atcf
> > >>
> > >> I haven’t run this using METplus scripts myself, but all the
pieces of
> > >> logic are there, and it should be possible.
> > >>
> > >> Make sense?
> > >>
> > >> Thanks
> > >> John
> > >>
> > >> On Tue, Nov 20, 2018 at 1:13 PM Rosalyn MacCracken - NOAA
Affiliate
> via
> > >> RT <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >> >
> > >> > Tue Nov 20 13:13:05 2018: Request 87885 was acted upon.
> > >> > Transaction: Ticket created by rosalyn.maccracken at noaa.gov
> > >> > Queue: met_help
> > >> > Subject: Help with using TC_pairs
> > >> > Owner: Nobody
> > >> > Requestors: rosalyn.maccracken at noaa.gov
> > >> > Status: new
> > >> > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
> > >
> > >> >
> > >> >
> > >> > Hi,
> > >> >
> > >> > I'm trying to use TC_pairs for extra-tropical cyclones
(basin=ML).
> I
> > >> think
> > >> > the format of the ATCF data is the issue, but, I'm not sure.
I have
> > an
> > >> > Adeck and a Bdeck file, where a sample of one of the lines
is:
> > >> > ML, 0003, 2018090100_F000_687N_0159W_FOF, 2018090100, 03,
GFSO, 000,
> > >> 687N,
> > >> > 159W, 32, 987, XX, 34, NEQ, 0301, 0338, 0000, 0000, 988,
52,
> > -99,
> > >> > 330, 86, -99, -9999, -9999, 175, 202, 137, 135, 0081
> > >> >
> > >> > This is slightly different from the NRL ATCF format on their
> website,
> > >> such
> > >> > as the second column. Anyway, I configured the config file
very
> > simply,
> > >> > (model=GFS, basin=ML), and ran tc_pairs, and got the output
to the
> > >> screen
> > >> > below.
> > >> >
> > >> > Is there a way to work around the diferent ATCF format? I've
> > attached a
> > >> > sample of my Adeck and Bdeck files.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Roz
> > >> >
> > >> > Output to the screen:
> > >> > /opt/MET7/bin/tc_pairs -adeck
> > >> >
> > >>
> >
>
~/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003
> > >> > -bdeck
> > >> >
> > >>
> >
>
~/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003
> > >> > -config
> > >> >
> >
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> > >> > -out
> > >> >
> > >> >
> > >>
> >
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt
> > >> > -log
> > >>
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> > >> > -v 3
> > >> > DEBUG 1: [Source 1 of 1] ADECK Source:
> > >> >
> > >> >
> > >>
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003,
> > >> > Model Suffix: (nul)
> > >> > DEBUG 1: [Source 1 of 1] BDECK Source:
> > >> >
> > >> >
> > >>
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003,
> > >> > Model Suffix: (nul)
> > >> > DEBUG 1: Config File Default:
> > >> > /opt/MET7/share/met/config/TCPairsConfig_default
> > >> > DEBUG 1: Config File User:
> > >> >
> > >> >
> > >>
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> > >> > DEBUG 1: Distance to land file: /opt/MET7/share/met/tc_data/
> > >> > dland_global_tenth_degree.nc
> > >> > DEBUG 2: Processing 1 BDECK file(s).
> > >> > WARNING:
> > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > >> > WARNING:
> > >> > WARNING:
> > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"687N"
> > >> > WARNING:
> > >> > WARNING:
> > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > >> > WARNING:
> > >> > WARNING:
> > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"701N"
> > >> > WARNING:
> > >> > WARNING:
> > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > >> > WARNING:
> > >> > WARNING:
> > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"711N"
> > >> > WARNING:
> > >> > WARNING:
> > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > >> > WARNING:
> > >> > WARNING:
> > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"720N"
> > >> > WARNING:
> > >> > WARNING:
> > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > >> > WARNING:
> > >> > WARNING:
> > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"730N"
> > >> > WARNING:
> > >> > WARNING:
> > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > >> > WARNING:
> > >> > WARNING:
> > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"741N"
> > >> > WARNING:
> > >> > WARNING:
> > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > >> > WARNING:
> > >> > WARNING:
> > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"751N"
> > >> > WARNING:
> > >> > WARNING:
> > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > >> > WARNING:
> > >> > WARNING:
> > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"760N"
> > >> > WARNING:
> > >> > WARNING:
> > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > >> > WARNING:
> > >> > WARNING:
> > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"764N"
> > >> > WARNING:
> > >> > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
> > >> > DEBUG 3: Identified 9 track(s).
> > >> > DEBUG 2: Found 9 BDECK track(s).
> > >> > DEBUG 2: Processing 1 ADECK file(s).
> > >> > DEBUG 3: Used 0 of 22 lines read from 1 file(s).
> > >> > DEBUG 3: Identified 0 track(s).
> > >> > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> > >> > DEBUG 2: Finished adding 0 and replacing 0 Interp12 track(s).
> > >> > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > >> > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > >> > DEBUG 2: Deriving 0 ADECK lag model(s).
> > >> > DEBUG 2: Added 0 ADECK lag tracks(s).
> > >> > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> > >> > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> > >> > DEBUG 2: Filtering 0 ADECK tracks based on config file
settings.
> > >> > DEBUG 3: Total tracks read = 0
> > >> > DEBUG 3: Total tracks kept = 0
> > >> > DEBUG 3: Rejected for storm name = 0
> > >> > DEBUG 3: Rejected for valid time = 0
> > >> > DEBUG 3: Rejected for required lead times = 0
> > >> > DEBUG 3: Rejected for init mask = 0
> > >> > DEBUG 3: Rejected for valid mask = 0
> > >> > DEBUG 2: Matching 0 ADECK tracks to 9 BDECK tracks.
> > >> > DEBUG 1: Watch/Warning file:
> /opt/MET7/share/met/tc_data/wwpts_us.txt
> > >> > DEBUG 1: Output file:
> > >> >
> > >> >
> > >>
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt.tcst
> > >> >
> > >> > --
> > >> > Rosalyn MacCracken
> > >> > Support Scientist
> > >> >
> > >> > Ocean Applications Branch
> > >> > NOAA/NWS Ocean Prediction Center
> > >> > NCWCP
> > >> > 5830 University Research Ct
> > >> > College Park, MD 20740-3818
> > >> >
> > >> > (p) 301-683-1551
> > >> > rosalyn.maccracken at noaa.gov
> > >> >
> > >> >
> > >>
> > >>
> > >
> > > --
> > > Rosalyn MacCracken
> > > Support Scientist
> > >
> > > Ocean Applications Branch
> > > NOAA/NWS Ocean Prediction Center
> > > NCWCP
> > > 5830 University Research Ct
> > > College Park, MD 20740-3818
> > >
> > > (p) 301-683-1551
> > > rosalyn.maccracken at noaa.gov
> > >
> >
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applications Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
> >
>
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Help with using TC_pairs
From: John Halley Gotway
Time: Mon Nov 26 12:06:36 2018
Roz,
Page 334 of the user's guide (
https://dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v8.0.pdf)
mentions that only the first 8 ATCF columns are required. If
additional
columns are present in the input ATCF files, then tc_pairs will
interpret
them as specified by the ATCF format.
So if the storm name column is present, MET in interpret those values
as
the storm name.
For Guang Ping's data, probably best to have that column empty.
John
On Mon, Nov 26, 2018 at 11:51 AM Rosalyn MacCracken - NOAA Affiliate
via RT
<met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
>
> Hi John,
>
> Ok, I will ignore the suffix error. I do have numeric values in
> STORMNAME. I wonder if in the modified files I have, this column is
not
> actually STORMNAME, but, some other variable. Now, I don't have the
number
> of columns that the NRL website has. Will that be a problem?
Should I
> create dummy columns?
>
> Roz
>
> On Mon, Nov 26, 2018 at 1:14 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Roz,
> >
> > I see that you're wondering about this log message:
> > *Model Suffix: (nul)*
> >
> > Everything is fine, you do not need to be concerned about this.
Here's a
> > line taken from the usage statement for the tc_pairs tool:
> > *Note: The "-adeck", "-edeck", and "-bdeck" options may include
> > "suffix=string" to modify the model names from that source.*
> >
> > Since you didn't specify "suffix=string" on the command line, the
suffix
> > remains the default value of an empty string, i.e. "(nul)". So
why would
> > you ever want to use a suffix? Sometimes data is messy. We were
running
> > tc_pairs to evaluate track data from multiple sources and we found
that
> > some model names we reused in both sources. Let's say the OFCL
forecast
> > showed up in both stream 1.0 and stream 1.5 data. Specifying a
suffix
> such
> > as "suffix=_STREAM1.5" changes the model name from "OFCL" to
> > "OFCL_STREAM1.5". And that enables us to distinguish between
those
> tracks
> > in the output.
> >
> > But uou probably don't have any need for this option in your
current
> work.
> >
> > However, I am concerned about these log messages:
> > DEBUG 4: Updating storm name from "-287" to "-305" for
ML0900022018.
> > DEBUG 4: Updating storm name from "-305" to "-255" for
ML0900022018.
> > DEBUG 4: Updating storm name from "-255" to "-203" for
ML0900022018.
> > DEBUG 4: Updating storm name from "-203" to "-205" for
ML0900022018.
> > DEBUG 4: Updating storm name from "-205" to "-217" for
ML0900022018.
> > DEBUG 4: Updating storm name from "-217" to "-211" for
ML0900022018.
> > DEBUG 4: Updating storm name from "-211" to "-182" for
ML0900022018.
> > DEBUG 4: Updating storm name from "-182" to "-138" for
ML0900022018.
> >
> > You likely have numeric values showing up in the "STORMNAME" ATCF
column:
> >
https://www.nrlmry.navy.mil/atcf_web/docs/database/new/abdeck.txt
> >
> > I'd recommend looking into that.
> >
> > Thanks,
> > John
> >
> > On Mon, Nov 26, 2018 at 8:44 AM Rosalyn MacCracken - NOAA
Affiliate via
> RT
> > <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
> > >
> > > Hi John,
> > >
> > > I figured out an easy way to read in and write out Guang Ping's
> modified
> > > ATCF file, so that it:
> > > 1) Removes the 3rd column
> > > 2) Replaces -99 with -9999 in column 21 and 24 (after the 3rd
column
> is
> > > removed)
> > > 3) Updates column 2 to include month+cyclone number. (You had
said to
> > do
> > > year+cyclone #, but the tc_pairs_wrapper.py said to do
month+cyclone#)
> > >
> > > I changed Adeck and Bdeck.
> > > Just a couple lines of my file:
> > >
> > > ML, 090002, 2018090100, 03, GFSO, 000, 511S, 897W, 40, 976,
XX, 34,
> > > NEQ, 0625, 0630, 0000, 0226, 978, 68, -9999, 123, 155, -9999,
-9999,
> > > -9999, -297, -394, -287, -458, 0073
> > > ML, 090002, 2018090106, 03, GFSO, 000, 537S, 854W, 44, 964,
XX, 34,
> > > NEQ, 0640, 0645, 0000, 0297, 969, 108, -9999, 119, 147, -9999,
-9999,
> > > -9999, -403, -648, -305, -394, 0055
> > >
> > > So, I've run tc_pairs again and my output is below. I don't
understand
> > the
> > > Model Suffix: (nul). Is the input format still wrong?
> > >
> > > Roz
> > >
> > > Output:
> > > /opt/MET7/bin/tc_pairs -adeck
> > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat
> > > -bdeck
> > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat
> > > -config
> ~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > > -out
> > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt
> > > -log
> >
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> > > -v 4
> > > DEBUG 1: [Source 1 of 1] ADECK Source:
> > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat,
> > > Model Suffix: (nul)
> > > DEBUG 1: [Source 1 of 1] BDECK Source:
> > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat,
> > > Model Suffix: (nul)
> > > DEBUG 1: Config File Default:
> > > /opt/MET7/share/met/config/TCPairsConfig_default
> > > DEBUG 1: Config File User:
> > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > > DEBUG 1: Distance to land file: /opt/MET7/share/met/tc_data/
> > > dland_global_tenth_degree.nc
> > > DEBUG 4:
> > > DEBUG 4: Latitude/Longitude Grid Data:
> > > DEBUG 4: lat_ll: -90
> > > DEBUG 4: lon_ll: 180
> > > DEBUG 4: delta_lat: 0.1
> > > DEBUG 4: delta_lon: 0.1
> > > DEBUG 4: Nlat: 1801
> > > DEBUG 4: Nlon: 3601
> > > DEBUG 4:
> > > DEBUG 2: Processing 1 BDECK file(s).
> > > DEBUG 4: Updating storm name from "-287" to "-305" for
ML0900022018.
> > > DEBUG 4: Updating storm name from "-305" to "-255" for
ML0900022018.
> > > DEBUG 4: Updating storm name from "-255" to "-203" for
ML0900022018.
> > > DEBUG 4: Updating storm name from "-203" to "-205" for
ML0900022018.
> > > DEBUG 4: Updating storm name from "-205" to "-217" for
ML0900022018.
> > > DEBUG 4: Updating storm name from "-217" to "-211" for
ML0900022018.
> > > DEBUG 4: Updating storm name from "-211" to "-182" for
ML0900022018.
> > > DEBUG 4: Updating storm name from "-182" to "-138" for
ML0900022018.
> > > DEBUG 4: [File 1 of 1] Used 9 of 9 lines read from file
> > >
> > >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat"
> > > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
> > > DEBUG 3: Identified 1 track(s).
> > > DEBUG 4: [Track 1 of 1] TrackInfo: StormId = "ML0900022018",
IsBest =
> > > false, IsOper = false, CheckAnly = true, IsAnly = true, Basin =
"ML",
> > > Cyclone = "090002", StormName = "-138", TechniqueNumber = 3,
Technique
> =
> > > "GFSO", Initials = "-9999", InitTime = NA, MinValidTime =
> > 20180901_000000,
> > > MaxValidTime = 20180903_000000, NPoints = 9, NAlloc = 100,
NTrackLines
> =
> > 0
> > > DEBUG 2: Found 1 BDECK track(s).
> > > DEBUG 2: Processing 1 ADECK file(s).
> > > DEBUG 4: [File 1 of 1] Used 0 of 21 lines read from file
> > >
> > >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat"
> > > DEBUG 3: Used 0 of 21 lines read from 1 file(s).
> > > DEBUG 3: Identified 0 track(s).
> > > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> > > DEBUG 2: Finished adding 0 and replacing 0 Interp12 track(s).
> > > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > > DEBUG 2: Deriving 0 ADECK lag model(s).
> > > DEBUG 2: Added 0 ADECK lag tracks(s).
> > > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> > > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> > > DEBUG 2: Filtering 0 ADECK tracks based on config file settings.
> > > DEBUG 3: Total tracks read = 0
> > > DEBUG 3: Total tracks kept = 0
> > > DEBUG 3: Rejected for storm name = 0
> > > DEBUG 3: Rejected for valid time = 0
> > > DEBUG 3: Rejected for required lead times = 0
> > > DEBUG 3: Rejected for init mask = 0
> > > DEBUG 3: Rejected for valid mask = 0
> > > DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
> > > DEBUG 1: Watch/Warning file:
/opt/MET7/share/met/tc_data/wwpts_us.txt
> > > DEBUG 1: Output file:
> > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt.tcst
> > >
> > > On Wed, Nov 21, 2018 at 9:16 AM Rosalyn MacCracken - NOAA
Affiliate <
> > > rosalyn.maccracken at noaa.gov> wrote:
> > >
> > > > Hi John,
> > > >
> > > > Yes, that makes sense, and yes, I am using Guang Pings'
modified
> files.
> > > > I'm not sure how to run tc_pairs_wrapper.py, though. I think
I need
> > some
> > > > type of *.conf file to pass in the files I need, or something
like
> > that.
> > > > I'm in the process of reading through the code now. I wonder
if I
> > could
> > > > just pull out the function process_non_atcf, to run in a
stand-alone
> > > mode.
> > > > I wish there was some sort of documentation with this code...
> > > >
> > > > And, so, once I can read these files, I can use tc_pairs to
create
> the
> > > > forecast error statistics, and then use tc_stats to aggregate
the
> files
> > > and
> > > > get the mean, etc, for a period of time, correct? Was there
some
> sort
> > of
> > > > plotting scripts for the output files?
> > > >
> > > > So, with tc_pairs, I can define a region, like I did with
point_stat,
> > > > correct? Can I use the same *.nc files, or do I use the
original
> > *.poly
> > > > files?
> > > >
> > > > Roz
> > > >
> > > > On Tue, Nov 20, 2018 at 7:53 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >> Hi Roz,
> > > >>
> > > >> Yes, I see you’re using the modified ATCF format that Guang
Ping
> uses.
> > > >> You’re correct that tc_pairs doesn’t parse this directly
since it
> > > differs
> > > >> slightly from the ATCF format.
> > > >>
> > > >> When we’ve processed this data in the past, we first pre-
processed
> it
> > > in a
> > > >> shell script to:
> > > >>
> > > >> 1. Replace the -99 value with -9999 for missing data.
> > > >> 2. Remove the 3rd column
> > > >> 3. Updating the 2nd column to include the year and cyclone
number.
> > > >>
> > > >> And that enable tc_pairs to read it.
> > > >>
> > > >> This is actually Done in python by one of the use cases in
METplus.
> > > >>
> > > >> Take a look at this file:
> > > >>
https://github.com/NCAR/METplus/blob/master/ush/tc_pairs_wrapper.py
> > > >>
> > > >> At the function named:
> > > >> process_non_atcf
> > > >>
> > > >> I haven’t run this using METplus scripts myself, but all the
pieces
> of
> > > >> logic are there, and it should be possible.
> > > >>
> > > >> Make sense?
> > > >>
> > > >> Thanks
> > > >> John
> > > >>
> > > >> On Tue, Nov 20, 2018 at 1:13 PM Rosalyn MacCracken - NOAA
Affiliate
> > via
> > > >> RT <
> > > >> met_help at ucar.edu> wrote:
> > > >>
> > > >> >
> > > >> > Tue Nov 20 13:13:05 2018: Request 87885 was acted upon.
> > > >> > Transaction: Ticket created by rosalyn.maccracken at noaa.gov
> > > >> > Queue: met_help
> > > >> > Subject: Help with using TC_pairs
> > > >> > Owner: Nobody
> > > >> > Requestors: rosalyn.maccracken at noaa.gov
> > > >> > Status: new
> > > >> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
> > > >
> > > >> >
> > > >> >
> > > >> > Hi,
> > > >> >
> > > >> > I'm trying to use TC_pairs for extra-tropical cyclones
(basin=ML).
> > I
> > > >> think
> > > >> > the format of the ATCF data is the issue, but, I'm not
sure. I
> have
> > > an
> > > >> > Adeck and a Bdeck file, where a sample of one of the lines
is:
> > > >> > ML, 0003, 2018090100_F000_687N_0159W_FOF, 2018090100, 03,
GFSO,
> 000,
> > > >> 687N,
> > > >> > 159W, 32, 987, XX, 34, NEQ, 0301, 0338, 0000, 0000,
988, 52,
> > > -99,
> > > >> > 330, 86, -99, -9999, -9999, 175, 202, 137, 135, 0081
> > > >> >
> > > >> > This is slightly different from the NRL ATCF format on
their
> > website,
> > > >> such
> > > >> > as the second column. Anyway, I configured the config file
very
> > > simply,
> > > >> > (model=GFS, basin=ML), and ran tc_pairs, and got the output
to the
> > > >> screen
> > > >> > below.
> > > >> >
> > > >> > Is there a way to work around the diferent ATCF format?
I've
> > > attached a
> > > >> > sample of my Adeck and Bdeck files.
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > Roz
> > > >> >
> > > >> > Output to the screen:
> > > >> > /opt/MET7/bin/tc_pairs -adeck
> > > >> >
> > > >>
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003
> > > >> > -bdeck
> > > >> >
> > > >>
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003
> > > >> > -config
> > > >> >
> > >
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> > > >> > -out
> > > >> >
> > > >> >
> > > >>
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt
> > > >> > -log
> > > >>
>
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> > > >> > -v 3
> > > >> > DEBUG 1: [Source 1 of 1] ADECK Source:
> > > >> >
> > > >> >
> > > >>
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003,
> > > >> > Model Suffix: (nul)
> > > >> > DEBUG 1: [Source 1 of 1] BDECK Source:
> > > >> >
> > > >> >
> > > >>
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003,
> > > >> > Model Suffix: (nul)
> > > >> > DEBUG 1: Config File Default:
> > > >> > /opt/MET7/share/met/config/TCPairsConfig_default
> > > >> > DEBUG 1: Config File User:
> > > >> >
> > > >> >
> > > >>
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> > > >> > DEBUG 1: Distance to land file:
/opt/MET7/share/met/tc_data/
> > > >> > dland_global_tenth_degree.nc
> > > >> > DEBUG 2: Processing 1 BDECK file(s).
> > > >> > WARNING:
> > > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > > >> > WARNING:
> > > >> > WARNING:
> > > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"687N"
> > > >> > WARNING:
> > > >> > WARNING:
> > > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > > >> > WARNING:
> > > >> > WARNING:
> > > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"701N"
> > > >> > WARNING:
> > > >> > WARNING:
> > > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > > >> > WARNING:
> > > >> > WARNING:
> > > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"711N"
> > > >> > WARNING:
> > > >> > WARNING:
> > > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > > >> > WARNING:
> > > >> > WARNING:
> > > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"720N"
> > > >> > WARNING:
> > > >> > WARNING:
> > > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > > >> > WARNING:
> > > >> > WARNING:
> > > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"730N"
> > > >> > WARNING:
> > > >> > WARNING:
> > > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > > >> > WARNING:
> > > >> > WARNING:
> > > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"741N"
> > > >> > WARNING:
> > > >> > WARNING:
> > > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > > >> > WARNING:
> > > >> > WARNING:
> > > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"751N"
> > > >> > WARNING:
> > > >> > WARNING:
> > > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > > >> > WARNING:
> > > >> > WARNING:
> > > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"760N"
> > > >> > WARNING:
> > > >> > WARNING:
> > > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > > >> > WARNING:
> > > >> > WARNING:
> > > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"764N"
> > > >> > WARNING:
> > > >> > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
> > > >> > DEBUG 3: Identified 9 track(s).
> > > >> > DEBUG 2: Found 9 BDECK track(s).
> > > >> > DEBUG 2: Processing 1 ADECK file(s).
> > > >> > DEBUG 3: Used 0 of 22 lines read from 1 file(s).
> > > >> > DEBUG 3: Identified 0 track(s).
> > > >> > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> > > >> > DEBUG 2: Finished adding 0 and replacing 0 Interp12
track(s).
> > > >> > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > > >> > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > > >> > DEBUG 2: Deriving 0 ADECK lag model(s).
> > > >> > DEBUG 2: Added 0 ADECK lag tracks(s).
> > > >> > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> > > >> > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> > > >> > DEBUG 2: Filtering 0 ADECK tracks based on config file
settings.
> > > >> > DEBUG 3: Total tracks read = 0
> > > >> > DEBUG 3: Total tracks kept = 0
> > > >> > DEBUG 3: Rejected for storm name = 0
> > > >> > DEBUG 3: Rejected for valid time = 0
> > > >> > DEBUG 3: Rejected for required lead times = 0
> > > >> > DEBUG 3: Rejected for init mask = 0
> > > >> > DEBUG 3: Rejected for valid mask = 0
> > > >> > DEBUG 2: Matching 0 ADECK tracks to 9 BDECK tracks.
> > > >> > DEBUG 1: Watch/Warning file:
> > /opt/MET7/share/met/tc_data/wwpts_us.txt
> > > >> > DEBUG 1: Output file:
> > > >> >
> > > >> >
> > > >>
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt.tcst
> > > >> >
> > > >> > --
> > > >> > Rosalyn MacCracken
> > > >> > Support Scientist
> > > >> >
> > > >> > Ocean Applications Branch
> > > >> > NOAA/NWS Ocean Prediction Center
> > > >> > NCWCP
> > > >> > 5830 University Research Ct
> > > >> > College Park, MD 20740-3818
> > > >> >
> > > >> > (p) 301-683-1551
> > > >> > rosalyn.maccracken at noaa.gov
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >
> > > > --
> > > > Rosalyn MacCracken
> > > > Support Scientist
> > > >
> > > > Ocean Applications Branch
> > > > NOAA/NWS Ocean Prediction Center
> > > > NCWCP
> > > > 5830 University Research Ct
> > > > College Park, MD 20740-3818
> > > >
> > > > (p) 301-683-1551
> > > > rosalyn.maccracken at noaa.gov
> > > >
> > >
> > >
> > > --
> > > Rosalyn MacCracken
> > > Support Scientist
> > >
> > > Ocean Applications Branch
> > > NOAA/NWS Ocean Prediction Center
> > > NCWCP
> > > 5830 University Research Ct
> > > College Park, MD 20740-3818
> > >
> > > (p) 301-683-1551
> > > rosalyn.maccracken at noaa.gov
> > >
> > >
> >
> >
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
>
------------------------------------------------
Subject: Help with using TC_pairs
From: Rosalyn MacCracken - NOAA Affiliate
Time: Mon Nov 26 12:22:30 2018
So, can I just have 8 columns for the input file, or, do I need to
make
columns of -9999 or NaN for all the rest of the columns?
I experimented, and just wrote out 10 columns for the input file:
agfso_trackdf_mod =
agfso_trackdf[['basin_name','storm_id','model_start_date','const','model','fcst_lead_time','vort_lat_cent','vort_lon_cent','vort_10m_max',
'vort_min_mslp','type','thresh_wspd']]
because I wanted stats for these variables, and I got the output
below.
So, maybe it's working, but, the config file needs to be different?
Any
other ideas of what I should try doing/looking at?
Roz
Output:
DEBUG 1: [Source 1 of 1] ADECK Source:
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018083118.gfso.0053_modified.dat,
Model Suffix: _EXP
DEBUG 1: [Source 1 of 1] BDECK Source:
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018083118.gfso.0053_modified.dat,
Model Suffix: (nul)
DEBUG 1: Config File Default:
/opt/MET7/share/met/config/TCPairsConfig_default
DEBUG 1: Config File User:
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
DEBUG 1: Distance to land file: /opt/MET7/share/met/tc_data/
dland_global_tenth_degree.nc
DEBUG 4:
DEBUG 4: Latitude/Longitude Grid Data:
DEBUG 4: lat_ll: -90
DEBUG 4: lon_ll: 180
DEBUG 4: delta_lat: 0.1
DEBUG 4: delta_lon: 0.1
DEBUG 4: Nlat: 1801
DEBUG 4: Nlon: 3601
DEBUG 4:
DEBUG 2: Processing 1 BDECK file(s).
DEBUG 4: [File 1 of 1] Used 30 of 30 lines read from file
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018083118.gfso.0053_modified.dat"
DEBUG 3: Used 30 of 30 lines read from 1 file(s).
DEBUG 3: Identified 1 track(s).
DEBUG 4: [Track 1 of 1] TrackInfo: StormId = "ML0800532018", IsBest =
false, IsOper = false, CheckAnly = true, IsAnly = true, Basin = "ML",
Cyclone = "080053", StormName = "", TechniqueNumber = 3, Technique =
"GFSO", Initials = "", InitTime = NA, MinValidTime = 20180805_060000,
MaxValidTime = 20180812_120000, NPoints = 30, NAlloc = 100,
NTrackLines = 0
DEBUG 2: Found 1 BDECK track(s).
DEBUG 2: Processing 1 ADECK file(s).
DEBUG 4: [File 1 of 1] Used 0 of 642 lines read from file
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018083118.gfso.0053_modified.dat"
DEBUG 3: Used 0 of 642 lines read from 1 file(s).
DEBUG 3: Identified 0 track(s).
DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
DEBUG 2: Finished adding 0 and replacing 0 Interp12 track(s).
DEBUG 2: Deriving 0 ADECK consensus model(s).
DEBUG 2: Added 0 ADECK consensus tracks(s).
DEBUG 2: Deriving 0 ADECK lag model(s).
DEBUG 2: Added 0 ADECK lag tracks(s).
DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
DEBUG 2: Filtering 0 ADECK tracks based on config file settings.
DEBUG 3: Total tracks read = 0
DEBUG 3: Total tracks kept = 0
DEBUG 3: Rejected for storm name = 0
DEBUG 3: Rejected for valid time = 0
DEBUG 3: Rejected for required lead times = 0
DEBUG 3: Rejected for init mask = 0
DEBUG 3: Rejected for valid mask = 0
DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
DEBUG 1: Watch/Warning file: /opt/MET7/share/met/tc_data/wwpts_us.txt
DEBUG 1: Output file:
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018083118.gfso.0053_test_modified_output.txt.tcst
On Mon, Nov 26, 2018 at 2:06 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Roz,
>
> Page 334 of the user's guide (
>
https://dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v8.0.pdf)
> mentions that only the first 8 ATCF columns are required. If
additional
> columns are present in the input ATCF files, then tc_pairs will
interpret
> them as specified by the ATCF format.
>
> So if the storm name column is present, MET in interpret those
values as
> the storm name.
>
> For Guang Ping's data, probably best to have that column empty.
>
> John
>
>
> On Mon, Nov 26, 2018 at 11:51 AM Rosalyn MacCracken - NOAA Affiliate
via RT
> <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
> >
> > Hi John,
> >
> > Ok, I will ignore the suffix error. I do have numeric values in
> > STORMNAME. I wonder if in the modified files I have, this column
is not
> > actually STORMNAME, but, some other variable. Now, I don't have
the
> number
> > of columns that the NRL website has. Will that be a problem?
Should I
> > create dummy columns?
> >
> > Roz
> >
> > On Mon, Nov 26, 2018 at 1:14 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Roz,
> > >
> > > I see that you're wondering about this log message:
> > > *Model Suffix: (nul)*
> > >
> > > Everything is fine, you do not need to be concerned about this.
> Here's a
> > > line taken from the usage statement for the tc_pairs tool:
> > > *Note: The "-adeck", "-edeck", and "-bdeck" options may include
> > > "suffix=string" to modify the model names from that source.*
> > >
> > > Since you didn't specify "suffix=string" on the command line,
the
> suffix
> > > remains the default value of an empty string, i.e. "(nul)". So
why
> would
> > > you ever want to use a suffix? Sometimes data is messy. We
were
> running
> > > tc_pairs to evaluate track data from multiple sources and we
found that
> > > some model names we reused in both sources. Let's say the OFCL
> forecast
> > > showed up in both stream 1.0 and stream 1.5 data. Specifying a
suffix
> > such
> > > as "suffix=_STREAM1.5" changes the model name from "OFCL" to
> > > "OFCL_STREAM1.5". And that enables us to distinguish between
those
> > tracks
> > > in the output.
> > >
> > > But uou probably don't have any need for this option in your
current
> > work.
> > >
> > > However, I am concerned about these log messages:
> > > DEBUG 4: Updating storm name from "-287" to "-305" for
ML0900022018.
> > > DEBUG 4: Updating storm name from "-305" to "-255" for
ML0900022018.
> > > DEBUG 4: Updating storm name from "-255" to "-203" for
ML0900022018.
> > > DEBUG 4: Updating storm name from "-203" to "-205" for
ML0900022018.
> > > DEBUG 4: Updating storm name from "-205" to "-217" for
ML0900022018.
> > > DEBUG 4: Updating storm name from "-217" to "-211" for
ML0900022018.
> > > DEBUG 4: Updating storm name from "-211" to "-182" for
ML0900022018.
> > > DEBUG 4: Updating storm name from "-182" to "-138" for
ML0900022018.
> > >
> > > You likely have numeric values showing up in the "STORMNAME"
ATCF
> column:
> > >
https://www.nrlmry.navy.mil/atcf_web/docs/database/new/abdeck.txt
> > >
> > > I'd recommend looking into that.
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Nov 26, 2018 at 8:44 AM Rosalyn MacCracken - NOAA
Affiliate via
> > RT
> > > <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
>
> > > >
> > > > Hi John,
> > > >
> > > > I figured out an easy way to read in and write out Guang
Ping's
> > modified
> > > > ATCF file, so that it:
> > > > 1) Removes the 3rd column
> > > > 2) Replaces -99 with -9999 in column 21 and 24 (after the 3rd
column
> > is
> > > > removed)
> > > > 3) Updates column 2 to include month+cyclone number. (You
had said
> to
> > > do
> > > > year+cyclone #, but the tc_pairs_wrapper.py said to do
> month+cyclone#)
> > > >
> > > > I changed Adeck and Bdeck.
> > > > Just a couple lines of my file:
> > > >
> > > > ML, 090002, 2018090100, 03, GFSO, 000, 511S, 897W, 40, 976,
XX,
> 34,
> > > > NEQ, 0625, 0630, 0000, 0226, 978, 68, -9999, 123, 155,
-9999,
> -9999,
> > > > -9999, -297, -394, -287, -458, 0073
> > > > ML, 090002, 2018090106, 03, GFSO, 000, 537S, 854W, 44, 964,
XX,
> 34,
> > > > NEQ, 0640, 0645, 0000, 0297, 969, 108, -9999, 119, 147,
-9999,
> -9999,
> > > > -9999, -403, -648, -305, -394, 0055
> > > >
> > > > So, I've run tc_pairs again and my output is below. I don't
> understand
> > > the
> > > > Model Suffix: (nul). Is the input format still wrong?
> > > >
> > > > Roz
> > > >
> > > > Output:
> > > > /opt/MET7/bin/tc_pairs -adeck
> > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat
> > > > -bdeck
> > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat
> > > > -config
> > ~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > > > -out
> > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt
> > > > -log
> > >
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> > > > -v 4
> > > > DEBUG 1: [Source 1 of 1] ADECK Source:
> > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat,
> > > > Model Suffix: (nul)
> > > > DEBUG 1: [Source 1 of 1] BDECK Source:
> > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat,
> > > > Model Suffix: (nul)
> > > > DEBUG 1: Config File Default:
> > > > /opt/MET7/share/met/config/TCPairsConfig_default
> > > > DEBUG 1: Config File User:
> > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > > > DEBUG 1: Distance to land file: /opt/MET7/share/met/tc_data/
> > > > dland_global_tenth_degree.nc
> > > > DEBUG 4:
> > > > DEBUG 4: Latitude/Longitude Grid Data:
> > > > DEBUG 4: lat_ll: -90
> > > > DEBUG 4: lon_ll: 180
> > > > DEBUG 4: delta_lat: 0.1
> > > > DEBUG 4: delta_lon: 0.1
> > > > DEBUG 4: Nlat: 1801
> > > > DEBUG 4: Nlon: 3601
> > > > DEBUG 4:
> > > > DEBUG 2: Processing 1 BDECK file(s).
> > > > DEBUG 4: Updating storm name from "-287" to "-305" for
ML0900022018.
> > > > DEBUG 4: Updating storm name from "-305" to "-255" for
ML0900022018.
> > > > DEBUG 4: Updating storm name from "-255" to "-203" for
ML0900022018.
> > > > DEBUG 4: Updating storm name from "-203" to "-205" for
ML0900022018.
> > > > DEBUG 4: Updating storm name from "-205" to "-217" for
ML0900022018.
> > > > DEBUG 4: Updating storm name from "-217" to "-211" for
ML0900022018.
> > > > DEBUG 4: Updating storm name from "-211" to "-182" for
ML0900022018.
> > > > DEBUG 4: Updating storm name from "-182" to "-138" for
ML0900022018.
> > > > DEBUG 4: [File 1 of 1] Used 9 of 9 lines read from file
> > > >
> > > >
> > >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat"
> > > > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
> > > > DEBUG 3: Identified 1 track(s).
> > > > DEBUG 4: [Track 1 of 1] TrackInfo: StormId = "ML0900022018",
IsBest =
> > > > false, IsOper = false, CheckAnly = true, IsAnly = true, Basin
= "ML",
> > > > Cyclone = "090002", StormName = "-138", TechniqueNumber = 3,
> Technique
> > =
> > > > "GFSO", Initials = "-9999", InitTime = NA, MinValidTime =
> > > 20180901_000000,
> > > > MaxValidTime = 20180903_000000, NPoints = 9, NAlloc = 100,
> NTrackLines
> > =
> > > 0
> > > > DEBUG 2: Found 1 BDECK track(s).
> > > > DEBUG 2: Processing 1 ADECK file(s).
> > > > DEBUG 4: [File 1 of 1] Used 0 of 21 lines read from file
> > > >
> > > >
> > >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat"
> > > > DEBUG 3: Used 0 of 21 lines read from 1 file(s).
> > > > DEBUG 3: Identified 0 track(s).
> > > > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> > > > DEBUG 2: Finished adding 0 and replacing 0 Interp12 track(s).
> > > > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > > > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > > > DEBUG 2: Deriving 0 ADECK lag model(s).
> > > > DEBUG 2: Added 0 ADECK lag tracks(s).
> > > > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> > > > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> > > > DEBUG 2: Filtering 0 ADECK tracks based on config file
settings.
> > > > DEBUG 3: Total tracks read = 0
> > > > DEBUG 3: Total tracks kept = 0
> > > > DEBUG 3: Rejected for storm name = 0
> > > > DEBUG 3: Rejected for valid time = 0
> > > > DEBUG 3: Rejected for required lead times = 0
> > > > DEBUG 3: Rejected for init mask = 0
> > > > DEBUG 3: Rejected for valid mask = 0
> > > > DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
> > > > DEBUG 1: Watch/Warning file:
/opt/MET7/share/met/tc_data/wwpts_us.txt
> > > > DEBUG 1: Output file:
> > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt.tcst
> > > >
> > > > On Wed, Nov 21, 2018 at 9:16 AM Rosalyn MacCracken - NOAA
Affiliate <
> > > > rosalyn.maccracken at noaa.gov> wrote:
> > > >
> > > > > Hi John,
> > > > >
> > > > > Yes, that makes sense, and yes, I am using Guang Pings'
modified
> > files.
> > > > > I'm not sure how to run tc_pairs_wrapper.py, though. I
think I
> need
> > > some
> > > > > type of *.conf file to pass in the files I need, or
something like
> > > that.
> > > > > I'm in the process of reading through the code now. I
wonder if I
> > > could
> > > > > just pull out the function process_non_atcf, to run in a
> stand-alone
> > > > mode.
> > > > > I wish there was some sort of documentation with this
code...
> > > > >
> > > > > And, so, once I can read these files, I can use tc_pairs to
create
> > the
> > > > > forecast error statistics, and then use tc_stats to
aggregate the
> > files
> > > > and
> > > > > get the mean, etc, for a period of time, correct? Was there
some
> > sort
> > > of
> > > > > plotting scripts for the output files?
> > > > >
> > > > > So, with tc_pairs, I can define a region, like I did with
> point_stat,
> > > > > correct? Can I use the same *.nc files, or do I use the
original
> > > *.poly
> > > > > files?
> > > > >
> > > > > Roz
> > > > >
> > > > > On Tue, Nov 20, 2018 at 7:53 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > >> Hi Roz,
> > > > >>
> > > > >> Yes, I see you’re using the modified ATCF format that Guang
Ping
> > uses.
> > > > >> You’re correct that tc_pairs doesn’t parse this directly
since it
> > > > differs
> > > > >> slightly from the ATCF format.
> > > > >>
> > > > >> When we’ve processed this data in the past, we first pre-
processed
> > it
> > > > in a
> > > > >> shell script to:
> > > > >>
> > > > >> 1. Replace the -99 value with -9999 for missing data.
> > > > >> 2. Remove the 3rd column
> > > > >> 3. Updating the 2nd column to include the year and cyclone
number.
> > > > >>
> > > > >> And that enable tc_pairs to read it.
> > > > >>
> > > > >> This is actually Done in python by one of the use cases in
> METplus.
> > > > >>
> > > > >> Take a look at this file:
> > > > >>
> https://github.com/NCAR/METplus/blob/master/ush/tc_pairs_wrapper.py
> > > > >>
> > > > >> At the function named:
> > > > >> process_non_atcf
> > > > >>
> > > > >> I haven’t run this using METplus scripts myself, but all
the
> pieces
> > of
> > > > >> logic are there, and it should be possible.
> > > > >>
> > > > >> Make sense?
> > > > >>
> > > > >> Thanks
> > > > >> John
> > > > >>
> > > > >> On Tue, Nov 20, 2018 at 1:13 PM Rosalyn MacCracken - NOAA
> Affiliate
> > > via
> > > > >> RT <
> > > > >> met_help at ucar.edu> wrote:
> > > > >>
> > > > >> >
> > > > >> > Tue Nov 20 13:13:05 2018: Request 87885 was acted upon.
> > > > >> > Transaction: Ticket created by
rosalyn.maccracken at noaa.gov
> > > > >> > Queue: met_help
> > > > >> > Subject: Help with using TC_pairs
> > > > >> > Owner: Nobody
> > > > >> > Requestors: rosalyn.maccracken at noaa.gov
> > > > >> > Status: new
> > > > >> > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
> > > > >
> > > > >> >
> > > > >> >
> > > > >> > Hi,
> > > > >> >
> > > > >> > I'm trying to use TC_pairs for extra-tropical cyclones
> (basin=ML).
> > > I
> > > > >> think
> > > > >> > the format of the ATCF data is the issue, but, I'm not
sure. I
> > have
> > > > an
> > > > >> > Adeck and a Bdeck file, where a sample of one of the
lines is:
> > > > >> > ML, 0003, 2018090100_F000_687N_0159W_FOF, 2018090100, 03,
GFSO,
> > 000,
> > > > >> 687N,
> > > > >> > 159W, 32, 987, XX, 34, NEQ, 0301, 0338, 0000, 0000,
988,
> 52,
> > > > -99,
> > > > >> > 330, 86, -99, -9999, -9999, 175, 202, 137, 135, 0081
> > > > >> >
> > > > >> > This is slightly different from the NRL ATCF format on
their
> > > website,
> > > > >> such
> > > > >> > as the second column. Anyway, I configured the config
file very
> > > > simply,
> > > > >> > (model=GFS, basin=ML), and ran tc_pairs, and got the
output to
> the
> > > > >> screen
> > > > >> > below.
> > > > >> >
> > > > >> > Is there a way to work around the diferent ATCF format?
I've
> > > > attached a
> > > > >> > sample of my Adeck and Bdeck files.
> > > > >> >
> > > > >> > Thanks,
> > > > >> >
> > > > >> > Roz
> > > > >> >
> > > > >> > Output to the screen:
> > > > >> > /opt/MET7/bin/tc_pairs -adeck
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003
> > > > >> > -bdeck
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003
> > > > >> > -config
> > > > >> >
> > > >
>
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> > > > >> > -out
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt
> > > > >> > -log
> > > > >>
> >
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> > > > >> > -v 3
> > > > >> > DEBUG 1: [Source 1 of 1] ADECK Source:
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003,
> > > > >> > Model Suffix: (nul)
> > > > >> > DEBUG 1: [Source 1 of 1] BDECK Source:
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003,
> > > > >> > Model Suffix: (nul)
> > > > >> > DEBUG 1: Config File Default:
> > > > >> > /opt/MET7/share/met/config/TCPairsConfig_default
> > > > >> > DEBUG 1: Config File User:
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> > > > >> > DEBUG 1: Distance to land file:
/opt/MET7/share/met/tc_data/
> > > > >> > dland_global_tenth_degree.nc
> > > > >> > DEBUG 2: Processing 1 BDECK file(s).
> > > > >> > WARNING:
> > > > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > > > >> > WARNING:
> > > > >> > WARNING:
> > > > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"687N"
> > > > >> > WARNING:
> > > > >> > WARNING:
> > > > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > > > >> > WARNING:
> > > > >> > WARNING:
> > > > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"701N"
> > > > >> > WARNING:
> > > > >> > WARNING:
> > > > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > > > >> > WARNING:
> > > > >> > WARNING:
> > > > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"711N"
> > > > >> > WARNING:
> > > > >> > WARNING:
> > > > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > > > >> > WARNING:
> > > > >> > WARNING:
> > > > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"720N"
> > > > >> > WARNING:
> > > > >> > WARNING:
> > > > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > > > >> > WARNING:
> > > > >> > WARNING:
> > > > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"730N"
> > > > >> > WARNING:
> > > > >> > WARNING:
> > > > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > > > >> > WARNING:
> > > > >> > WARNING:
> > > > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"741N"
> > > > >> > WARNING:
> > > > >> > WARNING:
> > > > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > > > >> > WARNING:
> > > > >> > WARNING:
> > > > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"751N"
> > > > >> > WARNING:
> > > > >> > WARNING:
> > > > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > > > >> > WARNING:
> > > > >> > WARNING:
> > > > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"760N"
> > > > >> > WARNING:
> > > > >> > WARNING:
> > > > >> > WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
> > > > >> > WARNING:
> > > > >> > WARNING:
> > > > >> > WARNING: int parse_lon(const char *) -> bad longitude ...
"764N"
> > > > >> > WARNING:
> > > > >> > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
> > > > >> > DEBUG 3: Identified 9 track(s).
> > > > >> > DEBUG 2: Found 9 BDECK track(s).
> > > > >> > DEBUG 2: Processing 1 ADECK file(s).
> > > > >> > DEBUG 3: Used 0 of 22 lines read from 1 file(s).
> > > > >> > DEBUG 3: Identified 0 track(s).
> > > > >> > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> > > > >> > DEBUG 2: Finished adding 0 and replacing 0 Interp12
track(s).
> > > > >> > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > > > >> > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > > > >> > DEBUG 2: Deriving 0 ADECK lag model(s).
> > > > >> > DEBUG 2: Added 0 ADECK lag tracks(s).
> > > > >> > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> > > > >> > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> > > > >> > DEBUG 2: Filtering 0 ADECK tracks based on config file
settings.
> > > > >> > DEBUG 3: Total tracks read = 0
> > > > >> > DEBUG 3: Total tracks kept = 0
> > > > >> > DEBUG 3: Rejected for storm name = 0
> > > > >> > DEBUG 3: Rejected for valid time = 0
> > > > >> > DEBUG 3: Rejected for required lead times = 0
> > > > >> > DEBUG 3: Rejected for init mask = 0
> > > > >> > DEBUG 3: Rejected for valid mask = 0
> > > > >> > DEBUG 2: Matching 0 ADECK tracks to 9 BDECK tracks.
> > > > >> > DEBUG 1: Watch/Warning file:
> > > /opt/MET7/share/met/tc_data/wwpts_us.txt
> > > > >> > DEBUG 1: Output file:
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt.tcst
> > > > >> >
> > > > >> > --
> > > > >> > Rosalyn MacCracken
> > > > >> > Support Scientist
> > > > >> >
> > > > >> > Ocean Applications Branch
> > > > >> > NOAA/NWS Ocean Prediction Center
> > > > >> > NCWCP
> > > > >> > 5830 University Research Ct
> > > > >> > College Park, MD 20740-3818
> > > > >> >
> > > > >> > (p) 301-683-1551
> > > > >> > rosalyn.maccracken at noaa.gov
> > > > >> >
> > > > >> >
> > > > >>
> > > > >>
> > > > >
> > > > > --
> > > > > Rosalyn MacCracken
> > > > > Support Scientist
> > > > >
> > > > > Ocean Applications Branch
> > > > > NOAA/NWS Ocean Prediction Center
> > > > > NCWCP
> > > > > 5830 University Research Ct
> > > > > College Park, MD 20740-3818
> > > > >
> > > > > (p) 301-683-1551
> > > > > rosalyn.maccracken at noaa.gov
> > > > >
> > > >
> > > >
> > > > --
> > > > Rosalyn MacCracken
> > > > Support Scientist
> > > >
> > > > Ocean Applications Branch
> > > > NOAA/NWS Ocean Prediction Center
> > > > NCWCP
> > > > 5830 University Research Ct
> > > > College Park, MD 20740-3818
> > > >
> > > > (p) 301-683-1551
> > > > rosalyn.maccracken at noaa.gov
> > > >
> > > >
> > >
> > >
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applications Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
> >
>
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Help with using TC_pairs
From: John Halley Gotway
Time: Mon Nov 26 15:22:20 2018
Roz,
Yes, the first 8 columns are required. The additional ones are
optional.
Looking at your log output, I see that tc_pairs is skipping over all
642
input lines. And the question is why? If you send me your latest
ADeck,
BDeck, and TC-Pairs config files, I could run it here and try to
figure out
why it isn't defining any ADECK tracks.
Thanks,
John
On Mon, Nov 26, 2018 at 12:23 PM Rosalyn MacCracken - NOAA Affiliate
via RT
<met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
>
> So, can I just have 8 columns for the input file, or, do I need to
make
> columns of -9999 or NaN for all the rest of the columns?
>
> I experimented, and just wrote out 10 columns for the input file:
> agfso_trackdf_mod =
>
>
agfso_trackdf[['basin_name','storm_id','model_start_date','const','model','fcst_lead_time','vort_lat_cent','vort_lon_cent','vort_10m_max',
> 'vort_min_mslp','type','thresh_wspd']]
>
> because I wanted stats for these variables, and I got the output
below.
> So, maybe it's working, but, the config file needs to be different?
Any
> other ideas of what I should try doing/looking at?
>
> Roz
>
> Output:
> DEBUG 1: [Source 1 of 1] ADECK Source:
>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018083118.gfso.0053_modified.dat,
> Model Suffix: _EXP
> DEBUG 1: [Source 1 of 1] BDECK Source:
>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018083118.gfso.0053_modified.dat,
> Model Suffix: (nul)
> DEBUG 1: Config File Default:
> /opt/MET7/share/met/config/TCPairsConfig_default
> DEBUG 1: Config File User:
>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> DEBUG 1: Distance to land file: /opt/MET7/share/met/tc_data/
> dland_global_tenth_degree.nc
> DEBUG 4:
> DEBUG 4: Latitude/Longitude Grid Data:
> DEBUG 4: lat_ll: -90
> DEBUG 4: lon_ll: 180
> DEBUG 4: delta_lat: 0.1
> DEBUG 4: delta_lon: 0.1
> DEBUG 4: Nlat: 1801
> DEBUG 4: Nlon: 3601
> DEBUG 4:
> DEBUG 2: Processing 1 BDECK file(s).
> DEBUG 4: [File 1 of 1] Used 30 of 30 lines read from file
>
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018083118.gfso.0053_modified.dat"
> DEBUG 3: Used 30 of 30 lines read from 1 file(s).
> DEBUG 3: Identified 1 track(s).
> DEBUG 4: [Track 1 of 1] TrackInfo: StormId = "ML0800532018", IsBest
=
> false, IsOper = false, CheckAnly = true, IsAnly = true, Basin =
"ML",
> Cyclone = "080053", StormName = "", TechniqueNumber = 3, Technique =
> "GFSO", Initials = "", InitTime = NA, MinValidTime =
20180805_060000,
> MaxValidTime = 20180812_120000, NPoints = 30, NAlloc = 100,
NTrackLines = 0
> DEBUG 2: Found 1 BDECK track(s).
> DEBUG 2: Processing 1 ADECK file(s).
> DEBUG 4: [File 1 of 1] Used 0 of 642 lines read from file
>
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018083118.gfso.0053_modified.dat"
> DEBUG 3: Used 0 of 642 lines read from 1 file(s).
> DEBUG 3: Identified 0 track(s).
> DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> DEBUG 2: Finished adding 0 and replacing 0 Interp12 track(s).
> DEBUG 2: Deriving 0 ADECK consensus model(s).
> DEBUG 2: Added 0 ADECK consensus tracks(s).
> DEBUG 2: Deriving 0 ADECK lag model(s).
> DEBUG 2: Added 0 ADECK lag tracks(s).
> DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> DEBUG 2: Filtering 0 ADECK tracks based on config file settings.
> DEBUG 3: Total tracks read = 0
> DEBUG 3: Total tracks kept = 0
> DEBUG 3: Rejected for storm name = 0
> DEBUG 3: Rejected for valid time = 0
> DEBUG 3: Rejected for required lead times = 0
> DEBUG 3: Rejected for init mask = 0
> DEBUG 3: Rejected for valid mask = 0
> DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
> DEBUG 1: Watch/Warning file:
/opt/MET7/share/met/tc_data/wwpts_us.txt
> DEBUG 1: Output file:
>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018083118.gfso.0053_test_modified_output.txt.tcst
>
>
> On Mon, Nov 26, 2018 at 2:06 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Roz,
> >
> > Page 334 of the user's guide (
> >
https://dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v8.0.pdf
> )
> > mentions that only the first 8 ATCF columns are required. If
additional
> > columns are present in the input ATCF files, then tc_pairs will
interpret
> > them as specified by the ATCF format.
> >
> > So if the storm name column is present, MET in interpret those
values as
> > the storm name.
> >
> > For Guang Ping's data, probably best to have that column empty.
> >
> > John
> >
> >
> > On Mon, Nov 26, 2018 at 11:51 AM Rosalyn MacCracken - NOAA
Affiliate via
> RT
> > <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
> > >
> > > Hi John,
> > >
> > > Ok, I will ignore the suffix error. I do have numeric values in
> > > STORMNAME. I wonder if in the modified files I have, this
column is
> not
> > > actually STORMNAME, but, some other variable. Now, I don't have
the
> > number
> > > of columns that the NRL website has. Will that be a problem?
Should I
> > > create dummy columns?
> > >
> > > Roz
> > >
> > > On Mon, Nov 26, 2018 at 1:14 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Roz,
> > > >
> > > > I see that you're wondering about this log message:
> > > > *Model Suffix: (nul)*
> > > >
> > > > Everything is fine, you do not need to be concerned about
this.
> > Here's a
> > > > line taken from the usage statement for the tc_pairs tool:
> > > > *Note: The "-adeck", "-edeck", and "-bdeck" options may
include
> > > > "suffix=string" to modify the model names from that source.*
> > > >
> > > > Since you didn't specify "suffix=string" on the command line,
the
> > suffix
> > > > remains the default value of an empty string, i.e. "(nul)".
So why
> > would
> > > > you ever want to use a suffix? Sometimes data is messy. We
were
> > running
> > > > tc_pairs to evaluate track data from multiple sources and we
found
> that
> > > > some model names we reused in both sources. Let's say the
OFCL
> > forecast
> > > > showed up in both stream 1.0 and stream 1.5 data. Specifying
a
> suffix
> > > such
> > > > as "suffix=_STREAM1.5" changes the model name from "OFCL" to
> > > > "OFCL_STREAM1.5". And that enables us to distinguish between
those
> > > tracks
> > > > in the output.
> > > >
> > > > But uou probably don't have any need for this option in your
current
> > > work.
> > > >
> > > > However, I am concerned about these log messages:
> > > > DEBUG 4: Updating storm name from "-287" to "-305" for
ML0900022018.
> > > > DEBUG 4: Updating storm name from "-305" to "-255" for
ML0900022018.
> > > > DEBUG 4: Updating storm name from "-255" to "-203" for
ML0900022018.
> > > > DEBUG 4: Updating storm name from "-203" to "-205" for
ML0900022018.
> > > > DEBUG 4: Updating storm name from "-205" to "-217" for
ML0900022018.
> > > > DEBUG 4: Updating storm name from "-217" to "-211" for
ML0900022018.
> > > > DEBUG 4: Updating storm name from "-211" to "-182" for
ML0900022018.
> > > > DEBUG 4: Updating storm name from "-182" to "-138" for
ML0900022018.
> > > >
> > > > You likely have numeric values showing up in the "STORMNAME"
ATCF
> > column:
> > > >
https://www.nrlmry.navy.mil/atcf_web/docs/database/new/abdeck.txt
> > > >
> > > > I'd recommend looking into that.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Mon, Nov 26, 2018 at 8:44 AM Rosalyn MacCracken - NOAA
Affiliate
> via
> > > RT
> > > > <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
> > > > >
> > > > > Hi John,
> > > > >
> > > > > I figured out an easy way to read in and write out Guang
Ping's
> > > modified
> > > > > ATCF file, so that it:
> > > > > 1) Removes the 3rd column
> > > > > 2) Replaces -99 with -9999 in column 21 and 24 (after the
3rd
> column
> > > is
> > > > > removed)
> > > > > 3) Updates column 2 to include month+cyclone number. (You
had
> said
> > to
> > > > do
> > > > > year+cyclone #, but the tc_pairs_wrapper.py said to do
> > month+cyclone#)
> > > > >
> > > > > I changed Adeck and Bdeck.
> > > > > Just a couple lines of my file:
> > > > >
> > > > > ML, 090002, 2018090100, 03, GFSO, 000, 511S, 897W, 40,
976, XX,
> > 34,
> > > > > NEQ, 0625, 0630, 0000, 0226, 978, 68, -9999, 123, 155,
-9999,
> > -9999,
> > > > > -9999, -297, -394, -287, -458, 0073
> > > > > ML, 090002, 2018090106, 03, GFSO, 000, 537S, 854W, 44,
964, XX,
> > 34,
> > > > > NEQ, 0640, 0645, 0000, 0297, 969, 108, -9999, 119, 147,
-9999,
> > -9999,
> > > > > -9999, -403, -648, -305, -394, 0055
> > > > >
> > > > > So, I've run tc_pairs again and my output is below. I don't
> > understand
> > > > the
> > > > > Model Suffix: (nul). Is the input format still wrong?
> > > > >
> > > > > Roz
> > > > >
> > > > > Output:
> > > > > /opt/MET7/bin/tc_pairs -adeck
> > > > >
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat
> > > > > -bdeck
> > > > >
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat
> > > > > -config
> > > ~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > > > > -out
> > > > >
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt
> > > > > -log
> > > >
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> > > > > -v 4
> > > > > DEBUG 1: [Source 1 of 1] ADECK Source:
> > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat,
> > > > > Model Suffix: (nul)
> > > > > DEBUG 1: [Source 1 of 1] BDECK Source:
> > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat,
> > > > > Model Suffix: (nul)
> > > > > DEBUG 1: Config File Default:
> > > > > /opt/MET7/share/met/config/TCPairsConfig_default
> > > > > DEBUG 1: Config File User:
> > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > > > > DEBUG 1: Distance to land file: /opt/MET7/share/met/tc_data/
> > > > > dland_global_tenth_degree.nc
> > > > > DEBUG 4:
> > > > > DEBUG 4: Latitude/Longitude Grid Data:
> > > > > DEBUG 4: lat_ll: -90
> > > > > DEBUG 4: lon_ll: 180
> > > > > DEBUG 4: delta_lat: 0.1
> > > > > DEBUG 4: delta_lon: 0.1
> > > > > DEBUG 4: Nlat: 1801
> > > > > DEBUG 4: Nlon: 3601
> > > > > DEBUG 4:
> > > > > DEBUG 2: Processing 1 BDECK file(s).
> > > > > DEBUG 4: Updating storm name from "-287" to "-305" for
> ML0900022018.
> > > > > DEBUG 4: Updating storm name from "-305" to "-255" for
> ML0900022018.
> > > > > DEBUG 4: Updating storm name from "-255" to "-203" for
> ML0900022018.
> > > > > DEBUG 4: Updating storm name from "-203" to "-205" for
> ML0900022018.
> > > > > DEBUG 4: Updating storm name from "-205" to "-217" for
> ML0900022018.
> > > > > DEBUG 4: Updating storm name from "-217" to "-211" for
> ML0900022018.
> > > > > DEBUG 4: Updating storm name from "-211" to "-182" for
> ML0900022018.
> > > > > DEBUG 4: Updating storm name from "-182" to "-138" for
> ML0900022018.
> > > > > DEBUG 4: [File 1 of 1] Used 9 of 9 lines read from file
> > > > >
> > > > >
> > > >
> > >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat"
> > > > > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
> > > > > DEBUG 3: Identified 1 track(s).
> > > > > DEBUG 4: [Track 1 of 1] TrackInfo: StormId = "ML0900022018",
> IsBest =
> > > > > false, IsOper = false, CheckAnly = true, IsAnly = true,
Basin =
> "ML",
> > > > > Cyclone = "090002", StormName = "-138", TechniqueNumber = 3,
> > Technique
> > > =
> > > > > "GFSO", Initials = "-9999", InitTime = NA, MinValidTime =
> > > > 20180901_000000,
> > > > > MaxValidTime = 20180903_000000, NPoints = 9, NAlloc = 100,
> > NTrackLines
> > > =
> > > > 0
> > > > > DEBUG 2: Found 1 BDECK track(s).
> > > > > DEBUG 2: Processing 1 ADECK file(s).
> > > > > DEBUG 4: [File 1 of 1] Used 0 of 21 lines read from file
> > > > >
> > > > >
> > > >
> > >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat"
> > > > > DEBUG 3: Used 0 of 21 lines read from 1 file(s).
> > > > > DEBUG 3: Identified 0 track(s).
> > > > > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> > > > > DEBUG 2: Finished adding 0 and replacing 0 Interp12
track(s).
> > > > > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > > > > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > > > > DEBUG 2: Deriving 0 ADECK lag model(s).
> > > > > DEBUG 2: Added 0 ADECK lag tracks(s).
> > > > > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> > > > > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> > > > > DEBUG 2: Filtering 0 ADECK tracks based on config file
settings.
> > > > > DEBUG 3: Total tracks read = 0
> > > > > DEBUG 3: Total tracks kept = 0
> > > > > DEBUG 3: Rejected for storm name = 0
> > > > > DEBUG 3: Rejected for valid time = 0
> > > > > DEBUG 3: Rejected for required lead times = 0
> > > > > DEBUG 3: Rejected for init mask = 0
> > > > > DEBUG 3: Rejected for valid mask = 0
> > > > > DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
> > > > > DEBUG 1: Watch/Warning file:
> /opt/MET7/share/met/tc_data/wwpts_us.txt
> > > > > DEBUG 1: Output file:
> > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt.tcst
> > > > >
> > > > > On Wed, Nov 21, 2018 at 9:16 AM Rosalyn MacCracken - NOAA
> Affiliate <
> > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > Yes, that makes sense, and yes, I am using Guang Pings'
modified
> > > files.
> > > > > > I'm not sure how to run tc_pairs_wrapper.py, though. I
think I
> > need
> > > > some
> > > > > > type of *.conf file to pass in the files I need, or
something
> like
> > > > that.
> > > > > > I'm in the process of reading through the code now. I
wonder if
> I
> > > > could
> > > > > > just pull out the function process_non_atcf, to run in a
> > stand-alone
> > > > > mode.
> > > > > > I wish there was some sort of documentation with this
code...
> > > > > >
> > > > > > And, so, once I can read these files, I can use tc_pairs
to
> create
> > > the
> > > > > > forecast error statistics, and then use tc_stats to
aggregate the
> > > files
> > > > > and
> > > > > > get the mean, etc, for a period of time, correct? Was
there some
> > > sort
> > > > of
> > > > > > plotting scripts for the output files?
> > > > > >
> > > > > > So, with tc_pairs, I can define a region, like I did with
> > point_stat,
> > > > > > correct? Can I use the same *.nc files, or do I use the
original
> > > > *.poly
> > > > > > files?
> > > > > >
> > > > > > Roz
> > > > > >
> > > > > > On Tue, Nov 20, 2018 at 7:53 PM John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > >> Hi Roz,
> > > > > >>
> > > > > >> Yes, I see you’re using the modified ATCF format that
Guang Ping
> > > uses.
> > > > > >> You’re correct that tc_pairs doesn’t parse this directly
since
> it
> > > > > differs
> > > > > >> slightly from the ATCF format.
> > > > > >>
> > > > > >> When we’ve processed this data in the past, we first
> pre-processed
> > > it
> > > > > in a
> > > > > >> shell script to:
> > > > > >>
> > > > > >> 1. Replace the -99 value with -9999 for missing data.
> > > > > >> 2. Remove the 3rd column
> > > > > >> 3. Updating the 2nd column to include the year and
cyclone
> number.
> > > > > >>
> > > > > >> And that enable tc_pairs to read it.
> > > > > >>
> > > > > >> This is actually Done in python by one of the use cases
in
> > METplus.
> > > > > >>
> > > > > >> Take a look at this file:
> > > > > >>
> >
https://github.com/NCAR/METplus/blob/master/ush/tc_pairs_wrapper.py
> > > > > >>
> > > > > >> At the function named:
> > > > > >> process_non_atcf
> > > > > >>
> > > > > >> I haven’t run this using METplus scripts myself, but all
the
> > pieces
> > > of
> > > > > >> logic are there, and it should be possible.
> > > > > >>
> > > > > >> Make sense?
> > > > > >>
> > > > > >> Thanks
> > > > > >> John
> > > > > >>
> > > > > >> On Tue, Nov 20, 2018 at 1:13 PM Rosalyn MacCracken - NOAA
> > Affiliate
> > > > via
> > > > > >> RT <
> > > > > >> met_help at ucar.edu> wrote:
> > > > > >>
> > > > > >> >
> > > > > >> > Tue Nov 20 13:13:05 2018: Request 87885 was acted upon.
> > > > > >> > Transaction: Ticket created by
rosalyn.maccracken at noaa.gov
> > > > > >> > Queue: met_help
> > > > > >> > Subject: Help with using TC_pairs
> > > > > >> > Owner: Nobody
> > > > > >> > Requestors: rosalyn.maccracken at noaa.gov
> > > > > >> > Status: new
> > > > > >> > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
> > > > > >
> > > > > >> >
> > > > > >> >
> > > > > >> > Hi,
> > > > > >> >
> > > > > >> > I'm trying to use TC_pairs for extra-tropical cyclones
> > (basin=ML).
> > > > I
> > > > > >> think
> > > > > >> > the format of the ATCF data is the issue, but, I'm not
sure.
> I
> > > have
> > > > > an
> > > > > >> > Adeck and a Bdeck file, where a sample of one of the
lines is:
> > > > > >> > ML, 0003, 2018090100_F000_687N_0159W_FOF, 2018090100,
03,
> GFSO,
> > > 000,
> > > > > >> 687N,
> > > > > >> > 159W, 32, 987, XX, 34, NEQ, 0301, 0338, 0000, 0000,
988,
> > 52,
> > > > > -99,
> > > > > >> > 330, 86, -99, -9999, -9999, 175, 202, 137, 135,
0081
> > > > > >> >
> > > > > >> > This is slightly different from the NRL ATCF format on
their
> > > > website,
> > > > > >> such
> > > > > >> > as the second column. Anyway, I configured the config
file
> very
> > > > > simply,
> > > > > >> > (model=GFS, basin=ML), and ran tc_pairs, and got the
output to
> > the
> > > > > >> screen
> > > > > >> > below.
> > > > > >> >
> > > > > >> > Is there a way to work around the diferent ATCF format?
I've
> > > > > attached a
> > > > > >> > sample of my Adeck and Bdeck files.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> >
> > > > > >> > Roz
> > > > > >> >
> > > > > >> > Output to the screen:
> > > > > >> > /opt/MET7/bin/tc_pairs -adeck
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003
> > > > > >> > -bdeck
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003
> > > > > >> > -config
> > > > > >> >
> > > > >
> >
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> > > > > >> > -out
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt
> > > > > >> > -log
> > > > > >>
> > >
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> > > > > >> > -v 3
> > > > > >> > DEBUG 1: [Source 1 of 1] ADECK Source:
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003,
> > > > > >> > Model Suffix: (nul)
> > > > > >> > DEBUG 1: [Source 1 of 1] BDECK Source:
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003,
> > > > > >> > Model Suffix: (nul)
> > > > > >> > DEBUG 1: Config File Default:
> > > > > >> > /opt/MET7/share/met/config/TCPairsConfig_default
> > > > > >> > DEBUG 1: Config File User:
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> > > > > >> > DEBUG 1: Distance to land file:
/opt/MET7/share/met/tc_data/
> > > > > >> > dland_global_tenth_degree.nc
> > > > > >> > DEBUG 2: Processing 1 BDECK file(s).
> > > > > >> > WARNING:
> > > > > >> > WARNING: int parse_lat(const char *) -> bad latitude
... "000"
> > > > > >> > WARNING:
> > > > > >> > WARNING:
> > > > > >> > WARNING: int parse_lon(const char *) -> bad longitude
...
> "687N"
> > > > > >> > WARNING:
> > > > > >> > WARNING:
> > > > > >> > WARNING: int parse_lat(const char *) -> bad latitude
... "000"
> > > > > >> > WARNING:
> > > > > >> > WARNING:
> > > > > >> > WARNING: int parse_lon(const char *) -> bad longitude
...
> "701N"
> > > > > >> > WARNING:
> > > > > >> > WARNING:
> > > > > >> > WARNING: int parse_lat(const char *) -> bad latitude
... "000"
> > > > > >> > WARNING:
> > > > > >> > WARNING:
> > > > > >> > WARNING: int parse_lon(const char *) -> bad longitude
...
> "711N"
> > > > > >> > WARNING:
> > > > > >> > WARNING:
> > > > > >> > WARNING: int parse_lat(const char *) -> bad latitude
... "000"
> > > > > >> > WARNING:
> > > > > >> > WARNING:
> > > > > >> > WARNING: int parse_lon(const char *) -> bad longitude
...
> "720N"
> > > > > >> > WARNING:
> > > > > >> > WARNING:
> > > > > >> > WARNING: int parse_lat(const char *) -> bad latitude
... "000"
> > > > > >> > WARNING:
> > > > > >> > WARNING:
> > > > > >> > WARNING: int parse_lon(const char *) -> bad longitude
...
> "730N"
> > > > > >> > WARNING:
> > > > > >> > WARNING:
> > > > > >> > WARNING: int parse_lat(const char *) -> bad latitude
... "000"
> > > > > >> > WARNING:
> > > > > >> > WARNING:
> > > > > >> > WARNING: int parse_lon(const char *) -> bad longitude
...
> "741N"
> > > > > >> > WARNING:
> > > > > >> > WARNING:
> > > > > >> > WARNING: int parse_lat(const char *) -> bad latitude
... "000"
> > > > > >> > WARNING:
> > > > > >> > WARNING:
> > > > > >> > WARNING: int parse_lon(const char *) -> bad longitude
...
> "751N"
> > > > > >> > WARNING:
> > > > > >> > WARNING:
> > > > > >> > WARNING: int parse_lat(const char *) -> bad latitude
... "000"
> > > > > >> > WARNING:
> > > > > >> > WARNING:
> > > > > >> > WARNING: int parse_lon(const char *) -> bad longitude
...
> "760N"
> > > > > >> > WARNING:
> > > > > >> > WARNING:
> > > > > >> > WARNING: int parse_lat(const char *) -> bad latitude
... "000"
> > > > > >> > WARNING:
> > > > > >> > WARNING:
> > > > > >> > WARNING: int parse_lon(const char *) -> bad longitude
...
> "764N"
> > > > > >> > WARNING:
> > > > > >> > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
> > > > > >> > DEBUG 3: Identified 9 track(s).
> > > > > >> > DEBUG 2: Found 9 BDECK track(s).
> > > > > >> > DEBUG 2: Processing 1 ADECK file(s).
> > > > > >> > DEBUG 3: Used 0 of 22 lines read from 1 file(s).
> > > > > >> > DEBUG 3: Identified 0 track(s).
> > > > > >> > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> > > > > >> > DEBUG 2: Finished adding 0 and replacing 0 Interp12
track(s).
> > > > > >> > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > > > > >> > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > > > > >> > DEBUG 2: Deriving 0 ADECK lag model(s).
> > > > > >> > DEBUG 2: Added 0 ADECK lag tracks(s).
> > > > > >> > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> > > > > >> > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> > > > > >> > DEBUG 2: Filtering 0 ADECK tracks based on config file
> settings.
> > > > > >> > DEBUG 3: Total tracks read = 0
> > > > > >> > DEBUG 3: Total tracks kept = 0
> > > > > >> > DEBUG 3: Rejected for storm name = 0
> > > > > >> > DEBUG 3: Rejected for valid time = 0
> > > > > >> > DEBUG 3: Rejected for required lead times = 0
> > > > > >> > DEBUG 3: Rejected for init mask = 0
> > > > > >> > DEBUG 3: Rejected for valid mask = 0
> > > > > >> > DEBUG 2: Matching 0 ADECK tracks to 9 BDECK tracks.
> > > > > >> > DEBUG 1: Watch/Warning file:
> > > > /opt/MET7/share/met/tc_data/wwpts_us.txt
> > > > > >> > DEBUG 1: Output file:
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt.tcst
> > > > > >> >
> > > > > >> > --
> > > > > >> > Rosalyn MacCracken
> > > > > >> > Support Scientist
> > > > > >> >
> > > > > >> > Ocean Applications Branch
> > > > > >> > NOAA/NWS Ocean Prediction Center
> > > > > >> > NCWCP
> > > > > >> > 5830 University Research Ct
> > > > > >> > College Park, MD 20740-3818
> > > > > >> >
> > > > > >> > (p) 301-683-1551
> > > > > >> > rosalyn.maccracken at noaa.gov
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >
> > > > > > --
> > > > > > Rosalyn MacCracken
> > > > > > Support Scientist
> > > > > >
> > > > > > Ocean Applications Branch
> > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > NCWCP
> > > > > > 5830 University Research Ct
> > > > > > College Park, MD 20740-3818
> > > > > >
> > > > > > (p) 301-683-1551
> > > > > > rosalyn.maccracken at noaa.gov
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Rosalyn MacCracken
> > > > > Support Scientist
> > > > >
> > > > > Ocean Applications Branch
> > > > > NOAA/NWS Ocean Prediction Center
> > > > > NCWCP
> > > > > 5830 University Research Ct
> > > > > College Park, MD 20740-3818
> > > > >
> > > > > (p) 301-683-1551
> > > > > rosalyn.maccracken at noaa.gov
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > Rosalyn MacCracken
> > > Support Scientist
> > >
> > > Ocean Applications Branch
> > > NOAA/NWS Ocean Prediction Center
> > > NCWCP
> > > 5830 University Research Ct
> > > College Park, MD 20740-3818
> > >
> > > (p) 301-683-1551
> > > rosalyn.maccracken at noaa.gov
> > >
> > >
> >
> >
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #87885] Help with using TC_pairs
From: Rosalyn MacCracken - NOAA Affiliate
Time: Mon Nov 26 15:43:25 2018
Hi John,
I’ll put the files in the ftp directory first thing in the morning.
Of course, you won’t be at work at that time, and may not even be
awake. Haha.
Yes, I can’t figure out why it is skipping over all the records in the
file. I’m hoping it’s something very easy to fix.
Roz
> On Nov 26, 2018, at 5:22 PM, John Halley Gotway via RT
<met_help at ucar.edu> wrote:
>
> Roz,
>
> Yes, the first 8 columns are required. The additional ones are
optional.
>
> Looking at your log output, I see that tc_pairs is skipping over all
642
> input lines. And the question is why? If you send me your latest
ADeck,
> BDeck, and TC-Pairs config files, I could run it here and try to
figure out
> why it isn't defining any ADECK tracks.
>
> Thanks,
> John
>
> On Mon, Nov 26, 2018 at 12:23 PM Rosalyn MacCracken - NOAA Affiliate
via RT
> <met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
>>
>> So, can I just have 8 columns for the input file, or, do I need to
make
>> columns of -9999 or NaN for all the rest of the columns?
>>
>> I experimented, and just wrote out 10 columns for the input file:
>> agfso_trackdf_mod =
>>
>>
agfso_trackdf[['basin_name','storm_id','model_start_date','const','model','fcst_lead_time','vort_lat_cent','vort_lon_cent','vort_10m_max',
>> 'vort_min_mslp','type','thresh_wspd']]
>>
>> because I wanted stats for these variables, and I got the output
below.
>> So, maybe it's working, but, the config file needs to be different?
Any
>> other ideas of what I should try doing/looking at?
>>
>> Roz
>>
>> Output:
>> DEBUG 1: [Source 1 of 1] ADECK Source:
>>
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018083118.gfso.0053_modified.dat,
>> Model Suffix: _EXP
>> DEBUG 1: [Source 1 of 1] BDECK Source:
>>
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018083118.gfso.0053_modified.dat,
>> Model Suffix: (nul)
>> DEBUG 1: Config File Default:
>> /opt/MET7/share/met/config/TCPairsConfig_default
>> DEBUG 1: Config File User:
>>
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
>> DEBUG 1: Distance to land file: /opt/MET7/share/met/tc_data/
>> dland_global_tenth_degree.nc
>> DEBUG 4:
>> DEBUG 4: Latitude/Longitude Grid Data:
>> DEBUG 4: lat_ll: -90
>> DEBUG 4: lon_ll: 180
>> DEBUG 4: delta_lat: 0.1
>> DEBUG 4: delta_lon: 0.1
>> DEBUG 4: Nlat: 1801
>> DEBUG 4: Nlon: 3601
>> DEBUG 4:
>> DEBUG 2: Processing 1 BDECK file(s).
>> DEBUG 4: [File 1 of 1] Used 30 of 30 lines read from file
>>
>>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018083118.gfso.0053_modified.dat"
>> DEBUG 3: Used 30 of 30 lines read from 1 file(s).
>> DEBUG 3: Identified 1 track(s).
>> DEBUG 4: [Track 1 of 1] TrackInfo: StormId = "ML0800532018", IsBest
=
>> false, IsOper = false, CheckAnly = true, IsAnly = true, Basin =
"ML",
>> Cyclone = "080053", StormName = "", TechniqueNumber = 3, Technique
=
>> "GFSO", Initials = "", InitTime = NA, MinValidTime =
20180805_060000,
>> MaxValidTime = 20180812_120000, NPoints = 30, NAlloc = 100,
NTrackLines = 0
>> DEBUG 2: Found 1 BDECK track(s).
>> DEBUG 2: Processing 1 ADECK file(s).
>> DEBUG 4: [File 1 of 1] Used 0 of 642 lines read from file
>>
>>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018083118.gfso.0053_modified.dat"
>> DEBUG 3: Used 0 of 642 lines read from 1 file(s).
>> DEBUG 3: Identified 0 track(s).
>> DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
>> DEBUG 2: Finished adding 0 and replacing 0 Interp12 track(s).
>> DEBUG 2: Deriving 0 ADECK consensus model(s).
>> DEBUG 2: Added 0 ADECK consensus tracks(s).
>> DEBUG 2: Deriving 0 ADECK lag model(s).
>> DEBUG 2: Added 0 ADECK lag tracks(s).
>> DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
>> DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
>> DEBUG 2: Filtering 0 ADECK tracks based on config file settings.
>> DEBUG 3: Total tracks read = 0
>> DEBUG 3: Total tracks kept = 0
>> DEBUG 3: Rejected for storm name = 0
>> DEBUG 3: Rejected for valid time = 0
>> DEBUG 3: Rejected for required lead times = 0
>> DEBUG 3: Rejected for init mask = 0
>> DEBUG 3: Rejected for valid mask = 0
>> DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
>> DEBUG 1: Watch/Warning file:
/opt/MET7/share/met/tc_data/wwpts_us.txt
>> DEBUG 1: Output file:
>>
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018083118.gfso.0053_test_modified_output.txt.tcst
>>
>>
>> On Mon, Nov 26, 2018 at 2:06 PM John Halley Gotway via RT <
>> met_help at ucar.edu>
>> wrote:
>>
>>> Roz,
>>>
>>> Page 334 of the user's guide (
>>>
https://dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v8.0.pdf
>> )
>>> mentions that only the first 8 ATCF columns are required. If
additional
>>> columns are present in the input ATCF files, then tc_pairs will
interpret
>>> them as specified by the ATCF format.
>>>
>>> So if the storm name column is present, MET in interpret those
values as
>>> the storm name.
>>>
>>> For Guang Ping's data, probably best to have that column empty.
>>>
>>> John
>>>
>>>
>>> On Mon, Nov 26, 2018 at 11:51 AM Rosalyn MacCracken - NOAA
Affiliate via
>> RT
>>> <met_help at ucar.edu> wrote:
>>>
>>>>
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
>>>>
>>>> Hi John,
>>>>
>>>> Ok, I will ignore the suffix error. I do have numeric values in
>>>> STORMNAME. I wonder if in the modified files I have, this column
is
>> not
>>>> actually STORMNAME, but, some other variable. Now, I don't have
the
>>> number
>>>> of columns that the NRL website has. Will that be a problem?
Should I
>>>> create dummy columns?
>>>>
>>>> Roz
>>>>
>>>> On Mon, Nov 26, 2018 at 1:14 PM John Halley Gotway via RT <
>>>> met_help at ucar.edu>
>>>> wrote:
>>>>
>>>>> Roz,
>>>>>
>>>>> I see that you're wondering about this log message:
>>>>> *Model Suffix: (nul)*
>>>>>
>>>>> Everything is fine, you do not need to be concerned about this.
>>> Here's a
>>>>> line taken from the usage statement for the tc_pairs tool:
>>>>> *Note: The "-adeck", "-edeck", and "-bdeck" options may include
>>>>> "suffix=string" to modify the model names from that source.*
>>>>>
>>>>> Since you didn't specify "suffix=string" on the command line,
the
>>> suffix
>>>>> remains the default value of an empty string, i.e. "(nul)". So
why
>>> would
>>>>> you ever want to use a suffix? Sometimes data is messy. We
were
>>> running
>>>>> tc_pairs to evaluate track data from multiple sources and we
found
>> that
>>>>> some model names we reused in both sources. Let's say the OFCL
>>> forecast
>>>>> showed up in both stream 1.0 and stream 1.5 data. Specifying a
>> suffix
>>>> such
>>>>> as "suffix=_STREAM1.5" changes the model name from "OFCL" to
>>>>> "OFCL_STREAM1.5". And that enables us to distinguish between
those
>>>> tracks
>>>>> in the output.
>>>>>
>>>>> But uou probably don't have any need for this option in your
current
>>>> work.
>>>>>
>>>>> However, I am concerned about these log messages:
>>>>> DEBUG 4: Updating storm name from "-287" to "-305" for
ML0900022018.
>>>>> DEBUG 4: Updating storm name from "-305" to "-255" for
ML0900022018.
>>>>> DEBUG 4: Updating storm name from "-255" to "-203" for
ML0900022018.
>>>>> DEBUG 4: Updating storm name from "-203" to "-205" for
ML0900022018.
>>>>> DEBUG 4: Updating storm name from "-205" to "-217" for
ML0900022018.
>>>>> DEBUG 4: Updating storm name from "-217" to "-211" for
ML0900022018.
>>>>> DEBUG 4: Updating storm name from "-211" to "-182" for
ML0900022018.
>>>>> DEBUG 4: Updating storm name from "-182" to "-138" for
ML0900022018.
>>>>>
>>>>> You likely have numeric values showing up in the "STORMNAME"
ATCF
>>> column:
>>>>>
https://www.nrlmry.navy.mil/atcf_web/docs/database/new/abdeck.txt
>>>>>
>>>>> I'd recommend looking into that.
>>>>>
>>>>> Thanks,
>>>>> John
>>>>>
>>>>> On Mon, Nov 26, 2018 at 8:44 AM Rosalyn MacCracken - NOAA
Affiliate
>> via
>>>> RT
>>>>> <
>>>>> met_help at ucar.edu> wrote:
>>>>>
>>>>>>
>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
>>>>>>
>>>>>> Hi John,
>>>>>>
>>>>>> I figured out an easy way to read in and write out Guang Ping's
>>>> modified
>>>>>> ATCF file, so that it:
>>>>>> 1) Removes the 3rd column
>>>>>> 2) Replaces -99 with -9999 in column 21 and 24 (after the 3rd
>> column
>>>> is
>>>>>> removed)
>>>>>> 3) Updates column 2 to include month+cyclone number. (You had
>> said
>>> to
>>>>> do
>>>>>> year+cyclone #, but the tc_pairs_wrapper.py said to do
>>> month+cyclone#)
>>>>>>
>>>>>> I changed Adeck and Bdeck.
>>>>>> Just a couple lines of my file:
>>>>>>
>>>>>> ML, 090002, 2018090100, 03, GFSO, 000, 511S, 897W, 40, 976,
XX,
>>> 34,
>>>>>> NEQ, 0625, 0630, 0000, 0226, 978, 68, -9999, 123, 155,
-9999,
>>> -9999,
>>>>>> -9999, -297, -394, -287, -458, 0073
>>>>>> ML, 090002, 2018090106, 03, GFSO, 000, 537S, 854W, 44, 964,
XX,
>>> 34,
>>>>>> NEQ, 0640, 0645, 0000, 0297, 969, 108, -9999, 119, 147,
-9999,
>>> -9999,
>>>>>> -9999, -403, -648, -305, -394, 0055
>>>>>>
>>>>>> So, I've run tc_pairs again and my output is below. I don't
>>> understand
>>>>> the
>>>>>> Model Suffix: (nul). Is the input format still wrong?
>>>>>>
>>>>>> Roz
>>>>>>
>>>>>> Output:
>>>>>> /opt/MET7/bin/tc_pairs -adeck
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat
>>>>>> -bdeck
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat
>>>>>> -config
>>>> ~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
>>>>>> -out
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt
>>>>>> -log
>>>>>
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
>>>>>> -v 4
>>>>>> DEBUG 1: [Source 1 of 1] ADECK Source:
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat,
>>>>>> Model Suffix: (nul)
>>>>>> DEBUG 1: [Source 1 of 1] BDECK Source:
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat,
>>>>>> Model Suffix: (nul)
>>>>>> DEBUG 1: Config File Default:
>>>>>> /opt/MET7/share/met/config/TCPairsConfig_default
>>>>>> DEBUG 1: Config File User:
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
>>>>>> DEBUG 1: Distance to land file: /opt/MET7/share/met/tc_data/
>>>>>> dland_global_tenth_degree.nc
>>>>>> DEBUG 4:
>>>>>> DEBUG 4: Latitude/Longitude Grid Data:
>>>>>> DEBUG 4: lat_ll: -90
>>>>>> DEBUG 4: lon_ll: 180
>>>>>> DEBUG 4: delta_lat: 0.1
>>>>>> DEBUG 4: delta_lon: 0.1
>>>>>> DEBUG 4: Nlat: 1801
>>>>>> DEBUG 4: Nlon: 3601
>>>>>> DEBUG 4:
>>>>>> DEBUG 2: Processing 1 BDECK file(s).
>>>>>> DEBUG 4: Updating storm name from "-287" to "-305" for
>> ML0900022018.
>>>>>> DEBUG 4: Updating storm name from "-305" to "-255" for
>> ML0900022018.
>>>>>> DEBUG 4: Updating storm name from "-255" to "-203" for
>> ML0900022018.
>>>>>> DEBUG 4: Updating storm name from "-203" to "-205" for
>> ML0900022018.
>>>>>> DEBUG 4: Updating storm name from "-205" to "-217" for
>> ML0900022018.
>>>>>> DEBUG 4: Updating storm name from "-217" to "-211" for
>> ML0900022018.
>>>>>> DEBUG 4: Updating storm name from "-211" to "-182" for
>> ML0900022018.
>>>>>> DEBUG 4: Updating storm name from "-182" to "-138" for
>> ML0900022018.
>>>>>> DEBUG 4: [File 1 of 1] Used 9 of 9 lines read from file
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat"
>>>>>> DEBUG 3: Used 9 of 9 lines read from 1 file(s).
>>>>>> DEBUG 3: Identified 1 track(s).
>>>>>> DEBUG 4: [Track 1 of 1] TrackInfo: StormId = "ML0900022018",
>> IsBest =
>>>>>> false, IsOper = false, CheckAnly = true, IsAnly = true, Basin =
>> "ML",
>>>>>> Cyclone = "090002", StormName = "-138", TechniqueNumber = 3,
>>> Technique
>>>> =
>>>>>> "GFSO", Initials = "-9999", InitTime = NA, MinValidTime =
>>>>> 20180901_000000,
>>>>>> MaxValidTime = 20180903_000000, NPoints = 9, NAlloc = 100,
>>> NTrackLines
>>>> =
>>>>> 0
>>>>>> DEBUG 2: Found 1 BDECK track(s).
>>>>>> DEBUG 2: Processing 1 ADECK file(s).
>>>>>> DEBUG 4: [File 1 of 1] Used 0 of 21 lines read from file
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat"
>>>>>> DEBUG 3: Used 0 of 21 lines read from 1 file(s).
>>>>>> DEBUG 3: Identified 0 track(s).
>>>>>> DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
>>>>>> DEBUG 2: Finished adding 0 and replacing 0 Interp12 track(s).
>>>>>> DEBUG 2: Deriving 0 ADECK consensus model(s).
>>>>>> DEBUG 2: Added 0 ADECK consensus tracks(s).
>>>>>> DEBUG 2: Deriving 0 ADECK lag model(s).
>>>>>> DEBUG 2: Added 0 ADECK lag tracks(s).
>>>>>> DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
>>>>>> DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
>>>>>> DEBUG 2: Filtering 0 ADECK tracks based on config file
settings.
>>>>>> DEBUG 3: Total tracks read = 0
>>>>>> DEBUG 3: Total tracks kept = 0
>>>>>> DEBUG 3: Rejected for storm name = 0
>>>>>> DEBUG 3: Rejected for valid time = 0
>>>>>> DEBUG 3: Rejected for required lead times = 0
>>>>>> DEBUG 3: Rejected for init mask = 0
>>>>>> DEBUG 3: Rejected for valid mask = 0
>>>>>> DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
>>>>>> DEBUG 1: Watch/Warning file:
>> /opt/MET7/share/met/tc_data/wwpts_us.txt
>>>>>> DEBUG 1: Output file:
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt.tcst
>>>>>>
>>>>>> On Wed, Nov 21, 2018 at 9:16 AM Rosalyn MacCracken - NOAA
>> Affiliate <
>>>>>> rosalyn.maccracken at noaa.gov> wrote:
>>>>>>
>>>>>>> Hi John,
>>>>>>>
>>>>>>> Yes, that makes sense, and yes, I am using Guang Pings'
modified
>>>> files.
>>>>>>> I'm not sure how to run tc_pairs_wrapper.py, though. I think
I
>>> need
>>>>> some
>>>>>>> type of *.conf file to pass in the files I need, or something
>> like
>>>>> that.
>>>>>>> I'm in the process of reading through the code now. I wonder
if
>> I
>>>>> could
>>>>>>> just pull out the function process_non_atcf, to run in a
>>> stand-alone
>>>>>> mode.
>>>>>>> I wish there was some sort of documentation with this code...
>>>>>>>
>>>>>>> And, so, once I can read these files, I can use tc_pairs to
>> create
>>>> the
>>>>>>> forecast error statistics, and then use tc_stats to aggregate
the
>>>> files
>>>>>> and
>>>>>>> get the mean, etc, for a period of time, correct? Was there
some
>>>> sort
>>>>> of
>>>>>>> plotting scripts for the output files?
>>>>>>>
>>>>>>> So, with tc_pairs, I can define a region, like I did with
>>> point_stat,
>>>>>>> correct? Can I use the same *.nc files, or do I use the
original
>>>>> *.poly
>>>>>>> files?
>>>>>>>
>>>>>>> Roz
>>>>>>>
>>>>>>> On Tue, Nov 20, 2018 at 7:53 PM John Halley Gotway via RT <
>>>>>>> met_help at ucar.edu> wrote:
>>>>>>>
>>>>>>>> Hi Roz,
>>>>>>>>
>>>>>>>> Yes, I see you’re using the modified ATCF format that Guang
Ping
>>>> uses.
>>>>>>>> You’re correct that tc_pairs doesn’t parse this directly
since
>> it
>>>>>> differs
>>>>>>>> slightly from the ATCF format.
>>>>>>>>
>>>>>>>> When we’ve processed this data in the past, we first
>> pre-processed
>>>> it
>>>>>> in a
>>>>>>>> shell script to:
>>>>>>>>
>>>>>>>> 1. Replace the -99 value with -9999 for missing data.
>>>>>>>> 2. Remove the 3rd column
>>>>>>>> 3. Updating the 2nd column to include the year and cyclone
>> number.
>>>>>>>>
>>>>>>>> And that enable tc_pairs to read it.
>>>>>>>>
>>>>>>>> This is actually Done in python by one of the use cases in
>>> METplus.
>>>>>>>>
>>>>>>>> Take a look at this file:
>>>>>>>>
>>>
https://github.com/NCAR/METplus/blob/master/ush/tc_pairs_wrapper.py
>>>>>>>>
>>>>>>>> At the function named:
>>>>>>>> process_non_atcf
>>>>>>>>
>>>>>>>> I haven’t run this using METplus scripts myself, but all the
>>> pieces
>>>> of
>>>>>>>> logic are there, and it should be possible.
>>>>>>>>
>>>>>>>> Make sense?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> John
>>>>>>>>
>>>>>>>> On Tue, Nov 20, 2018 at 1:13 PM Rosalyn MacCracken - NOAA
>>> Affiliate
>>>>> via
>>>>>>>> RT <
>>>>>>>> met_help at ucar.edu> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Tue Nov 20 13:13:05 2018: Request 87885 was acted upon.
>>>>>>>>> Transaction: Ticket created by rosalyn.maccracken at noaa.gov
>>>>>>>>> Queue: met_help
>>>>>>>>> Subject: Help with using TC_pairs
>>>>>>>>> Owner: Nobody
>>>>>>>>> Requestors: rosalyn.maccracken at noaa.gov
>>>>>>>>> Status: new
>>>>>>>>> Ticket <URL:
>>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I'm trying to use TC_pairs for extra-tropical cyclones
>>> (basin=ML).
>>>>> I
>>>>>>>> think
>>>>>>>>> the format of the ATCF data is the issue, but, I'm not sure.
>> I
>>>> have
>>>>>> an
>>>>>>>>> Adeck and a Bdeck file, where a sample of one of the lines
is:
>>>>>>>>> ML, 0003, 2018090100_F000_687N_0159W_FOF, 2018090100, 03,
>> GFSO,
>>>> 000,
>>>>>>>> 687N,
>>>>>>>>> 159W, 32, 987, XX, 34, NEQ, 0301, 0338, 0000, 0000, 988,
>>> 52,
>>>>>> -99,
>>>>>>>>> 330, 86, -99, -9999, -9999, 175, 202, 137, 135, 0081
>>>>>>>>>
>>>>>>>>> This is slightly different from the NRL ATCF format on their
>>>>> website,
>>>>>>>> such
>>>>>>>>> as the second column. Anyway, I configured the config file
>> very
>>>>>> simply,
>>>>>>>>> (model=GFS, basin=ML), and ran tc_pairs, and got the output
to
>>> the
>>>>>>>> screen
>>>>>>>>> below.
>>>>>>>>>
>>>>>>>>> Is there a way to work around the diferent ATCF format?
I've
>>>>>> attached a
>>>>>>>>> sample of my Adeck and Bdeck files.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Roz
>>>>>>>>>
>>>>>>>>> Output to the screen:
>>>>>>>>> /opt/MET7/bin/tc_pairs -adeck
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
~/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003
>>>>>>>>> -bdeck
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
~/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003
>>>>>>>>> -config
>>>>>>>>>
>>>>>>
>>>
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
>>>>>>>>> -out
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt
>>>>>>>>> -log
>>>>>>>>
>>>>
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
>>>>>>>>> -v 3
>>>>>>>>> DEBUG 1: [Source 1 of 1] ADECK Source:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003,
>>>>>>>>> Model Suffix: (nul)
>>>>>>>>> DEBUG 1: [Source 1 of 1] BDECK Source:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003,
>>>>>>>>> Model Suffix: (nul)
>>>>>>>>> DEBUG 1: Config File Default:
>>>>>>>>> /opt/MET7/share/met/config/TCPairsConfig_default
>>>>>>>>> DEBUG 1: Config File User:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
>>>>>>>>> DEBUG 1: Distance to land file: /opt/MET7/share/met/tc_data/
>>>>>>>>> dland_global_tenth_degree.nc
>>>>>>>>> DEBUG 2: Processing 1 BDECK file(s).
>>>>>>>>> WARNING:
>>>>>>>>> WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
>>>>>>>>> WARNING:
>>>>>>>>> WARNING:
>>>>>>>>> WARNING: int parse_lon(const char *) -> bad longitude ...
>> "687N"
>>>>>>>>> WARNING:
>>>>>>>>> WARNING:
>>>>>>>>> WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
>>>>>>>>> WARNING:
>>>>>>>>> WARNING:
>>>>>>>>> WARNING: int parse_lon(const char *) -> bad longitude ...
>> "701N"
>>>>>>>>> WARNING:
>>>>>>>>> WARNING:
>>>>>>>>> WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
>>>>>>>>> WARNING:
>>>>>>>>> WARNING:
>>>>>>>>> WARNING: int parse_lon(const char *) -> bad longitude ...
>> "711N"
>>>>>>>>> WARNING:
>>>>>>>>> WARNING:
>>>>>>>>> WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
>>>>>>>>> WARNING:
>>>>>>>>> WARNING:
>>>>>>>>> WARNING: int parse_lon(const char *) -> bad longitude ...
>> "720N"
>>>>>>>>> WARNING:
>>>>>>>>> WARNING:
>>>>>>>>> WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
>>>>>>>>> WARNING:
>>>>>>>>> WARNING:
>>>>>>>>> WARNING: int parse_lon(const char *) -> bad longitude ...
>> "730N"
>>>>>>>>> WARNING:
>>>>>>>>> WARNING:
>>>>>>>>> WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
>>>>>>>>> WARNING:
>>>>>>>>> WARNING:
>>>>>>>>> WARNING: int parse_lon(const char *) -> bad longitude ...
>> "741N"
>>>>>>>>> WARNING:
>>>>>>>>> WARNING:
>>>>>>>>> WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
>>>>>>>>> WARNING:
>>>>>>>>> WARNING:
>>>>>>>>> WARNING: int parse_lon(const char *) -> bad longitude ...
>> "751N"
>>>>>>>>> WARNING:
>>>>>>>>> WARNING:
>>>>>>>>> WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
>>>>>>>>> WARNING:
>>>>>>>>> WARNING:
>>>>>>>>> WARNING: int parse_lon(const char *) -> bad longitude ...
>> "760N"
>>>>>>>>> WARNING:
>>>>>>>>> WARNING:
>>>>>>>>> WARNING: int parse_lat(const char *) -> bad latitude ...
"000"
>>>>>>>>> WARNING:
>>>>>>>>> WARNING:
>>>>>>>>> WARNING: int parse_lon(const char *) -> bad longitude ...
>> "764N"
>>>>>>>>> WARNING:
>>>>>>>>> DEBUG 3: Used 9 of 9 lines read from 1 file(s).
>>>>>>>>> DEBUG 3: Identified 9 track(s).
>>>>>>>>> DEBUG 2: Found 9 BDECK track(s).
>>>>>>>>> DEBUG 2: Processing 1 ADECK file(s).
>>>>>>>>> DEBUG 3: Used 0 of 22 lines read from 1 file(s).
>>>>>>>>> DEBUG 3: Identified 0 track(s).
>>>>>>>>> DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
>>>>>>>>> DEBUG 2: Finished adding 0 and replacing 0 Interp12
track(s).
>>>>>>>>> DEBUG 2: Deriving 0 ADECK consensus model(s).
>>>>>>>>> DEBUG 2: Added 0 ADECK consensus tracks(s).
>>>>>>>>> DEBUG 2: Deriving 0 ADECK lag model(s).
>>>>>>>>> DEBUG 2: Added 0 ADECK lag tracks(s).
>>>>>>>>> DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
>>>>>>>>> DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
>>>>>>>>> DEBUG 2: Filtering 0 ADECK tracks based on config file
>> settings.
>>>>>>>>> DEBUG 3: Total tracks read = 0
>>>>>>>>> DEBUG 3: Total tracks kept = 0
>>>>>>>>> DEBUG 3: Rejected for storm name = 0
>>>>>>>>> DEBUG 3: Rejected for valid time = 0
>>>>>>>>> DEBUG 3: Rejected for required lead times = 0
>>>>>>>>> DEBUG 3: Rejected for init mask = 0
>>>>>>>>> DEBUG 3: Rejected for valid mask = 0
>>>>>>>>> DEBUG 2: Matching 0 ADECK tracks to 9 BDECK tracks.
>>>>>>>>> DEBUG 1: Watch/Warning file:
>>>>> /opt/MET7/share/met/tc_data/wwpts_us.txt
>>>>>>>>> DEBUG 1: Output file:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt.tcst
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Rosalyn MacCracken
>>>>>>>>> Support Scientist
>>>>>>>>>
>>>>>>>>> Ocean Applications Branch
>>>>>>>>> NOAA/NWS Ocean Prediction Center
>>>>>>>>> NCWCP
>>>>>>>>> 5830 University Research Ct
>>>>>>>>> College Park, MD 20740-3818
>>>>>>>>>
>>>>>>>>> (p) 301-683-1551
>>>>>>>>> rosalyn.maccracken at noaa.gov
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Rosalyn MacCracken
>>>>>>> Support Scientist
>>>>>>>
>>>>>>> Ocean Applications Branch
>>>>>>> NOAA/NWS Ocean Prediction Center
>>>>>>> NCWCP
>>>>>>> 5830 University Research Ct
>>>>>>> College Park, MD 20740-3818
>>>>>>>
>>>>>>> (p) 301-683-1551
>>>>>>> rosalyn.maccracken at noaa.gov
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Rosalyn MacCracken
>>>>>> Support Scientist
>>>>>>
>>>>>> Ocean Applications Branch
>>>>>> NOAA/NWS Ocean Prediction Center
>>>>>> NCWCP
>>>>>> 5830 University Research Ct
>>>>>> College Park, MD 20740-3818
>>>>>>
>>>>>> (p) 301-683-1551
>>>>>> rosalyn.maccracken at noaa.gov
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Rosalyn MacCracken
>>>> Support Scientist
>>>>
>>>> Ocean Applications Branch
>>>> NOAA/NWS Ocean Prediction Center
>>>> NCWCP
>>>> 5830 University Research Ct
>>>> College Park, MD 20740-3818
>>>>
>>>> (p) 301-683-1551
>>>> rosalyn.maccracken at noaa.gov
>>>>
>>>>
>>>
>>>
>>
>> --
>> Rosalyn MacCracken
>> Support Scientist
>>
>> Ocean Applications Branch
>> NOAA/NWS Ocean Prediction Center
>> NCWCP
>> 5830 University Research Ct
>> College Park, MD 20740-3818
>>
>> (p) 301-683-1551
>> rosalyn.maccracken at noaa.gov
>>
>>
>
------------------------------------------------
Subject: Help with using TC_pairs
From: Rosalyn MacCracken - NOAA Affiliate
Time: Tue Nov 27 04:49:49 2018
Hi John,
I uploaded the original files (Adeck = aml*, Bdeck=bml*, without the
*modified.dat ending), the modified files (with the *modified.dat
ending)
and the config file into my ftp directory (*cd
incoming/irap/met_help/maccracken). *Let me know what you find.
Thanks for your help!
Roz
On Mon, Nov 26, 2018 at 5:22 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Roz,
>
> Yes, the first 8 columns are required. The additional ones are
optional.
>
> Looking at your log output, I see that tc_pairs is skipping over all
642
> input lines. And the question is why? If you send me your latest
ADeck,
> BDeck, and TC-Pairs config files, I could run it here and try to
figure out
> why it isn't defining any ADECK tracks.
>
> Thanks,
> John
>
> On Mon, Nov 26, 2018 at 12:23 PM Rosalyn MacCracken - NOAA Affiliate
via RT
> <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
> >
> > So, can I just have 8 columns for the input file, or, do I need to
make
> > columns of -9999 or NaN for all the rest of the columns?
> >
> > I experimented, and just wrote out 10 columns for the input file:
> > agfso_trackdf_mod =
> >
> >
>
agfso_trackdf[['basin_name','storm_id','model_start_date','const','model','fcst_lead_time','vort_lat_cent','vort_lon_cent','vort_10m_max',
> > 'vort_min_mslp','type','thresh_wspd']]
> >
> > because I wanted stats for these variables, and I got the output
below.
> > So, maybe it's working, but, the config file needs to be
different? Any
> > other ideas of what I should try doing/looking at?
> >
> > Roz
> >
> > Output:
> > DEBUG 1: [Source 1 of 1] ADECK Source:
> >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018083118.gfso.0053_modified.dat,
> > Model Suffix: _EXP
> > DEBUG 1: [Source 1 of 1] BDECK Source:
> >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018083118.gfso.0053_modified.dat,
> > Model Suffix: (nul)
> > DEBUG 1: Config File Default:
> > /opt/MET7/share/met/config/TCPairsConfig_default
> > DEBUG 1: Config File User:
> >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > DEBUG 1: Distance to land file: /opt/MET7/share/met/tc_data/
> > dland_global_tenth_degree.nc
> > DEBUG 4:
> > DEBUG 4: Latitude/Longitude Grid Data:
> > DEBUG 4: lat_ll: -90
> > DEBUG 4: lon_ll: 180
> > DEBUG 4: delta_lat: 0.1
> > DEBUG 4: delta_lon: 0.1
> > DEBUG 4: Nlat: 1801
> > DEBUG 4: Nlon: 3601
> > DEBUG 4:
> > DEBUG 2: Processing 1 BDECK file(s).
> > DEBUG 4: [File 1 of 1] Used 30 of 30 lines read from file
> >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018083118.gfso.0053_modified.dat"
> > DEBUG 3: Used 30 of 30 lines read from 1 file(s).
> > DEBUG 3: Identified 1 track(s).
> > DEBUG 4: [Track 1 of 1] TrackInfo: StormId = "ML0800532018",
IsBest =
> > false, IsOper = false, CheckAnly = true, IsAnly = true, Basin =
"ML",
> > Cyclone = "080053", StormName = "", TechniqueNumber = 3, Technique
=
> > "GFSO", Initials = "", InitTime = NA, MinValidTime =
20180805_060000,
> > MaxValidTime = 20180812_120000, NPoints = 30, NAlloc = 100,
NTrackLines
> = 0
> > DEBUG 2: Found 1 BDECK track(s).
> > DEBUG 2: Processing 1 ADECK file(s).
> > DEBUG 4: [File 1 of 1] Used 0 of 642 lines read from file
> >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018083118.gfso.0053_modified.dat"
> > DEBUG 3: Used 0 of 642 lines read from 1 file(s).
> > DEBUG 3: Identified 0 track(s).
> > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> > DEBUG 2: Finished adding 0 and replacing 0 Interp12 track(s).
> > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > DEBUG 2: Deriving 0 ADECK lag model(s).
> > DEBUG 2: Added 0 ADECK lag tracks(s).
> > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> > DEBUG 2: Filtering 0 ADECK tracks based on config file settings.
> > DEBUG 3: Total tracks read = 0
> > DEBUG 3: Total tracks kept = 0
> > DEBUG 3: Rejected for storm name = 0
> > DEBUG 3: Rejected for valid time = 0
> > DEBUG 3: Rejected for required lead times = 0
> > DEBUG 3: Rejected for init mask = 0
> > DEBUG 3: Rejected for valid mask = 0
> > DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
> > DEBUG 1: Watch/Warning file:
/opt/MET7/share/met/tc_data/wwpts_us.txt
> > DEBUG 1: Output file:
> >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018083118.gfso.0053_test_modified_output.txt.tcst
> >
> >
> > On Mon, Nov 26, 2018 at 2:06 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Roz,
> > >
> > > Page 334 of the user's guide (
> > >
>
https://dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v8.0.pdf
> > )
> > > mentions that only the first 8 ATCF columns are required. If
> additional
> > > columns are present in the input ATCF files, then tc_pairs will
> interpret
> > > them as specified by the ATCF format.
> > >
> > > So if the storm name column is present, MET in interpret those
values
> as
> > > the storm name.
> > >
> > > For Guang Ping's data, probably best to have that column empty.
> > >
> > > John
> > >
> > >
> > > On Mon, Nov 26, 2018 at 11:51 AM Rosalyn MacCracken - NOAA
Affiliate
> via
> > RT
> > > <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
>
> > > >
> > > > Hi John,
> > > >
> > > > Ok, I will ignore the suffix error. I do have numeric values
in
> > > > STORMNAME. I wonder if in the modified files I have, this
column is
> > not
> > > > actually STORMNAME, but, some other variable. Now, I don't
have the
> > > number
> > > > of columns that the NRL website has. Will that be a problem?
> Should I
> > > > create dummy columns?
> > > >
> > > > Roz
> > > >
> > > > On Mon, Nov 26, 2018 at 1:14 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Roz,
> > > > >
> > > > > I see that you're wondering about this log message:
> > > > > *Model Suffix: (nul)*
> > > > >
> > > > > Everything is fine, you do not need to be concerned about
this.
> > > Here's a
> > > > > line taken from the usage statement for the tc_pairs tool:
> > > > > *Note: The "-adeck", "-edeck", and "-bdeck" options may
include
> > > > > "suffix=string" to modify the model names from that source.*
> > > > >
> > > > > Since you didn't specify "suffix=string" on the command
line, the
> > > suffix
> > > > > remains the default value of an empty string, i.e. "(nul)".
So why
> > > would
> > > > > you ever want to use a suffix? Sometimes data is messy. We
were
> > > running
> > > > > tc_pairs to evaluate track data from multiple sources and we
found
> > that
> > > > > some model names we reused in both sources. Let's say the
OFCL
> > > forecast
> > > > > showed up in both stream 1.0 and stream 1.5 data.
Specifying a
> > suffix
> > > > such
> > > > > as "suffix=_STREAM1.5" changes the model name from "OFCL" to
> > > > > "OFCL_STREAM1.5". And that enables us to distinguish
between those
> > > > tracks
> > > > > in the output.
> > > > >
> > > > > But uou probably don't have any need for this option in your
> current
> > > > work.
> > > > >
> > > > > However, I am concerned about these log messages:
> > > > > DEBUG 4: Updating storm name from "-287" to "-305" for
> ML0900022018.
> > > > > DEBUG 4: Updating storm name from "-305" to "-255" for
> ML0900022018.
> > > > > DEBUG 4: Updating storm name from "-255" to "-203" for
> ML0900022018.
> > > > > DEBUG 4: Updating storm name from "-203" to "-205" for
> ML0900022018.
> > > > > DEBUG 4: Updating storm name from "-205" to "-217" for
> ML0900022018.
> > > > > DEBUG 4: Updating storm name from "-217" to "-211" for
> ML0900022018.
> > > > > DEBUG 4: Updating storm name from "-211" to "-182" for
> ML0900022018.
> > > > > DEBUG 4: Updating storm name from "-182" to "-138" for
> ML0900022018.
> > > > >
> > > > > You likely have numeric values showing up in the "STORMNAME"
ATCF
> > > column:
> > > > >
> https://www.nrlmry.navy.mil/atcf_web/docs/database/new/abdeck.txt
> > > > >
> > > > > I'd recommend looking into that.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Mon, Nov 26, 2018 at 8:44 AM Rosalyn MacCracken - NOAA
Affiliate
> > via
> > > > RT
> > > > > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
> > > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > I figured out an easy way to read in and write out Guang
Ping's
> > > > modified
> > > > > > ATCF file, so that it:
> > > > > > 1) Removes the 3rd column
> > > > > > 2) Replaces -99 with -9999 in column 21 and 24 (after the
3rd
> > column
> > > > is
> > > > > > removed)
> > > > > > 3) Updates column 2 to include month+cyclone number.
(You had
> > said
> > > to
> > > > > do
> > > > > > year+cyclone #, but the tc_pairs_wrapper.py said to do
> > > month+cyclone#)
> > > > > >
> > > > > > I changed Adeck and Bdeck.
> > > > > > Just a couple lines of my file:
> > > > > >
> > > > > > ML, 090002, 2018090100, 03, GFSO, 000, 511S, 897W, 40,
976,
> XX,
> > > 34,
> > > > > > NEQ, 0625, 0630, 0000, 0226, 978, 68, -9999, 123, 155,
-9999,
> > > -9999,
> > > > > > -9999, -297, -394, -287, -458, 0073
> > > > > > ML, 090002, 2018090106, 03, GFSO, 000, 537S, 854W, 44,
964,
> XX,
> > > 34,
> > > > > > NEQ, 0640, 0645, 0000, 0297, 969, 108, -9999, 119, 147,
-9999,
> > > -9999,
> > > > > > -9999, -403, -648, -305, -394, 0055
> > > > > >
> > > > > > So, I've run tc_pairs again and my output is below. I
don't
> > > understand
> > > > > the
> > > > > > Model Suffix: (nul). Is the input format still wrong?
> > > > > >
> > > > > > Roz
> > > > > >
> > > > > > Output:
> > > > > > /opt/MET7/bin/tc_pairs -adeck
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat
> > > > > > -bdeck
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat
> > > > > > -config
> > > >
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > > > > > -out
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt
> > > > > > -log
> > > > >
>
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> > > > > > -v 4
> > > > > > DEBUG 1: [Source 1 of 1] ADECK Source:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat,
> > > > > > Model Suffix: (nul)
> > > > > > DEBUG 1: [Source 1 of 1] BDECK Source:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat,
> > > > > > Model Suffix: (nul)
> > > > > > DEBUG 1: Config File Default:
> > > > > > /opt/MET7/share/met/config/TCPairsConfig_default
> > > > > > DEBUG 1: Config File User:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > > > > > DEBUG 1: Distance to land file:
/opt/MET7/share/met/tc_data/
> > > > > > dland_global_tenth_degree.nc
> > > > > > DEBUG 4:
> > > > > > DEBUG 4: Latitude/Longitude Grid Data:
> > > > > > DEBUG 4: lat_ll: -90
> > > > > > DEBUG 4: lon_ll: 180
> > > > > > DEBUG 4: delta_lat: 0.1
> > > > > > DEBUG 4: delta_lon: 0.1
> > > > > > DEBUG 4: Nlat: 1801
> > > > > > DEBUG 4: Nlon: 3601
> > > > > > DEBUG 4:
> > > > > > DEBUG 2: Processing 1 BDECK file(s).
> > > > > > DEBUG 4: Updating storm name from "-287" to "-305" for
> > ML0900022018.
> > > > > > DEBUG 4: Updating storm name from "-305" to "-255" for
> > ML0900022018.
> > > > > > DEBUG 4: Updating storm name from "-255" to "-203" for
> > ML0900022018.
> > > > > > DEBUG 4: Updating storm name from "-203" to "-205" for
> > ML0900022018.
> > > > > > DEBUG 4: Updating storm name from "-205" to "-217" for
> > ML0900022018.
> > > > > > DEBUG 4: Updating storm name from "-217" to "-211" for
> > ML0900022018.
> > > > > > DEBUG 4: Updating storm name from "-211" to "-182" for
> > ML0900022018.
> > > > > > DEBUG 4: Updating storm name from "-182" to "-138" for
> > ML0900022018.
> > > > > > DEBUG 4: [File 1 of 1] Used 9 of 9 lines read from file
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat"
> > > > > > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
> > > > > > DEBUG 3: Identified 1 track(s).
> > > > > > DEBUG 4: [Track 1 of 1] TrackInfo: StormId =
"ML0900022018",
> > IsBest =
> > > > > > false, IsOper = false, CheckAnly = true, IsAnly = true,
Basin =
> > "ML",
> > > > > > Cyclone = "090002", StormName = "-138", TechniqueNumber =
3,
> > > Technique
> > > > =
> > > > > > "GFSO", Initials = "-9999", InitTime = NA, MinValidTime =
> > > > > 20180901_000000,
> > > > > > MaxValidTime = 20180903_000000, NPoints = 9, NAlloc = 100,
> > > NTrackLines
> > > > =
> > > > > 0
> > > > > > DEBUG 2: Found 1 BDECK track(s).
> > > > > > DEBUG 2: Processing 1 ADECK file(s).
> > > > > > DEBUG 4: [File 1 of 1] Used 0 of 21 lines read from file
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat"
> > > > > > DEBUG 3: Used 0 of 21 lines read from 1 file(s).
> > > > > > DEBUG 3: Identified 0 track(s).
> > > > > > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> > > > > > DEBUG 2: Finished adding 0 and replacing 0 Interp12
track(s).
> > > > > > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > > > > > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > > > > > DEBUG 2: Deriving 0 ADECK lag model(s).
> > > > > > DEBUG 2: Added 0 ADECK lag tracks(s).
> > > > > > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> > > > > > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> > > > > > DEBUG 2: Filtering 0 ADECK tracks based on config file
settings.
> > > > > > DEBUG 3: Total tracks read = 0
> > > > > > DEBUG 3: Total tracks kept = 0
> > > > > > DEBUG 3: Rejected for storm name = 0
> > > > > > DEBUG 3: Rejected for valid time = 0
> > > > > > DEBUG 3: Rejected for required lead times = 0
> > > > > > DEBUG 3: Rejected for init mask = 0
> > > > > > DEBUG 3: Rejected for valid mask = 0
> > > > > > DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
> > > > > > DEBUG 1: Watch/Warning file:
> > /opt/MET7/share/met/tc_data/wwpts_us.txt
> > > > > > DEBUG 1: Output file:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt.tcst
> > > > > >
> > > > > > On Wed, Nov 21, 2018 at 9:16 AM Rosalyn MacCracken - NOAA
> > Affiliate <
> > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > >
> > > > > > > Hi John,
> > > > > > >
> > > > > > > Yes, that makes sense, and yes, I am using Guang Pings'
> modified
> > > > files.
> > > > > > > I'm not sure how to run tc_pairs_wrapper.py, though. I
think I
> > > need
> > > > > some
> > > > > > > type of *.conf file to pass in the files I need, or
something
> > like
> > > > > that.
> > > > > > > I'm in the process of reading through the code now. I
wonder
> if
> > I
> > > > > could
> > > > > > > just pull out the function process_non_atcf, to run in a
> > > stand-alone
> > > > > > mode.
> > > > > > > I wish there was some sort of documentation with this
code...
> > > > > > >
> > > > > > > And, so, once I can read these files, I can use tc_pairs
to
> > create
> > > > the
> > > > > > > forecast error statistics, and then use tc_stats to
aggregate
> the
> > > > files
> > > > > > and
> > > > > > > get the mean, etc, for a period of time, correct? Was
there
> some
> > > > sort
> > > > > of
> > > > > > > plotting scripts for the output files?
> > > > > > >
> > > > > > > So, with tc_pairs, I can define a region, like I did
with
> > > point_stat,
> > > > > > > correct? Can I use the same *.nc files, or do I use the
> original
> > > > > *.poly
> > > > > > > files?
> > > > > > >
> > > > > > > Roz
> > > > > > >
> > > > > > > On Tue, Nov 20, 2018 at 7:53 PM John Halley Gotway via
RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > >> Hi Roz,
> > > > > > >>
> > > > > > >> Yes, I see you’re using the modified ATCF format that
Guang
> Ping
> > > > uses.
> > > > > > >> You’re correct that tc_pairs doesn’t parse this
directly since
> > it
> > > > > > differs
> > > > > > >> slightly from the ATCF format.
> > > > > > >>
> > > > > > >> When we’ve processed this data in the past, we first
> > pre-processed
> > > > it
> > > > > > in a
> > > > > > >> shell script to:
> > > > > > >>
> > > > > > >> 1. Replace the -99 value with -9999 for missing data.
> > > > > > >> 2. Remove the 3rd column
> > > > > > >> 3. Updating the 2nd column to include the year and
cyclone
> > number.
> > > > > > >>
> > > > > > >> And that enable tc_pairs to read it.
> > > > > > >>
> > > > > > >> This is actually Done in python by one of the use cases
in
> > > METplus.
> > > > > > >>
> > > > > > >> Take a look at this file:
> > > > > > >>
> > >
https://github.com/NCAR/METplus/blob/master/ush/tc_pairs_wrapper.py
> > > > > > >>
> > > > > > >> At the function named:
> > > > > > >> process_non_atcf
> > > > > > >>
> > > > > > >> I haven’t run this using METplus scripts myself, but
all the
> > > pieces
> > > > of
> > > > > > >> logic are there, and it should be possible.
> > > > > > >>
> > > > > > >> Make sense?
> > > > > > >>
> > > > > > >> Thanks
> > > > > > >> John
> > > > > > >>
> > > > > > >> On Tue, Nov 20, 2018 at 1:13 PM Rosalyn MacCracken -
NOAA
> > > Affiliate
> > > > > via
> > > > > > >> RT <
> > > > > > >> met_help at ucar.edu> wrote:
> > > > > > >>
> > > > > > >> >
> > > > > > >> > Tue Nov 20 13:13:05 2018: Request 87885 was acted
upon.
> > > > > > >> > Transaction: Ticket created by
rosalyn.maccracken at noaa.gov
> > > > > > >> > Queue: met_help
> > > > > > >> > Subject: Help with using TC_pairs
> > > > > > >> > Owner: Nobody
> > > > > > >> > Requestors: rosalyn.maccracken at noaa.gov
> > > > > > >> > Status: new
> > > > > > >> > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
> > > > > > >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > Hi,
> > > > > > >> >
> > > > > > >> > I'm trying to use TC_pairs for extra-tropical
cyclones
> > > (basin=ML).
> > > > > I
> > > > > > >> think
> > > > > > >> > the format of the ATCF data is the issue, but, I'm
not sure.
> > I
> > > > have
> > > > > > an
> > > > > > >> > Adeck and a Bdeck file, where a sample of one of the
lines
> is:
> > > > > > >> > ML, 0003, 2018090100_F000_687N_0159W_FOF, 2018090100,
03,
> > GFSO,
> > > > 000,
> > > > > > >> 687N,
> > > > > > >> > 159W, 32, 987, XX, 34, NEQ, 0301, 0338, 0000,
0000, 988,
> > > 52,
> > > > > > -99,
> > > > > > >> > 330, 86, -99, -9999, -9999, 175, 202, 137, 135,
0081
> > > > > > >> >
> > > > > > >> > This is slightly different from the NRL ATCF format
on their
> > > > > website,
> > > > > > >> such
> > > > > > >> > as the second column. Anyway, I configured the
config file
> > very
> > > > > > simply,
> > > > > > >> > (model=GFS, basin=ML), and ran tc_pairs, and got the
output
> to
> > > the
> > > > > > >> screen
> > > > > > >> > below.
> > > > > > >> >
> > > > > > >> > Is there a way to work around the diferent ATCF
format?
> I've
> > > > > > attached a
> > > > > > >> > sample of my Adeck and Bdeck files.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> >
> > > > > > >> > Roz
> > > > > > >> >
> > > > > > >> > Output to the screen:
> > > > > > >> > /opt/MET7/bin/tc_pairs -adeck
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003
> > > > > > >> > -bdeck
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003
> > > > > > >> > -config
> > > > > > >> >
> > > > > >
> > >
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> > > > > > >> > -out
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt
> > > > > > >> > -log
> > > > > > >>
> > > >
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> > > > > > >> > -v 3
> > > > > > >> > DEBUG 1: [Source 1 of 1] ADECK Source:
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003,
> > > > > > >> > Model Suffix: (nul)
> > > > > > >> > DEBUG 1: [Source 1 of 1] BDECK Source:
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003,
> > > > > > >> > Model Suffix: (nul)
> > > > > > >> > DEBUG 1: Config File Default:
> > > > > > >> > /opt/MET7/share/met/config/TCPairsConfig_default
> > > > > > >> > DEBUG 1: Config File User:
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> > > > > > >> > DEBUG 1: Distance to land file:
/opt/MET7/share/met/tc_data/
> > > > > > >> > dland_global_tenth_degree.nc
> > > > > > >> > DEBUG 2: Processing 1 BDECK file(s).
> > > > > > >> > WARNING:
> > > > > > >> > WARNING: int parse_lat(const char *) -> bad latitude
...
> "000"
> > > > > > >> > WARNING:
> > > > > > >> > WARNING:
> > > > > > >> > WARNING: int parse_lon(const char *) -> bad longitude
...
> > "687N"
> > > > > > >> > WARNING:
> > > > > > >> > WARNING:
> > > > > > >> > WARNING: int parse_lat(const char *) -> bad latitude
...
> "000"
> > > > > > >> > WARNING:
> > > > > > >> > WARNING:
> > > > > > >> > WARNING: int parse_lon(const char *) -> bad longitude
...
> > "701N"
> > > > > > >> > WARNING:
> > > > > > >> > WARNING:
> > > > > > >> > WARNING: int parse_lat(const char *) -> bad latitude
...
> "000"
> > > > > > >> > WARNING:
> > > > > > >> > WARNING:
> > > > > > >> > WARNING: int parse_lon(const char *) -> bad longitude
...
> > "711N"
> > > > > > >> > WARNING:
> > > > > > >> > WARNING:
> > > > > > >> > WARNING: int parse_lat(const char *) -> bad latitude
...
> "000"
> > > > > > >> > WARNING:
> > > > > > >> > WARNING:
> > > > > > >> > WARNING: int parse_lon(const char *) -> bad longitude
...
> > "720N"
> > > > > > >> > WARNING:
> > > > > > >> > WARNING:
> > > > > > >> > WARNING: int parse_lat(const char *) -> bad latitude
...
> "000"
> > > > > > >> > WARNING:
> > > > > > >> > WARNING:
> > > > > > >> > WARNING: int parse_lon(const char *) -> bad longitude
...
> > "730N"
> > > > > > >> > WARNING:
> > > > > > >> > WARNING:
> > > > > > >> > WARNING: int parse_lat(const char *) -> bad latitude
...
> "000"
> > > > > > >> > WARNING:
> > > > > > >> > WARNING:
> > > > > > >> > WARNING: int parse_lon(const char *) -> bad longitude
...
> > "741N"
> > > > > > >> > WARNING:
> > > > > > >> > WARNING:
> > > > > > >> > WARNING: int parse_lat(const char *) -> bad latitude
...
> "000"
> > > > > > >> > WARNING:
> > > > > > >> > WARNING:
> > > > > > >> > WARNING: int parse_lon(const char *) -> bad longitude
...
> > "751N"
> > > > > > >> > WARNING:
> > > > > > >> > WARNING:
> > > > > > >> > WARNING: int parse_lat(const char *) -> bad latitude
...
> "000"
> > > > > > >> > WARNING:
> > > > > > >> > WARNING:
> > > > > > >> > WARNING: int parse_lon(const char *) -> bad longitude
...
> > "760N"
> > > > > > >> > WARNING:
> > > > > > >> > WARNING:
> > > > > > >> > WARNING: int parse_lat(const char *) -> bad latitude
...
> "000"
> > > > > > >> > WARNING:
> > > > > > >> > WARNING:
> > > > > > >> > WARNING: int parse_lon(const char *) -> bad longitude
...
> > "764N"
> > > > > > >> > WARNING:
> > > > > > >> > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
> > > > > > >> > DEBUG 3: Identified 9 track(s).
> > > > > > >> > DEBUG 2: Found 9 BDECK track(s).
> > > > > > >> > DEBUG 2: Processing 1 ADECK file(s).
> > > > > > >> > DEBUG 3: Used 0 of 22 lines read from 1 file(s).
> > > > > > >> > DEBUG 3: Identified 0 track(s).
> > > > > > >> > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> > > > > > >> > DEBUG 2: Finished adding 0 and replacing 0 Interp12
> track(s).
> > > > > > >> > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > > > > > >> > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > > > > > >> > DEBUG 2: Deriving 0 ADECK lag model(s).
> > > > > > >> > DEBUG 2: Added 0 ADECK lag tracks(s).
> > > > > > >> > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> > > > > > >> > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> > > > > > >> > DEBUG 2: Filtering 0 ADECK tracks based on config
file
> > settings.
> > > > > > >> > DEBUG 3: Total tracks read = 0
> > > > > > >> > DEBUG 3: Total tracks kept = 0
> > > > > > >> > DEBUG 3: Rejected for storm name = 0
> > > > > > >> > DEBUG 3: Rejected for valid time = 0
> > > > > > >> > DEBUG 3: Rejected for required lead times = 0
> > > > > > >> > DEBUG 3: Rejected for init mask = 0
> > > > > > >> > DEBUG 3: Rejected for valid mask = 0
> > > > > > >> > DEBUG 2: Matching 0 ADECK tracks to 9 BDECK tracks.
> > > > > > >> > DEBUG 1: Watch/Warning file:
> > > > > /opt/MET7/share/met/tc_data/wwpts_us.txt
> > > > > > >> > DEBUG 1: Output file:
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt.tcst
> > > > > > >> >
> > > > > > >> > --
> > > > > > >> > Rosalyn MacCracken
> > > > > > >> > Support Scientist
> > > > > > >> >
> > > > > > >> > Ocean Applications Branch
> > > > > > >> > NOAA/NWS Ocean Prediction Center
> > > > > > >> > NCWCP
> > > > > > >> > 5830 University Research Ct
> > > > > > >> > College Park, MD 20740-3818
> > > > > > >> >
> > > > > > >> > (p) 301-683-1551
> > > > > > >> > rosalyn.maccracken at noaa.gov
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > > > > --
> > > > > > > Rosalyn MacCracken
> > > > > > > Support Scientist
> > > > > > >
> > > > > > > Ocean Applications Branch
> > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > NCWCP
> > > > > > > 5830 University Research Ct
> > > > > > > College Park, MD 20740-3818
> > > > > > >
> > > > > > > (p) 301-683-1551
> > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Rosalyn MacCracken
> > > > > > Support Scientist
> > > > > >
> > > > > > Ocean Applications Branch
> > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > NCWCP
> > > > > > 5830 University Research Ct
> > > > > > College Park, MD 20740-3818
> > > > > >
> > > > > > (p) 301-683-1551
> > > > > > rosalyn.maccracken at noaa.gov
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > Rosalyn MacCracken
> > > > Support Scientist
> > > >
> > > > Ocean Applications Branch
> > > > NOAA/NWS Ocean Prediction Center
> > > > NCWCP
> > > > 5830 University Research Ct
> > > > College Park, MD 20740-3818
> > > >
> > > > (p) 301-683-1551
> > > > rosalyn.maccracken at noaa.gov
> > > >
> > > >
> > >
> > >
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applications Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
> >
>
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Help with using TC_pairs
From: John Halley Gotway
Time: Tue Nov 27 09:43:40 2018
Roz,
Thanks for posting your sample data. I found the problem in your
TCPairsConfig file. Change...
FROM: storm_id = [ "" ];
TO: storm_id = [ ];
The storm_id entry is an array of storm ID's to match. An empty array
is
just square brackets with nothing inside ([ ]). Since you had
double-quotes inside, that's an array of length 1 containing an empty
string.
Thanks,
John
On Tue, Nov 27, 2018 at 4:50 AM Rosalyn MacCracken - NOAA Affiliate
via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
>
> Hi John,
>
> I uploaded the original files (Adeck = aml*, Bdeck=bml*, without the
> *modified.dat ending), the modified files (with the *modified.dat
ending)
> and the config file into my ftp directory (*cd
> incoming/irap/met_help/maccracken). *Let me know what you find.
>
> Thanks for your help!
>
> Roz
>
> On Mon, Nov 26, 2018 at 5:22 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Roz,
> >
> > Yes, the first 8 columns are required. The additional ones are
optional.
> >
> > Looking at your log output, I see that tc_pairs is skipping over
all 642
> > input lines. And the question is why? If you send me your latest
ADeck,
> > BDeck, and TC-Pairs config files, I could run it here and try to
figure
> out
> > why it isn't defining any ADECK tracks.
> >
> > Thanks,
> > John
> >
> > On Mon, Nov 26, 2018 at 12:23 PM Rosalyn MacCracken - NOAA
Affiliate via
> RT
> > <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
> > >
> > > So, can I just have 8 columns for the input file, or, do I need
to make
> > > columns of -9999 or NaN for all the rest of the columns?
> > >
> > > I experimented, and just wrote out 10 columns for the input
file:
> > > agfso_trackdf_mod =
> > >
> > >
> >
>
agfso_trackdf[['basin_name','storm_id','model_start_date','const','model','fcst_lead_time','vort_lat_cent','vort_lon_cent','vort_10m_max',
> > > 'vort_min_mslp','type','thresh_wspd']]
> > >
> > > because I wanted stats for these variables, and I got the output
below.
> > > So, maybe it's working, but, the config file needs to be
different?
> Any
> > > other ideas of what I should try doing/looking at?
> > >
> > > Roz
> > >
> > > Output:
> > > DEBUG 1: [Source 1 of 1] ADECK Source:
> > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018083118.gfso.0053_modified.dat,
> > > Model Suffix: _EXP
> > > DEBUG 1: [Source 1 of 1] BDECK Source:
> > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018083118.gfso.0053_modified.dat,
> > > Model Suffix: (nul)
> > > DEBUG 1: Config File Default:
> > > /opt/MET7/share/met/config/TCPairsConfig_default
> > > DEBUG 1: Config File User:
> > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > > DEBUG 1: Distance to land file: /opt/MET7/share/met/tc_data/
> > > dland_global_tenth_degree.nc
> > > DEBUG 4:
> > > DEBUG 4: Latitude/Longitude Grid Data:
> > > DEBUG 4: lat_ll: -90
> > > DEBUG 4: lon_ll: 180
> > > DEBUG 4: delta_lat: 0.1
> > > DEBUG 4: delta_lon: 0.1
> > > DEBUG 4: Nlat: 1801
> > > DEBUG 4: Nlon: 3601
> > > DEBUG 4:
> > > DEBUG 2: Processing 1 BDECK file(s).
> > > DEBUG 4: [File 1 of 1] Used 30 of 30 lines read from file
> > >
> > >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018083118.gfso.0053_modified.dat"
> > > DEBUG 3: Used 30 of 30 lines read from 1 file(s).
> > > DEBUG 3: Identified 1 track(s).
> > > DEBUG 4: [Track 1 of 1] TrackInfo: StormId = "ML0800532018",
IsBest =
> > > false, IsOper = false, CheckAnly = true, IsAnly = true, Basin =
"ML",
> > > Cyclone = "080053", StormName = "", TechniqueNumber = 3,
Technique =
> > > "GFSO", Initials = "", InitTime = NA, MinValidTime =
20180805_060000,
> > > MaxValidTime = 20180812_120000, NPoints = 30, NAlloc = 100,
NTrackLines
> > = 0
> > > DEBUG 2: Found 1 BDECK track(s).
> > > DEBUG 2: Processing 1 ADECK file(s).
> > > DEBUG 4: [File 1 of 1] Used 0 of 642 lines read from file
> > >
> > >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018083118.gfso.0053_modified.dat"
> > > DEBUG 3: Used 0 of 642 lines read from 1 file(s).
> > > DEBUG 3: Identified 0 track(s).
> > > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> > > DEBUG 2: Finished adding 0 and replacing 0 Interp12 track(s).
> > > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > > DEBUG 2: Deriving 0 ADECK lag model(s).
> > > DEBUG 2: Added 0 ADECK lag tracks(s).
> > > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> > > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> > > DEBUG 2: Filtering 0 ADECK tracks based on config file settings.
> > > DEBUG 3: Total tracks read = 0
> > > DEBUG 3: Total tracks kept = 0
> > > DEBUG 3: Rejected for storm name = 0
> > > DEBUG 3: Rejected for valid time = 0
> > > DEBUG 3: Rejected for required lead times = 0
> > > DEBUG 3: Rejected for init mask = 0
> > > DEBUG 3: Rejected for valid mask = 0
> > > DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
> > > DEBUG 1: Watch/Warning file:
/opt/MET7/share/met/tc_data/wwpts_us.txt
> > > DEBUG 1: Output file:
> > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018083118.gfso.0053_test_modified_output.txt.tcst
> > >
> > >
> > > On Mon, Nov 26, 2018 at 2:06 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Roz,
> > > >
> > > > Page 334 of the user's guide (
> > > >
> >
https://dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v8.0.pdf
> > > )
> > > > mentions that only the first 8 ATCF columns are required. If
> > additional
> > > > columns are present in the input ATCF files, then tc_pairs
will
> > interpret
> > > > them as specified by the ATCF format.
> > > >
> > > > So if the storm name column is present, MET in interpret those
values
> > as
> > > > the storm name.
> > > >
> > > > For Guang Ping's data, probably best to have that column
empty.
> > > >
> > > > John
> > > >
> > > >
> > > > On Mon, Nov 26, 2018 at 11:51 AM Rosalyn MacCracken - NOAA
Affiliate
> > via
> > > RT
> > > > <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
> > > > >
> > > > > Hi John,
> > > > >
> > > > > Ok, I will ignore the suffix error. I do have numeric
values in
> > > > > STORMNAME. I wonder if in the modified files I have, this
column
> is
> > > not
> > > > > actually STORMNAME, but, some other variable. Now, I don't
have
> the
> > > > number
> > > > > of columns that the NRL website has. Will that be a
problem?
> > Should I
> > > > > create dummy columns?
> > > > >
> > > > > Roz
> > > > >
> > > > > On Mon, Nov 26, 2018 at 1:14 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Roz,
> > > > > >
> > > > > > I see that you're wondering about this log message:
> > > > > > *Model Suffix: (nul)*
> > > > > >
> > > > > > Everything is fine, you do not need to be concerned about
this.
> > > > Here's a
> > > > > > line taken from the usage statement for the tc_pairs tool:
> > > > > > *Note: The "-adeck", "-edeck", and "-bdeck" options may
include
> > > > > > "suffix=string" to modify the model names from that
source.*
> > > > > >
> > > > > > Since you didn't specify "suffix=string" on the command
line, the
> > > > suffix
> > > > > > remains the default value of an empty string, i.e.
"(nul)". So
> why
> > > > would
> > > > > > you ever want to use a suffix? Sometimes data is messy.
We were
> > > > running
> > > > > > tc_pairs to evaluate track data from multiple sources and
we
> found
> > > that
> > > > > > some model names we reused in both sources. Let's say the
OFCL
> > > > forecast
> > > > > > showed up in both stream 1.0 and stream 1.5 data.
Specifying a
> > > suffix
> > > > > such
> > > > > > as "suffix=_STREAM1.5" changes the model name from "OFCL"
to
> > > > > > "OFCL_STREAM1.5". And that enables us to distinguish
between
> those
> > > > > tracks
> > > > > > in the output.
> > > > > >
> > > > > > But uou probably don't have any need for this option in
your
> > current
> > > > > work.
> > > > > >
> > > > > > However, I am concerned about these log messages:
> > > > > > DEBUG 4: Updating storm name from "-287" to "-305" for
> > ML0900022018.
> > > > > > DEBUG 4: Updating storm name from "-305" to "-255" for
> > ML0900022018.
> > > > > > DEBUG 4: Updating storm name from "-255" to "-203" for
> > ML0900022018.
> > > > > > DEBUG 4: Updating storm name from "-203" to "-205" for
> > ML0900022018.
> > > > > > DEBUG 4: Updating storm name from "-205" to "-217" for
> > ML0900022018.
> > > > > > DEBUG 4: Updating storm name from "-217" to "-211" for
> > ML0900022018.
> > > > > > DEBUG 4: Updating storm name from "-211" to "-182" for
> > ML0900022018.
> > > > > > DEBUG 4: Updating storm name from "-182" to "-138" for
> > ML0900022018.
> > > > > >
> > > > > > You likely have numeric values showing up in the
"STORMNAME" ATCF
> > > > column:
> > > > > >
> > https://www.nrlmry.navy.mil/atcf_web/docs/database/new/abdeck.txt
> > > > > >
> > > > > > I'd recommend looking into that.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Mon, Nov 26, 2018 at 8:44 AM Rosalyn MacCracken - NOAA
> Affiliate
> > > via
> > > > > RT
> > > > > > <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
> >
> > > > > > >
> > > > > > > Hi John,
> > > > > > >
> > > > > > > I figured out an easy way to read in and write out Guang
Ping's
> > > > > modified
> > > > > > > ATCF file, so that it:
> > > > > > > 1) Removes the 3rd column
> > > > > > > 2) Replaces -99 with -9999 in column 21 and 24 (after
the 3rd
> > > column
> > > > > is
> > > > > > > removed)
> > > > > > > 3) Updates column 2 to include month+cyclone number.
(You had
> > > said
> > > > to
> > > > > > do
> > > > > > > year+cyclone #, but the tc_pairs_wrapper.py said to do
> > > > month+cyclone#)
> > > > > > >
> > > > > > > I changed Adeck and Bdeck.
> > > > > > > Just a couple lines of my file:
> > > > > > >
> > > > > > > ML, 090002, 2018090100, 03, GFSO, 000, 511S, 897W, 40,
976,
> > XX,
> > > > 34,
> > > > > > > NEQ, 0625, 0630, 0000, 0226, 978, 68, -9999, 123,
155,
> -9999,
> > > > -9999,
> > > > > > > -9999, -297, -394, -287, -458, 0073
> > > > > > > ML, 090002, 2018090106, 03, GFSO, 000, 537S, 854W, 44,
964,
> > XX,
> > > > 34,
> > > > > > > NEQ, 0640, 0645, 0000, 0297, 969, 108, -9999, 119,
147,
> -9999,
> > > > -9999,
> > > > > > > -9999, -403, -648, -305, -394, 0055
> > > > > > >
> > > > > > > So, I've run tc_pairs again and my output is below. I
don't
> > > > understand
> > > > > > the
> > > > > > > Model Suffix: (nul). Is the input format still wrong?
> > > > > > >
> > > > > > > Roz
> > > > > > >
> > > > > > > Output:
> > > > > > > /opt/MET7/bin/tc_pairs -adeck
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat
> > > > > > > -bdeck
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat
> > > > > > > -config
> > > > >
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > > > > > > -out
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt
> > > > > > > -log
> > > > > >
> >
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> > > > > > > -v 4
> > > > > > > DEBUG 1: [Source 1 of 1] ADECK Source:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat,
> > > > > > > Model Suffix: (nul)
> > > > > > > DEBUG 1: [Source 1 of 1] BDECK Source:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat,
> > > > > > > Model Suffix: (nul)
> > > > > > > DEBUG 1: Config File Default:
> > > > > > > /opt/MET7/share/met/config/TCPairsConfig_default
> > > > > > > DEBUG 1: Config File User:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > > > > > > DEBUG 1: Distance to land file:
/opt/MET7/share/met/tc_data/
> > > > > > > dland_global_tenth_degree.nc
> > > > > > > DEBUG 4:
> > > > > > > DEBUG 4: Latitude/Longitude Grid Data:
> > > > > > > DEBUG 4: lat_ll: -90
> > > > > > > DEBUG 4: lon_ll: 180
> > > > > > > DEBUG 4: delta_lat: 0.1
> > > > > > > DEBUG 4: delta_lon: 0.1
> > > > > > > DEBUG 4: Nlat: 1801
> > > > > > > DEBUG 4: Nlon: 3601
> > > > > > > DEBUG 4:
> > > > > > > DEBUG 2: Processing 1 BDECK file(s).
> > > > > > > DEBUG 4: Updating storm name from "-287" to "-305" for
> > > ML0900022018.
> > > > > > > DEBUG 4: Updating storm name from "-305" to "-255" for
> > > ML0900022018.
> > > > > > > DEBUG 4: Updating storm name from "-255" to "-203" for
> > > ML0900022018.
> > > > > > > DEBUG 4: Updating storm name from "-203" to "-205" for
> > > ML0900022018.
> > > > > > > DEBUG 4: Updating storm name from "-205" to "-217" for
> > > ML0900022018.
> > > > > > > DEBUG 4: Updating storm name from "-217" to "-211" for
> > > ML0900022018.
> > > > > > > DEBUG 4: Updating storm name from "-211" to "-182" for
> > > ML0900022018.
> > > > > > > DEBUG 4: Updating storm name from "-182" to "-138" for
> > > ML0900022018.
> > > > > > > DEBUG 4: [File 1 of 1] Used 9 of 9 lines read from file
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat"
> > > > > > > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
> > > > > > > DEBUG 3: Identified 1 track(s).
> > > > > > > DEBUG 4: [Track 1 of 1] TrackInfo: StormId =
"ML0900022018",
> > > IsBest =
> > > > > > > false, IsOper = false, CheckAnly = true, IsAnly = true,
Basin =
> > > "ML",
> > > > > > > Cyclone = "090002", StormName = "-138", TechniqueNumber
= 3,
> > > > Technique
> > > > > =
> > > > > > > "GFSO", Initials = "-9999", InitTime = NA, MinValidTime
=
> > > > > > 20180901_000000,
> > > > > > > MaxValidTime = 20180903_000000, NPoints = 9, NAlloc =
100,
> > > > NTrackLines
> > > > > =
> > > > > > 0
> > > > > > > DEBUG 2: Found 1 BDECK track(s).
> > > > > > > DEBUG 2: Processing 1 ADECK file(s).
> > > > > > > DEBUG 4: [File 1 of 1] Used 0 of 21 lines read from file
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat"
> > > > > > > DEBUG 3: Used 0 of 21 lines read from 1 file(s).
> > > > > > > DEBUG 3: Identified 0 track(s).
> > > > > > > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> > > > > > > DEBUG 2: Finished adding 0 and replacing 0 Interp12
track(s).
> > > > > > > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > > > > > > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > > > > > > DEBUG 2: Deriving 0 ADECK lag model(s).
> > > > > > > DEBUG 2: Added 0 ADECK lag tracks(s).
> > > > > > > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> > > > > > > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> > > > > > > DEBUG 2: Filtering 0 ADECK tracks based on config file
> settings.
> > > > > > > DEBUG 3: Total tracks read = 0
> > > > > > > DEBUG 3: Total tracks kept = 0
> > > > > > > DEBUG 3: Rejected for storm name = 0
> > > > > > > DEBUG 3: Rejected for valid time = 0
> > > > > > > DEBUG 3: Rejected for required lead times = 0
> > > > > > > DEBUG 3: Rejected for init mask = 0
> > > > > > > DEBUG 3: Rejected for valid mask = 0
> > > > > > > DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
> > > > > > > DEBUG 1: Watch/Warning file:
> > > /opt/MET7/share/met/tc_data/wwpts_us.txt
> > > > > > > DEBUG 1: Output file:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt.tcst
> > > > > > >
> > > > > > > On Wed, Nov 21, 2018 at 9:16 AM Rosalyn MacCracken -
NOAA
> > > Affiliate <
> > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > >
> > > > > > > > Hi John,
> > > > > > > >
> > > > > > > > Yes, that makes sense, and yes, I am using Guang
Pings'
> > modified
> > > > > files.
> > > > > > > > I'm not sure how to run tc_pairs_wrapper.py, though.
I
> think I
> > > > need
> > > > > > some
> > > > > > > > type of *.conf file to pass in the files I need, or
something
> > > like
> > > > > > that.
> > > > > > > > I'm in the process of reading through the code now. I
wonder
> > if
> > > I
> > > > > > could
> > > > > > > > just pull out the function process_non_atcf, to run in
a
> > > > stand-alone
> > > > > > > mode.
> > > > > > > > I wish there was some sort of documentation with this
code...
> > > > > > > >
> > > > > > > > And, so, once I can read these files, I can use
tc_pairs to
> > > create
> > > > > the
> > > > > > > > forecast error statistics, and then use tc_stats to
aggregate
> > the
> > > > > files
> > > > > > > and
> > > > > > > > get the mean, etc, for a period of time, correct? Was
there
> > some
> > > > > sort
> > > > > > of
> > > > > > > > plotting scripts for the output files?
> > > > > > > >
> > > > > > > > So, with tc_pairs, I can define a region, like I did
with
> > > > point_stat,
> > > > > > > > correct? Can I use the same *.nc files, or do I use
the
> > original
> > > > > > *.poly
> > > > > > > > files?
> > > > > > > >
> > > > > > > > Roz
> > > > > > > >
> > > > > > > > On Tue, Nov 20, 2018 at 7:53 PM John Halley Gotway via
RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > >> Hi Roz,
> > > > > > > >>
> > > > > > > >> Yes, I see you’re using the modified ATCF format that
Guang
> > Ping
> > > > > uses.
> > > > > > > >> You’re correct that tc_pairs doesn’t parse this
directly
> since
> > > it
> > > > > > > differs
> > > > > > > >> slightly from the ATCF format.
> > > > > > > >>
> > > > > > > >> When we’ve processed this data in the past, we first
> > > pre-processed
> > > > > it
> > > > > > > in a
> > > > > > > >> shell script to:
> > > > > > > >>
> > > > > > > >> 1. Replace the -99 value with -9999 for missing data.
> > > > > > > >> 2. Remove the 3rd column
> > > > > > > >> 3. Updating the 2nd column to include the year and
cyclone
> > > number.
> > > > > > > >>
> > > > > > > >> And that enable tc_pairs to read it.
> > > > > > > >>
> > > > > > > >> This is actually Done in python by one of the use
cases in
> > > > METplus.
> > > > > > > >>
> > > > > > > >> Take a look at this file:
> > > > > > > >>
> > > >
https://github.com/NCAR/METplus/blob/master/ush/tc_pairs_wrapper.py
> > > > > > > >>
> > > > > > > >> At the function named:
> > > > > > > >> process_non_atcf
> > > > > > > >>
> > > > > > > >> I haven’t run this using METplus scripts myself, but
all the
> > > > pieces
> > > > > of
> > > > > > > >> logic are there, and it should be possible.
> > > > > > > >>
> > > > > > > >> Make sense?
> > > > > > > >>
> > > > > > > >> Thanks
> > > > > > > >> John
> > > > > > > >>
> > > > > > > >> On Tue, Nov 20, 2018 at 1:13 PM Rosalyn MacCracken -
NOAA
> > > > Affiliate
> > > > > > via
> > > > > > > >> RT <
> > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > >>
> > > > > > > >> >
> > > > > > > >> > Tue Nov 20 13:13:05 2018: Request 87885 was acted
upon.
> > > > > > > >> > Transaction: Ticket created by
> rosalyn.maccracken at noaa.gov
> > > > > > > >> > Queue: met_help
> > > > > > > >> > Subject: Help with using TC_pairs
> > > > > > > >> > Owner: Nobody
> > > > > > > >> > Requestors: rosalyn.maccracken at noaa.gov
> > > > > > > >> > Status: new
> > > > > > > >> > Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
> > > > > > > >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > Hi,
> > > > > > > >> >
> > > > > > > >> > I'm trying to use TC_pairs for extra-tropical
cyclones
> > > > (basin=ML).
> > > > > > I
> > > > > > > >> think
> > > > > > > >> > the format of the ATCF data is the issue, but, I'm
not
> sure.
> > > I
> > > > > have
> > > > > > > an
> > > > > > > >> > Adeck and a Bdeck file, where a sample of one of
the lines
> > is:
> > > > > > > >> > ML, 0003, 2018090100_F000_687N_0159W_FOF,
2018090100, 03,
> > > GFSO,
> > > > > 000,
> > > > > > > >> 687N,
> > > > > > > >> > 159W, 32, 987, XX, 34, NEQ, 0301, 0338, 0000,
0000,
> 988,
> > > > 52,
> > > > > > > -99,
> > > > > > > >> > 330, 86, -99, -9999, -9999, 175, 202, 137,
135, 0081
> > > > > > > >> >
> > > > > > > >> > This is slightly different from the NRL ATCF format
on
> their
> > > > > > website,
> > > > > > > >> such
> > > > > > > >> > as the second column. Anyway, I configured the
config
> file
> > > very
> > > > > > > simply,
> > > > > > > >> > (model=GFS, basin=ML), and ran tc_pairs, and got
the
> output
> > to
> > > > the
> > > > > > > >> screen
> > > > > > > >> > below.
> > > > > > > >> >
> > > > > > > >> > Is there a way to work around the diferent ATCF
format?
> > I've
> > > > > > > attached a
> > > > > > > >> > sample of my Adeck and Bdeck files.
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> >
> > > > > > > >> > Roz
> > > > > > > >> >
> > > > > > > >> > Output to the screen:
> > > > > > > >> > /opt/MET7/bin/tc_pairs -adeck
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003
> > > > > > > >> > -bdeck
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003
> > > > > > > >> > -config
> > > > > > > >> >
> > > > > > >
> > > >
>
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> > > > > > > >> > -out
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt
> > > > > > > >> > -log
> > > > > > > >>
> > > > >
>
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> > > > > > > >> > -v 3
> > > > > > > >> > DEBUG 1: [Source 1 of 1] ADECK Source:
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003,
> > > > > > > >> > Model Suffix: (nul)
> > > > > > > >> > DEBUG 1: [Source 1 of 1] BDECK Source:
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003,
> > > > > > > >> > Model Suffix: (nul)
> > > > > > > >> > DEBUG 1: Config File Default:
> > > > > > > >> > /opt/MET7/share/met/config/TCPairsConfig_default
> > > > > > > >> > DEBUG 1: Config File User:
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> > > > > > > >> > DEBUG 1: Distance to land file:
> /opt/MET7/share/met/tc_data/
> > > > > > > >> > dland_global_tenth_degree.nc
> > > > > > > >> > DEBUG 2: Processing 1 BDECK file(s).
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude ...
> > "000"
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude ...
> > > "687N"
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude ...
> > "000"
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude ...
> > > "701N"
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude ...
> > "000"
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude ...
> > > "711N"
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude ...
> > "000"
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude ...
> > > "720N"
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude ...
> > "000"
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude ...
> > > "730N"
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude ...
> > "000"
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude ...
> > > "741N"
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude ...
> > "000"
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude ...
> > > "751N"
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude ...
> > "000"
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude ...
> > > "760N"
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude ...
> > "000"
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING:
> > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude ...
> > > "764N"
> > > > > > > >> > WARNING:
> > > > > > > >> > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
> > > > > > > >> > DEBUG 3: Identified 9 track(s).
> > > > > > > >> > DEBUG 2: Found 9 BDECK track(s).
> > > > > > > >> > DEBUG 2: Processing 1 ADECK file(s).
> > > > > > > >> > DEBUG 3: Used 0 of 22 lines read from 1 file(s).
> > > > > > > >> > DEBUG 3: Identified 0 track(s).
> > > > > > > >> > DEBUG 2: Deriving 12-hour interpolated ADECK
tracks.
> > > > > > > >> > DEBUG 2: Finished adding 0 and replacing 0 Interp12
> > track(s).
> > > > > > > >> > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > > > > > > >> > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > > > > > > >> > DEBUG 2: Deriving 0 ADECK lag model(s).
> > > > > > > >> > DEBUG 2: Added 0 ADECK lag tracks(s).
> > > > > > > >> > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline
model(s).
> > > > > > > >> > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> > > > > > > >> > DEBUG 2: Filtering 0 ADECK tracks based on config
file
> > > settings.
> > > > > > > >> > DEBUG 3: Total tracks read = 0
> > > > > > > >> > DEBUG 3: Total tracks kept = 0
> > > > > > > >> > DEBUG 3: Rejected for storm name = 0
> > > > > > > >> > DEBUG 3: Rejected for valid time = 0
> > > > > > > >> > DEBUG 3: Rejected for required lead times = 0
> > > > > > > >> > DEBUG 3: Rejected for init mask = 0
> > > > > > > >> > DEBUG 3: Rejected for valid mask = 0
> > > > > > > >> > DEBUG 2: Matching 0 ADECK tracks to 9 BDECK tracks.
> > > > > > > >> > DEBUG 1: Watch/Warning file:
> > > > > > /opt/MET7/share/met/tc_data/wwpts_us.txt
> > > > > > > >> > DEBUG 1: Output file:
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt.tcst
> > > > > > > >> >
> > > > > > > >> > --
> > > > > > > >> > Rosalyn MacCracken
> > > > > > > >> > Support Scientist
> > > > > > > >> >
> > > > > > > >> > Ocean Applications Branch
> > > > > > > >> > NOAA/NWS Ocean Prediction Center
> > > > > > > >> > NCWCP
> > > > > > > >> > 5830 University Research Ct
> > > > > > > >> > College Park, MD 20740-3818
> > > > > > > >> >
> > > > > > > >> > (p) 301-683-1551
> > > > > > > >> > rosalyn.maccracken at noaa.gov
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >>
> > > > > > > >
> > > > > > > > --
> > > > > > > > Rosalyn MacCracken
> > > > > > > > Support Scientist
> > > > > > > >
> > > > > > > > Ocean Applications Branch
> > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > NCWCP
> > > > > > > > 5830 University Research Ct
> > > > > > > > College Park, MD 20740-3818
> > > > > > > >
> > > > > > > > (p) 301-683-1551
> > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Rosalyn MacCracken
> > > > > > > Support Scientist
> > > > > > >
> > > > > > > Ocean Applications Branch
> > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > NCWCP
> > > > > > > 5830 University Research Ct
> > > > > > > College Park, MD 20740-3818
> > > > > > >
> > > > > > > (p) 301-683-1551
> > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Rosalyn MacCracken
> > > > > Support Scientist
> > > > >
> > > > > Ocean Applications Branch
> > > > > NOAA/NWS Ocean Prediction Center
> > > > > NCWCP
> > > > > 5830 University Research Ct
> > > > > College Park, MD 20740-3818
> > > > >
> > > > > (p) 301-683-1551
> > > > > rosalyn.maccracken at noaa.gov
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > Rosalyn MacCracken
> > > Support Scientist
> > >
> > > Ocean Applications Branch
> > > NOAA/NWS Ocean Prediction Center
> > > NCWCP
> > > 5830 University Research Ct
> > > College Park, MD 20740-3818
> > >
> > > (p) 301-683-1551
> > > rosalyn.maccracken at noaa.gov
> > >
> > >
> >
> >
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
>
------------------------------------------------
Subject: Help with using TC_pairs
From: Rosalyn MacCracken - NOAA Affiliate
Time: Tue Nov 27 12:31:25 2018
Hi John,
That worked! Thank you so much for seeing my little mistake. I
think,
originally, I had something in those brackets, and took it out, but,
left
the quotes inside the brackets. I was hoping it was something easy
like
that.
So, I have a few more questions.
1) I want to use a domain file, like I did with point_stat. For
that, I
created a *.poly file, then, converted it to a *.nc. Which do I use
for
tc_pairs?
2) Is there a way to change the wind speed warning categories, from
34kts,
50kts and 64kts, just in case OPC might want some other category?
Thinking about what I'm going to do with this using tc_stat, I will
need to
calculate track errors(in nm) and intensity errors by pressure, and
also,
intensity errors by lead time. I imagine I will be able to do that
using
-column_thresh, or something else, correct? I saw that on the Online
Tutorial page. But, I guess my question about those things would be,
is
there anything I need to make sure that is in my adeck/bdeck files
that is
not included now, that I should include to make those tasks easier? I
just
want to make sure that I've included everything I need.
Roz
On Tue, Nov 27, 2018 at 11:43 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Roz,
>
> Thanks for posting your sample data. I found the problem in your
> TCPairsConfig file. Change...
> FROM: storm_id = [ "" ];
> TO: storm_id = [ ];
>
> The storm_id entry is an array of storm ID's to match. An empty
array is
> just square brackets with nothing inside ([ ]). Since you had
> double-quotes inside, that's an array of length 1 containing an
empty
> string.
>
> Thanks,
> John
>
>
> On Tue, Nov 27, 2018 at 4:50 AM Rosalyn MacCracken - NOAA Affiliate
via RT
> <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
> >
> > Hi John,
> >
> > I uploaded the original files (Adeck = aml*, Bdeck=bml*, without
the
> > *modified.dat ending), the modified files (with the *modified.dat
ending)
> > and the config file into my ftp directory (*cd
> > incoming/irap/met_help/maccracken). *Let me know what you find.
> >
> > Thanks for your help!
> >
> > Roz
> >
> > On Mon, Nov 26, 2018 at 5:22 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Roz,
> > >
> > > Yes, the first 8 columns are required. The additional ones are
> optional.
> > >
> > > Looking at your log output, I see that tc_pairs is skipping over
all
> 642
> > > input lines. And the question is why? If you send me your
latest
> ADeck,
> > > BDeck, and TC-Pairs config files, I could run it here and try to
figure
> > out
> > > why it isn't defining any ADECK tracks.
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Nov 26, 2018 at 12:23 PM Rosalyn MacCracken - NOAA
Affiliate
> via
> > RT
> > > <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
>
> > > >
> > > > So, can I just have 8 columns for the input file, or, do I
need to
> make
> > > > columns of -9999 or NaN for all the rest of the columns?
> > > >
> > > > I experimented, and just wrote out 10 columns for the input
file:
> > > > agfso_trackdf_mod =
> > > >
> > > >
> > >
> >
>
agfso_trackdf[['basin_name','storm_id','model_start_date','const','model','fcst_lead_time','vort_lat_cent','vort_lon_cent','vort_10m_max',
> > > > 'vort_min_mslp','type','thresh_wspd']]
> > > >
> > > > because I wanted stats for these variables, and I got the
output
> below.
> > > > So, maybe it's working, but, the config file needs to be
different?
> > Any
> > > > other ideas of what I should try doing/looking at?
> > > >
> > > > Roz
> > > >
> > > > Output:
> > > > DEBUG 1: [Source 1 of 1] ADECK Source:
> > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018083118.gfso.0053_modified.dat,
> > > > Model Suffix: _EXP
> > > > DEBUG 1: [Source 1 of 1] BDECK Source:
> > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018083118.gfso.0053_modified.dat,
> > > > Model Suffix: (nul)
> > > > DEBUG 1: Config File Default:
> > > > /opt/MET7/share/met/config/TCPairsConfig_default
> > > > DEBUG 1: Config File User:
> > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > > > DEBUG 1: Distance to land file: /opt/MET7/share/met/tc_data/
> > > > dland_global_tenth_degree.nc
> > > > DEBUG 4:
> > > > DEBUG 4: Latitude/Longitude Grid Data:
> > > > DEBUG 4: lat_ll: -90
> > > > DEBUG 4: lon_ll: 180
> > > > DEBUG 4: delta_lat: 0.1
> > > > DEBUG 4: delta_lon: 0.1
> > > > DEBUG 4: Nlat: 1801
> > > > DEBUG 4: Nlon: 3601
> > > > DEBUG 4:
> > > > DEBUG 2: Processing 1 BDECK file(s).
> > > > DEBUG 4: [File 1 of 1] Used 30 of 30 lines read from file
> > > >
> > > >
> > >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018083118.gfso.0053_modified.dat"
> > > > DEBUG 3: Used 30 of 30 lines read from 1 file(s).
> > > > DEBUG 3: Identified 1 track(s).
> > > > DEBUG 4: [Track 1 of 1] TrackInfo: StormId = "ML0800532018",
IsBest =
> > > > false, IsOper = false, CheckAnly = true, IsAnly = true, Basin
= "ML",
> > > > Cyclone = "080053", StormName = "", TechniqueNumber = 3,
Technique =
> > > > "GFSO", Initials = "", InitTime = NA, MinValidTime =
20180805_060000,
> > > > MaxValidTime = 20180812_120000, NPoints = 30, NAlloc = 100,
> NTrackLines
> > > = 0
> > > > DEBUG 2: Found 1 BDECK track(s).
> > > > DEBUG 2: Processing 1 ADECK file(s).
> > > > DEBUG 4: [File 1 of 1] Used 0 of 642 lines read from file
> > > >
> > > >
> > >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018083118.gfso.0053_modified.dat"
> > > > DEBUG 3: Used 0 of 642 lines read from 1 file(s).
> > > > DEBUG 3: Identified 0 track(s).
> > > > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> > > > DEBUG 2: Finished adding 0 and replacing 0 Interp12 track(s).
> > > > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > > > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > > > DEBUG 2: Deriving 0 ADECK lag model(s).
> > > > DEBUG 2: Added 0 ADECK lag tracks(s).
> > > > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> > > > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> > > > DEBUG 2: Filtering 0 ADECK tracks based on config file
settings.
> > > > DEBUG 3: Total tracks read = 0
> > > > DEBUG 3: Total tracks kept = 0
> > > > DEBUG 3: Rejected for storm name = 0
> > > > DEBUG 3: Rejected for valid time = 0
> > > > DEBUG 3: Rejected for required lead times = 0
> > > > DEBUG 3: Rejected for init mask = 0
> > > > DEBUG 3: Rejected for valid mask = 0
> > > > DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
> > > > DEBUG 1: Watch/Warning file:
/opt/MET7/share/met/tc_data/wwpts_us.txt
> > > > DEBUG 1: Output file:
> > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018083118.gfso.0053_test_modified_output.txt.tcst
> > > >
> > > >
> > > > On Mon, Nov 26, 2018 at 2:06 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Roz,
> > > > >
> > > > > Page 334 of the user's guide (
> > > > >
> > >
>
https://dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v8.0.pdf
> > > > )
> > > > > mentions that only the first 8 ATCF columns are required.
If
> > > additional
> > > > > columns are present in the input ATCF files, then tc_pairs
will
> > > interpret
> > > > > them as specified by the ATCF format.
> > > > >
> > > > > So if the storm name column is present, MET in interpret
those
> values
> > > as
> > > > > the storm name.
> > > > >
> > > > > For Guang Ping's data, probably best to have that column
empty.
> > > > >
> > > > > John
> > > > >
> > > > >
> > > > > On Mon, Nov 26, 2018 at 11:51 AM Rosalyn MacCracken - NOAA
> Affiliate
> > > via
> > > > RT
> > > > > <met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
> > > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > Ok, I will ignore the suffix error. I do have numeric
values in
> > > > > > STORMNAME. I wonder if in the modified files I have, this
column
> > is
> > > > not
> > > > > > actually STORMNAME, but, some other variable. Now, I
don't have
> > the
> > > > > number
> > > > > > of columns that the NRL website has. Will that be a
problem?
> > > Should I
> > > > > > create dummy columns?
> > > > > >
> > > > > > Roz
> > > > > >
> > > > > > On Mon, Nov 26, 2018 at 1:14 PM John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Roz,
> > > > > > >
> > > > > > > I see that you're wondering about this log message:
> > > > > > > *Model Suffix: (nul)*
> > > > > > >
> > > > > > > Everything is fine, you do not need to be concerned
about this.
> > > > > Here's a
> > > > > > > line taken from the usage statement for the tc_pairs
tool:
> > > > > > > *Note: The "-adeck", "-edeck", and "-bdeck" options may
include
> > > > > > > "suffix=string" to modify the model names from that
source.*
> > > > > > >
> > > > > > > Since you didn't specify "suffix=string" on the command
line,
> the
> > > > > suffix
> > > > > > > remains the default value of an empty string, i.e.
"(nul)". So
> > why
> > > > > would
> > > > > > > you ever want to use a suffix? Sometimes data is messy.
We
> were
> > > > > running
> > > > > > > tc_pairs to evaluate track data from multiple sources
and we
> > found
> > > > that
> > > > > > > some model names we reused in both sources. Let's say
the OFCL
> > > > > forecast
> > > > > > > showed up in both stream 1.0 and stream 1.5 data.
Specifying a
> > > > suffix
> > > > > > such
> > > > > > > as "suffix=_STREAM1.5" changes the model name from
"OFCL" to
> > > > > > > "OFCL_STREAM1.5". And that enables us to distinguish
between
> > those
> > > > > > tracks
> > > > > > > in the output.
> > > > > > >
> > > > > > > But uou probably don't have any need for this option in
your
> > > current
> > > > > > work.
> > > > > > >
> > > > > > > However, I am concerned about these log messages:
> > > > > > > DEBUG 4: Updating storm name from "-287" to "-305" for
> > > ML0900022018.
> > > > > > > DEBUG 4: Updating storm name from "-305" to "-255" for
> > > ML0900022018.
> > > > > > > DEBUG 4: Updating storm name from "-255" to "-203" for
> > > ML0900022018.
> > > > > > > DEBUG 4: Updating storm name from "-203" to "-205" for
> > > ML0900022018.
> > > > > > > DEBUG 4: Updating storm name from "-205" to "-217" for
> > > ML0900022018.
> > > > > > > DEBUG 4: Updating storm name from "-217" to "-211" for
> > > ML0900022018.
> > > > > > > DEBUG 4: Updating storm name from "-211" to "-182" for
> > > ML0900022018.
> > > > > > > DEBUG 4: Updating storm name from "-182" to "-138" for
> > > ML0900022018.
> > > > > > >
> > > > > > > You likely have numeric values showing up in the
"STORMNAME"
> ATCF
> > > > > column:
> > > > > > >
> > >
https://www.nrlmry.navy.mil/atcf_web/docs/database/new/abdeck.txt
> > > > > > >
> > > > > > > I'd recommend looking into that.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Mon, Nov 26, 2018 at 8:44 AM Rosalyn MacCracken -
NOAA
> > Affiliate
> > > > via
> > > > > > RT
> > > > > > > <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
> > >
> > > > > > > >
> > > > > > > > Hi John,
> > > > > > > >
> > > > > > > > I figured out an easy way to read in and write out
Guang
> Ping's
> > > > > > modified
> > > > > > > > ATCF file, so that it:
> > > > > > > > 1) Removes the 3rd column
> > > > > > > > 2) Replaces -99 with -9999 in column 21 and 24 (after
the
> 3rd
> > > > column
> > > > > > is
> > > > > > > > removed)
> > > > > > > > 3) Updates column 2 to include month+cyclone number.
(You
> had
> > > > said
> > > > > to
> > > > > > > do
> > > > > > > > year+cyclone #, but the tc_pairs_wrapper.py said to do
> > > > > month+cyclone#)
> > > > > > > >
> > > > > > > > I changed Adeck and Bdeck.
> > > > > > > > Just a couple lines of my file:
> > > > > > > >
> > > > > > > > ML, 090002, 2018090100, 03, GFSO, 000, 511S, 897W,
40,
> 976,
> > > XX,
> > > > > 34,
> > > > > > > > NEQ, 0625, 0630, 0000, 0226, 978, 68, -9999, 123,
155,
> > -9999,
> > > > > -9999,
> > > > > > > > -9999, -297, -394, -287, -458, 0073
> > > > > > > > ML, 090002, 2018090106, 03, GFSO, 000, 537S, 854W,
44,
> 964,
> > > XX,
> > > > > 34,
> > > > > > > > NEQ, 0640, 0645, 0000, 0297, 969, 108, -9999, 119,
147,
> > -9999,
> > > > > -9999,
> > > > > > > > -9999, -403, -648, -305, -394, 0055
> > > > > > > >
> > > > > > > > So, I've run tc_pairs again and my output is below. I
don't
> > > > > understand
> > > > > > > the
> > > > > > > > Model Suffix: (nul). Is the input format still wrong?
> > > > > > > >
> > > > > > > > Roz
> > > > > > > >
> > > > > > > > Output:
> > > > > > > > /opt/MET7/bin/tc_pairs -adeck
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat
> > > > > > > > -bdeck
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat
> > > > > > > > -config
> > > > > >
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > > > > > > > -out
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt
> > > > > > > > -log
> > > > > > >
> > >
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> > > > > > > > -v 4
> > > > > > > > DEBUG 1: [Source 1 of 1] ADECK Source:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat,
> > > > > > > > Model Suffix: (nul)
> > > > > > > > DEBUG 1: [Source 1 of 1] BDECK Source:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat,
> > > > > > > > Model Suffix: (nul)
> > > > > > > > DEBUG 1: Config File Default:
> > > > > > > > /opt/MET7/share/met/config/TCPairsConfig_default
> > > > > > > > DEBUG 1: Config File User:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > > > > > > > DEBUG 1: Distance to land file:
/opt/MET7/share/met/tc_data/
> > > > > > > > dland_global_tenth_degree.nc
> > > > > > > > DEBUG 4:
> > > > > > > > DEBUG 4: Latitude/Longitude Grid Data:
> > > > > > > > DEBUG 4: lat_ll: -90
> > > > > > > > DEBUG 4: lon_ll: 180
> > > > > > > > DEBUG 4: delta_lat: 0.1
> > > > > > > > DEBUG 4: delta_lon: 0.1
> > > > > > > > DEBUG 4: Nlat: 1801
> > > > > > > > DEBUG 4: Nlon: 3601
> > > > > > > > DEBUG 4:
> > > > > > > > DEBUG 2: Processing 1 BDECK file(s).
> > > > > > > > DEBUG 4: Updating storm name from "-287" to "-305" for
> > > > ML0900022018.
> > > > > > > > DEBUG 4: Updating storm name from "-305" to "-255" for
> > > > ML0900022018.
> > > > > > > > DEBUG 4: Updating storm name from "-255" to "-203" for
> > > > ML0900022018.
> > > > > > > > DEBUG 4: Updating storm name from "-203" to "-205" for
> > > > ML0900022018.
> > > > > > > > DEBUG 4: Updating storm name from "-205" to "-217" for
> > > > ML0900022018.
> > > > > > > > DEBUG 4: Updating storm name from "-217" to "-211" for
> > > > ML0900022018.
> > > > > > > > DEBUG 4: Updating storm name from "-211" to "-182" for
> > > > ML0900022018.
> > > > > > > > DEBUG 4: Updating storm name from "-182" to "-138" for
> > > > ML0900022018.
> > > > > > > > DEBUG 4: [File 1 of 1] Used 9 of 9 lines read from
file
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat"
> > > > > > > > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
> > > > > > > > DEBUG 3: Identified 1 track(s).
> > > > > > > > DEBUG 4: [Track 1 of 1] TrackInfo: StormId =
"ML0900022018",
> > > > IsBest =
> > > > > > > > false, IsOper = false, CheckAnly = true, IsAnly =
true,
> Basin =
> > > > "ML",
> > > > > > > > Cyclone = "090002", StormName = "-138",
TechniqueNumber = 3,
> > > > > Technique
> > > > > > =
> > > > > > > > "GFSO", Initials = "-9999", InitTime = NA,
MinValidTime =
> > > > > > > 20180901_000000,
> > > > > > > > MaxValidTime = 20180903_000000, NPoints = 9, NAlloc =
100,
> > > > > NTrackLines
> > > > > > =
> > > > > > > 0
> > > > > > > > DEBUG 2: Found 1 BDECK track(s).
> > > > > > > > DEBUG 2: Processing 1 ADECK file(s).
> > > > > > > > DEBUG 4: [File 1 of 1] Used 0 of 21 lines read from
file
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat"
> > > > > > > > DEBUG 3: Used 0 of 21 lines read from 1 file(s).
> > > > > > > > DEBUG 3: Identified 0 track(s).
> > > > > > > > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> > > > > > > > DEBUG 2: Finished adding 0 and replacing 0 Interp12
track(s).
> > > > > > > > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > > > > > > > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > > > > > > > DEBUG 2: Deriving 0 ADECK lag model(s).
> > > > > > > > DEBUG 2: Added 0 ADECK lag tracks(s).
> > > > > > > > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> > > > > > > > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> > > > > > > > DEBUG 2: Filtering 0 ADECK tracks based on config file
> > settings.
> > > > > > > > DEBUG 3: Total tracks read = 0
> > > > > > > > DEBUG 3: Total tracks kept = 0
> > > > > > > > DEBUG 3: Rejected for storm name = 0
> > > > > > > > DEBUG 3: Rejected for valid time = 0
> > > > > > > > DEBUG 3: Rejected for required lead times = 0
> > > > > > > > DEBUG 3: Rejected for init mask = 0
> > > > > > > > DEBUG 3: Rejected for valid mask = 0
> > > > > > > > DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
> > > > > > > > DEBUG 1: Watch/Warning file:
> > > > /opt/MET7/share/met/tc_data/wwpts_us.txt
> > > > > > > > DEBUG 1: Output file:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt.tcst
> > > > > > > >
> > > > > > > > On Wed, Nov 21, 2018 at 9:16 AM Rosalyn MacCracken -
NOAA
> > > > Affiliate <
> > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > > > > > >
> > > > > > > > > Hi John,
> > > > > > > > >
> > > > > > > > > Yes, that makes sense, and yes, I am using Guang
Pings'
> > > modified
> > > > > > files.
> > > > > > > > > I'm not sure how to run tc_pairs_wrapper.py, though.
I
> > think I
> > > > > need
> > > > > > > some
> > > > > > > > > type of *.conf file to pass in the files I need, or
> something
> > > > like
> > > > > > > that.
> > > > > > > > > I'm in the process of reading through the code now.
I
> wonder
> > > if
> > > > I
> > > > > > > could
> > > > > > > > > just pull out the function process_non_atcf, to run
in a
> > > > > stand-alone
> > > > > > > > mode.
> > > > > > > > > I wish there was some sort of documentation with
this
> code...
> > > > > > > > >
> > > > > > > > > And, so, once I can read these files, I can use
tc_pairs to
> > > > create
> > > > > > the
> > > > > > > > > forecast error statistics, and then use tc_stats to
> aggregate
> > > the
> > > > > > files
> > > > > > > > and
> > > > > > > > > get the mean, etc, for a period of time, correct?
Was
> there
> > > some
> > > > > > sort
> > > > > > > of
> > > > > > > > > plotting scripts for the output files?
> > > > > > > > >
> > > > > > > > > So, with tc_pairs, I can define a region, like I did
with
> > > > > point_stat,
> > > > > > > > > correct? Can I use the same *.nc files, or do I use
the
> > > original
> > > > > > > *.poly
> > > > > > > > > files?
> > > > > > > > >
> > > > > > > > > Roz
> > > > > > > > >
> > > > > > > > > On Tue, Nov 20, 2018 at 7:53 PM John Halley Gotway
via RT <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > >> Hi Roz,
> > > > > > > > >>
> > > > > > > > >> Yes, I see you’re using the modified ATCF format
that
> Guang
> > > Ping
> > > > > > uses.
> > > > > > > > >> You’re correct that tc_pairs doesn’t parse this
directly
> > since
> > > > it
> > > > > > > > differs
> > > > > > > > >> slightly from the ATCF format.
> > > > > > > > >>
> > > > > > > > >> When we’ve processed this data in the past, we
first
> > > > pre-processed
> > > > > > it
> > > > > > > > in a
> > > > > > > > >> shell script to:
> > > > > > > > >>
> > > > > > > > >> 1. Replace the -99 value with -9999 for missing
data.
> > > > > > > > >> 2. Remove the 3rd column
> > > > > > > > >> 3. Updating the 2nd column to include the year and
cyclone
> > > > number.
> > > > > > > > >>
> > > > > > > > >> And that enable tc_pairs to read it.
> > > > > > > > >>
> > > > > > > > >> This is actually Done in python by one of the use
cases in
> > > > > METplus.
> > > > > > > > >>
> > > > > > > > >> Take a look at this file:
> > > > > > > > >>
> > > > >
> https://github.com/NCAR/METplus/blob/master/ush/tc_pairs_wrapper.py
> > > > > > > > >>
> > > > > > > > >> At the function named:
> > > > > > > > >> process_non_atcf
> > > > > > > > >>
> > > > > > > > >> I haven’t run this using METplus scripts myself,
but all
> the
> > > > > pieces
> > > > > > of
> > > > > > > > >> logic are there, and it should be possible.
> > > > > > > > >>
> > > > > > > > >> Make sense?
> > > > > > > > >>
> > > > > > > > >> Thanks
> > > > > > > > >> John
> > > > > > > > >>
> > > > > > > > >> On Tue, Nov 20, 2018 at 1:13 PM Rosalyn MacCracken
- NOAA
> > > > > Affiliate
> > > > > > > via
> > > > > > > > >> RT <
> > > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > > >>
> > > > > > > > >> >
> > > > > > > > >> > Tue Nov 20 13:13:05 2018: Request 87885 was acted
upon.
> > > > > > > > >> > Transaction: Ticket created by
> > rosalyn.maccracken at noaa.gov
> > > > > > > > >> > Queue: met_help
> > > > > > > > >> > Subject: Help with using TC_pairs
> > > > > > > > >> > Owner: Nobody
> > > > > > > > >> > Requestors: rosalyn.maccracken at noaa.gov
> > > > > > > > >> > Status: new
> > > > > > > > >> > Ticket <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
> > > > > > > > >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > Hi,
> > > > > > > > >> >
> > > > > > > > >> > I'm trying to use TC_pairs for extra-tropical
cyclones
> > > > > (basin=ML).
> > > > > > > I
> > > > > > > > >> think
> > > > > > > > >> > the format of the ATCF data is the issue, but,
I'm not
> > sure.
> > > > I
> > > > > > have
> > > > > > > > an
> > > > > > > > >> > Adeck and a Bdeck file, where a sample of one of
the
> lines
> > > is:
> > > > > > > > >> > ML, 0003, 2018090100_F000_687N_0159W_FOF,
2018090100,
> 03,
> > > > GFSO,
> > > > > > 000,
> > > > > > > > >> 687N,
> > > > > > > > >> > 159W, 32, 987, XX, 34, NEQ, 0301, 0338, 0000,
0000,
> > 988,
> > > > > 52,
> > > > > > > > -99,
> > > > > > > > >> > 330, 86, -99, -9999, -9999, 175, 202, 137,
135,
> 0081
> > > > > > > > >> >
> > > > > > > > >> > This is slightly different from the NRL ATCF
format on
> > their
> > > > > > > website,
> > > > > > > > >> such
> > > > > > > > >> > as the second column. Anyway, I configured the
config
> > file
> > > > very
> > > > > > > > simply,
> > > > > > > > >> > (model=GFS, basin=ML), and ran tc_pairs, and got
the
> > output
> > > to
> > > > > the
> > > > > > > > >> screen
> > > > > > > > >> > below.
> > > > > > > > >> >
> > > > > > > > >> > Is there a way to work around the diferent ATCF
format?
> > > I've
> > > > > > > > attached a
> > > > > > > > >> > sample of my Adeck and Bdeck files.
> > > > > > > > >> >
> > > > > > > > >> > Thanks,
> > > > > > > > >> >
> > > > > > > > >> > Roz
> > > > > > > > >> >
> > > > > > > > >> > Output to the screen:
> > > > > > > > >> > /opt/MET7/bin/tc_pairs -adeck
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003
> > > > > > > > >> > -bdeck
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003
> > > > > > > > >> > -config
> > > > > > > > >> >
> > > > > > > >
> > > > >
> >
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> > > > > > > > >> > -out
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt
> > > > > > > > >> > -log
> > > > > > > > >>
> > > > > >
> >
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> > > > > > > > >> > -v 3
> > > > > > > > >> > DEBUG 1: [Source 1 of 1] ADECK Source:
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003,
> > > > > > > > >> > Model Suffix: (nul)
> > > > > > > > >> > DEBUG 1: [Source 1 of 1] BDECK Source:
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003,
> > > > > > > > >> > Model Suffix: (nul)
> > > > > > > > >> > DEBUG 1: Config File Default:
> > > > > > > > >> > /opt/MET7/share/met/config/TCPairsConfig_default
> > > > > > > > >> > DEBUG 1: Config File User:
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> > > > > > > > >> > DEBUG 1: Distance to land file:
> > /opt/MET7/share/met/tc_data/
> > > > > > > > >> > dland_global_tenth_degree.nc
> > > > > > > > >> > DEBUG 2: Processing 1 BDECK file(s).
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude ...
> > > "000"
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
> ...
> > > > "687N"
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude ...
> > > "000"
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
> ...
> > > > "701N"
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude ...
> > > "000"
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
> ...
> > > > "711N"
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude ...
> > > "000"
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
> ...
> > > > "720N"
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude ...
> > > "000"
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
> ...
> > > > "730N"
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude ...
> > > "000"
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
> ...
> > > > "741N"
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude ...
> > > "000"
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
> ...
> > > > "751N"
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude ...
> > > "000"
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
> ...
> > > > "760N"
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude ...
> > > "000"
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING:
> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
> ...
> > > > "764N"
> > > > > > > > >> > WARNING:
> > > > > > > > >> > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
> > > > > > > > >> > DEBUG 3: Identified 9 track(s).
> > > > > > > > >> > DEBUG 2: Found 9 BDECK track(s).
> > > > > > > > >> > DEBUG 2: Processing 1 ADECK file(s).
> > > > > > > > >> > DEBUG 3: Used 0 of 22 lines read from 1 file(s).
> > > > > > > > >> > DEBUG 3: Identified 0 track(s).
> > > > > > > > >> > DEBUG 2: Deriving 12-hour interpolated ADECK
tracks.
> > > > > > > > >> > DEBUG 2: Finished adding 0 and replacing 0
Interp12
> > > track(s).
> > > > > > > > >> > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > > > > > > > >> > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > > > > > > > >> > DEBUG 2: Deriving 0 ADECK lag model(s).
> > > > > > > > >> > DEBUG 2: Added 0 ADECK lag tracks(s).
> > > > > > > > >> > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline
model(s).
> > > > > > > > >> > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> > > > > > > > >> > DEBUG 2: Filtering 0 ADECK tracks based on config
file
> > > > settings.
> > > > > > > > >> > DEBUG 3: Total tracks read = 0
> > > > > > > > >> > DEBUG 3: Total tracks kept = 0
> > > > > > > > >> > DEBUG 3: Rejected for storm name = 0
> > > > > > > > >> > DEBUG 3: Rejected for valid time = 0
> > > > > > > > >> > DEBUG 3: Rejected for required lead times = 0
> > > > > > > > >> > DEBUG 3: Rejected for init mask = 0
> > > > > > > > >> > DEBUG 3: Rejected for valid mask = 0
> > > > > > > > >> > DEBUG 2: Matching 0 ADECK tracks to 9 BDECK
tracks.
> > > > > > > > >> > DEBUG 1: Watch/Warning file:
> > > > > > > /opt/MET7/share/met/tc_data/wwpts_us.txt
> > > > > > > > >> > DEBUG 1: Output file:
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt.tcst
> > > > > > > > >> >
> > > > > > > > >> > --
> > > > > > > > >> > Rosalyn MacCracken
> > > > > > > > >> > Support Scientist
> > > > > > > > >> >
> > > > > > > > >> > Ocean Applications Branch
> > > > > > > > >> > NOAA/NWS Ocean Prediction Center
> > > > > > > > >> > NCWCP
> > > > > > > > >> > 5830 University Research Ct
> > > > > > > > >> > College Park, MD 20740-3818
> > > > > > > > >> >
> > > > > > > > >> > (p) 301-683-1551
> > > > > > > > >> > rosalyn.maccracken at noaa.gov
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Rosalyn MacCracken
> > > > > > > > > Support Scientist
> > > > > > > > >
> > > > > > > > > Ocean Applications Branch
> > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > > NCWCP
> > > > > > > > > 5830 University Research Ct
> > > > > > > > > College Park, MD 20740-3818
> > > > > > > > >
> > > > > > > > > (p) 301-683-1551
> > > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Rosalyn MacCracken
> > > > > > > > Support Scientist
> > > > > > > >
> > > > > > > > Ocean Applications Branch
> > > > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > > > NCWCP
> > > > > > > > 5830 University Research Ct
> > > > > > > > College Park, MD 20740-3818
> > > > > > > >
> > > > > > > > (p) 301-683-1551
> > > > > > > > rosalyn.maccracken at noaa.gov
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Rosalyn MacCracken
> > > > > > Support Scientist
> > > > > >
> > > > > > Ocean Applications Branch
> > > > > > NOAA/NWS Ocean Prediction Center
> > > > > > NCWCP
> > > > > > 5830 University Research Ct
> > > > > > College Park, MD 20740-3818
> > > > > >
> > > > > > (p) 301-683-1551
> > > > > > rosalyn.maccracken at noaa.gov
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > Rosalyn MacCracken
> > > > Support Scientist
> > > >
> > > > Ocean Applications Branch
> > > > NOAA/NWS Ocean Prediction Center
> > > > NCWCP
> > > > 5830 University Research Ct
> > > > College Park, MD 20740-3818
> > > >
> > > > (p) 301-683-1551
> > > > rosalyn.maccracken at noaa.gov
> > > >
> > > >
> > >
> > >
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applications Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
> >
>
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Help with using TC_pairs
From: Rosalyn MacCracken - NOAA Affiliate
Time: Thu Dec 06 09:45:35 2018
Hi John,
I got tc_stat to work the way I wanted to, and was able to produce
summary
statistics. Now that I have matched pairs, I was thinking about doing
some
other things, unrelated to MET (like data mining, clustering or
classification of errors). To do this data mining, I bet it would be
easier if I had all the data in one big file of all the storm ids,
from
201711 - 201809. I could accomplish this with python, opening the
files
and appending to dataframes, etc. But, what I thought I should do is
ask
you if there was an easy way to do that in MET. Is there a way to
concat
all the files with the *tcst ending into one big file?
Thanks!
Roz
On Tue, Nov 27, 2018 at 2:31 PM Rosalyn MacCracken - NOAA Affiliate <
rosalyn.maccracken at noaa.gov> wrote:
> Hi John,
>
> That worked! Thank you so much for seeing my little mistake. I
think,
> originally, I had something in those brackets, and took it out, but,
left
> the quotes inside the brackets. I was hoping it was something easy
like
> that.
>
> So, I have a few more questions.
> 1) I want to use a domain file, like I did with point_stat. For
that, I
> created a *.poly file, then, converted it to a *.nc. Which do I use
for
> tc_pairs?
> 2) Is there a way to change the wind speed warning categories, from
> 34kts, 50kts and 64kts, just in case OPC might want some other
category?
>
> Thinking about what I'm going to do with this using tc_stat, I will
need
> to calculate track errors(in nm) and intensity errors by pressure,
and
> also, intensity errors by lead time. I imagine I will be able to do
that
> using -column_thresh, or something else, correct? I saw that on the
Online
> Tutorial page. But, I guess my question about those things would
be, is
> there anything I need to make sure that is in my adeck/bdeck files
that is
> not included now, that I should include to make those tasks easier?
I just
> want to make sure that I've included everything I need.
>
> Roz
>
> On Tue, Nov 27, 2018 at 11:43 AM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Roz,
>>
>> Thanks for posting your sample data. I found the problem in your
>> TCPairsConfig file. Change...
>> FROM: storm_id = [ "" ];
>> TO: storm_id = [ ];
>>
>> The storm_id entry is an array of storm ID's to match. An empty
array is
>> just square brackets with nothing inside ([ ]). Since you had
>> double-quotes inside, that's an array of length 1 containing an
empty
>> string.
>>
>> Thanks,
>> John
>>
>>
>> On Tue, Nov 27, 2018 at 4:50 AM Rosalyn MacCracken - NOAA Affiliate
via
>> RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
>> >
>> > Hi John,
>> >
>> > I uploaded the original files (Adeck = aml*, Bdeck=bml*, without
the
>> > *modified.dat ending), the modified files (with the *modified.dat
>> ending)
>> > and the config file into my ftp directory (*cd
>> > incoming/irap/met_help/maccracken). *Let me know what you find.
>> >
>> > Thanks for your help!
>> >
>> > Roz
>> >
>> > On Mon, Nov 26, 2018 at 5:22 PM John Halley Gotway via RT <
>> > met_help at ucar.edu>
>> > wrote:
>> >
>> > > Roz,
>> > >
>> > > Yes, the first 8 columns are required. The additional ones are
>> optional.
>> > >
>> > > Looking at your log output, I see that tc_pairs is skipping
over all
>> 642
>> > > input lines. And the question is why? If you send me your
latest
>> ADeck,
>> > > BDeck, and TC-Pairs config files, I could run it here and try
to
>> figure
>> > out
>> > > why it isn't defining any ADECK tracks.
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > > On Mon, Nov 26, 2018 at 12:23 PM Rosalyn MacCracken - NOAA
Affiliate
>> via
>> > RT
>> > > <met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
>
>> > > >
>> > > > So, can I just have 8 columns for the input file, or, do I
need to
>> make
>> > > > columns of -9999 or NaN for all the rest of the columns?
>> > > >
>> > > > I experimented, and just wrote out 10 columns for the input
file:
>> > > > agfso_trackdf_mod =
>> > > >
>> > > >
>> > >
>> >
>>
agfso_trackdf[['basin_name','storm_id','model_start_date','const','model','fcst_lead_time','vort_lat_cent','vort_lon_cent','vort_10m_max',
>> > > > 'vort_min_mslp','type','thresh_wspd']]
>> > > >
>> > > > because I wanted stats for these variables, and I got the
output
>> below.
>> > > > So, maybe it's working, but, the config file needs to be
different?
>> > Any
>> > > > other ideas of what I should try doing/looking at?
>> > > >
>> > > > Roz
>> > > >
>> > > > Output:
>> > > > DEBUG 1: [Source 1 of 1] ADECK Source:
>> > > >
>> > > >
>> > >
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018083118.gfso.0053_modified.dat,
>> > > > Model Suffix: _EXP
>> > > > DEBUG 1: [Source 1 of 1] BDECK Source:
>> > > >
>> > > >
>> > >
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018083118.gfso.0053_modified.dat,
>> > > > Model Suffix: (nul)
>> > > > DEBUG 1: Config File Default:
>> > > > /opt/MET7/share/met/config/TCPairsConfig_default
>> > > > DEBUG 1: Config File User:
>> > > >
>> > > >
>> > >
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
>> > > > DEBUG 1: Distance to land file: /opt/MET7/share/met/tc_data/
>> > > > dland_global_tenth_degree.nc
>> > > > DEBUG 4:
>> > > > DEBUG 4: Latitude/Longitude Grid Data:
>> > > > DEBUG 4: lat_ll: -90
>> > > > DEBUG 4: lon_ll: 180
>> > > > DEBUG 4: delta_lat: 0.1
>> > > > DEBUG 4: delta_lon: 0.1
>> > > > DEBUG 4: Nlat: 1801
>> > > > DEBUG 4: Nlon: 3601
>> > > > DEBUG 4:
>> > > > DEBUG 2: Processing 1 BDECK file(s).
>> > > > DEBUG 4: [File 1 of 1] Used 30 of 30 lines read from file
>> > > >
>> > > >
>> > >
>> >
>>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018083118.gfso.0053_modified.dat"
>> > > > DEBUG 3: Used 30 of 30 lines read from 1 file(s).
>> > > > DEBUG 3: Identified 1 track(s).
>> > > > DEBUG 4: [Track 1 of 1] TrackInfo: StormId = "ML0800532018",
IsBest
>> =
>> > > > false, IsOper = false, CheckAnly = true, IsAnly = true, Basin
=
>> "ML",
>> > > > Cyclone = "080053", StormName = "", TechniqueNumber = 3,
Technique =
>> > > > "GFSO", Initials = "", InitTime = NA, MinValidTime =
>> 20180805_060000,
>> > > > MaxValidTime = 20180812_120000, NPoints = 30, NAlloc = 100,
>> NTrackLines
>> > > = 0
>> > > > DEBUG 2: Found 1 BDECK track(s).
>> > > > DEBUG 2: Processing 1 ADECK file(s).
>> > > > DEBUG 4: [File 1 of 1] Used 0 of 642 lines read from file
>> > > >
>> > > >
>> > >
>> >
>>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018083118.gfso.0053_modified.dat"
>> > > > DEBUG 3: Used 0 of 642 lines read from 1 file(s).
>> > > > DEBUG 3: Identified 0 track(s).
>> > > > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
>> > > > DEBUG 2: Finished adding 0 and replacing 0 Interp12 track(s).
>> > > > DEBUG 2: Deriving 0 ADECK consensus model(s).
>> > > > DEBUG 2: Added 0 ADECK consensus tracks(s).
>> > > > DEBUG 2: Deriving 0 ADECK lag model(s).
>> > > > DEBUG 2: Added 0 ADECK lag tracks(s).
>> > > > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
>> > > > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
>> > > > DEBUG 2: Filtering 0 ADECK tracks based on config file
settings.
>> > > > DEBUG 3: Total tracks read = 0
>> > > > DEBUG 3: Total tracks kept = 0
>> > > > DEBUG 3: Rejected for storm name = 0
>> > > > DEBUG 3: Rejected for valid time = 0
>> > > > DEBUG 3: Rejected for required lead times = 0
>> > > > DEBUG 3: Rejected for init mask = 0
>> > > > DEBUG 3: Rejected for valid mask = 0
>> > > > DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
>> > > > DEBUG 1: Watch/Warning file:
>> /opt/MET7/share/met/tc_data/wwpts_us.txt
>> > > > DEBUG 1: Output file:
>> > > >
>> > > >
>> > >
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018083118.gfso.0053_test_modified_output.txt.tcst
>> > > >
>> > > >
>> > > > On Mon, Nov 26, 2018 at 2:06 PM John Halley Gotway via RT <
>> > > > met_help at ucar.edu>
>> > > > wrote:
>> > > >
>> > > > > Roz,
>> > > > >
>> > > > > Page 334 of the user's guide (
>> > > > >
>> > >
>>
https://dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v8.0.pdf
>> > > > )
>> > > > > mentions that only the first 8 ATCF columns are required.
If
>> > > additional
>> > > > > columns are present in the input ATCF files, then tc_pairs
will
>> > > interpret
>> > > > > them as specified by the ATCF format.
>> > > > >
>> > > > > So if the storm name column is present, MET in interpret
those
>> values
>> > > as
>> > > > > the storm name.
>> > > > >
>> > > > > For Guang Ping's data, probably best to have that column
empty.
>> > > > >
>> > > > > John
>> > > > >
>> > > > >
>> > > > > On Mon, Nov 26, 2018 at 11:51 AM Rosalyn MacCracken - NOAA
>> Affiliate
>> > > via
>> > > > RT
>> > > > > <met_help at ucar.edu> wrote:
>> > > > >
>> > > > > >
>> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
>> > > > > >
>> > > > > > Hi John,
>> > > > > >
>> > > > > > Ok, I will ignore the suffix error. I do have numeric
values in
>> > > > > > STORMNAME. I wonder if in the modified files I have,
this
>> column
>> > is
>> > > > not
>> > > > > > actually STORMNAME, but, some other variable. Now, I
don't have
>> > the
>> > > > > number
>> > > > > > of columns that the NRL website has. Will that be a
problem?
>> > > Should I
>> > > > > > create dummy columns?
>> > > > > >
>> > > > > > Roz
>> > > > > >
>> > > > > > On Mon, Nov 26, 2018 at 1:14 PM John Halley Gotway via RT
<
>> > > > > > met_help at ucar.edu>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Roz,
>> > > > > > >
>> > > > > > > I see that you're wondering about this log message:
>> > > > > > > *Model Suffix: (nul)*
>> > > > > > >
>> > > > > > > Everything is fine, you do not need to be concerned
about
>> this.
>> > > > > Here's a
>> > > > > > > line taken from the usage statement for the tc_pairs
tool:
>> > > > > > > *Note: The "-adeck", "-edeck", and "-bdeck" options may
>> include
>> > > > > > > "suffix=string" to modify the model names from that
source.*
>> > > > > > >
>> > > > > > > Since you didn't specify "suffix=string" on the command
line,
>> the
>> > > > > suffix
>> > > > > > > remains the default value of an empty string, i.e.
"(nul)".
>> So
>> > why
>> > > > > would
>> > > > > > > you ever want to use a suffix? Sometimes data is
messy. We
>> were
>> > > > > running
>> > > > > > > tc_pairs to evaluate track data from multiple sources
and we
>> > found
>> > > > that
>> > > > > > > some model names we reused in both sources. Let's say
the
>> OFCL
>> > > > > forecast
>> > > > > > > showed up in both stream 1.0 and stream 1.5 data.
Specifying
>> a
>> > > > suffix
>> > > > > > such
>> > > > > > > as "suffix=_STREAM1.5" changes the model name from
"OFCL" to
>> > > > > > > "OFCL_STREAM1.5". And that enables us to distinguish
between
>> > those
>> > > > > > tracks
>> > > > > > > in the output.
>> > > > > > >
>> > > > > > > But uou probably don't have any need for this option in
your
>> > > current
>> > > > > > work.
>> > > > > > >
>> > > > > > > However, I am concerned about these log messages:
>> > > > > > > DEBUG 4: Updating storm name from "-287" to "-305" for
>> > > ML0900022018.
>> > > > > > > DEBUG 4: Updating storm name from "-305" to "-255" for
>> > > ML0900022018.
>> > > > > > > DEBUG 4: Updating storm name from "-255" to "-203" for
>> > > ML0900022018.
>> > > > > > > DEBUG 4: Updating storm name from "-203" to "-205" for
>> > > ML0900022018.
>> > > > > > > DEBUG 4: Updating storm name from "-205" to "-217" for
>> > > ML0900022018.
>> > > > > > > DEBUG 4: Updating storm name from "-217" to "-211" for
>> > > ML0900022018.
>> > > > > > > DEBUG 4: Updating storm name from "-211" to "-182" for
>> > > ML0900022018.
>> > > > > > > DEBUG 4: Updating storm name from "-182" to "-138" for
>> > > ML0900022018.
>> > > > > > >
>> > > > > > > You likely have numeric values showing up in the
"STORMNAME"
>> ATCF
>> > > > > column:
>> > > > > > >
>> > >
https://www.nrlmry.navy.mil/atcf_web/docs/database/new/abdeck.txt
>> > > > > > >
>> > > > > > > I'd recommend looking into that.
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > John
>> > > > > > >
>> > > > > > > On Mon, Nov 26, 2018 at 8:44 AM Rosalyn MacCracken -
NOAA
>> > Affiliate
>> > > > via
>> > > > > > RT
>> > > > > > > <
>> > > > > > > met_help at ucar.edu> wrote:
>> > > > > > >
>> > > > > > > >
>> > > > > > > > <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
>> > >
>> > > > > > > >
>> > > > > > > > Hi John,
>> > > > > > > >
>> > > > > > > > I figured out an easy way to read in and write out
Guang
>> Ping's
>> > > > > > modified
>> > > > > > > > ATCF file, so that it:
>> > > > > > > > 1) Removes the 3rd column
>> > > > > > > > 2) Replaces -99 with -9999 in column 21 and 24
(after the
>> 3rd
>> > > > column
>> > > > > > is
>> > > > > > > > removed)
>> > > > > > > > 3) Updates column 2 to include month+cyclone number.
(You
>> had
>> > > > said
>> > > > > to
>> > > > > > > do
>> > > > > > > > year+cyclone #, but the tc_pairs_wrapper.py said to
do
>> > > > > month+cyclone#)
>> > > > > > > >
>> > > > > > > > I changed Adeck and Bdeck.
>> > > > > > > > Just a couple lines of my file:
>> > > > > > > >
>> > > > > > > > ML, 090002, 2018090100, 03, GFSO, 000, 511S, 897W,
40,
>> 976,
>> > > XX,
>> > > > > 34,
>> > > > > > > > NEQ, 0625, 0630, 0000, 0226, 978, 68, -9999, 123,
155,
>> > -9999,
>> > > > > -9999,
>> > > > > > > > -9999, -297, -394, -287, -458, 0073
>> > > > > > > > ML, 090002, 2018090106, 03, GFSO, 000, 537S, 854W,
44,
>> 964,
>> > > XX,
>> > > > > 34,
>> > > > > > > > NEQ, 0640, 0645, 0000, 0297, 969, 108, -9999, 119,
147,
>> > -9999,
>> > > > > -9999,
>> > > > > > > > -9999, -403, -648, -305, -394, 0055
>> > > > > > > >
>> > > > > > > > So, I've run tc_pairs again and my output is below.
I don't
>> > > > > understand
>> > > > > > > the
>> > > > > > > > Model Suffix: (nul). Is the input format still
wrong?
>> > > > > > > >
>> > > > > > > > Roz
>> > > > > > > >
>> > > > > > > > Output:
>> > > > > > > > /opt/MET7/bin/tc_pairs -adeck
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat
>> > > > > > > > -bdeck
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat
>> > > > > > > > -config
>> > > > > >
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
>> > > > > > > > -out
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt
>> > > > > > > > -log
>> > > > > > >
>> > >
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
>> > > > > > > > -v 4
>> > > > > > > > DEBUG 1: [Source 1 of 1] ADECK Source:
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat,
>> > > > > > > > Model Suffix: (nul)
>> > > > > > > > DEBUG 1: [Source 1 of 1] BDECK Source:
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat,
>> > > > > > > > Model Suffix: (nul)
>> > > > > > > > DEBUG 1: Config File Default:
>> > > > > > > > /opt/MET7/share/met/config/TCPairsConfig_default
>> > > > > > > > DEBUG 1: Config File User:
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
>> > > > > > > > DEBUG 1: Distance to land file:
/opt/MET7/share/met/tc_data/
>> > > > > > > > dland_global_tenth_degree.nc
>> > > > > > > > DEBUG 4:
>> > > > > > > > DEBUG 4: Latitude/Longitude Grid Data:
>> > > > > > > > DEBUG 4: lat_ll: -90
>> > > > > > > > DEBUG 4: lon_ll: 180
>> > > > > > > > DEBUG 4: delta_lat: 0.1
>> > > > > > > > DEBUG 4: delta_lon: 0.1
>> > > > > > > > DEBUG 4: Nlat: 1801
>> > > > > > > > DEBUG 4: Nlon: 3601
>> > > > > > > > DEBUG 4:
>> > > > > > > > DEBUG 2: Processing 1 BDECK file(s).
>> > > > > > > > DEBUG 4: Updating storm name from "-287" to "-305"
for
>> > > > ML0900022018.
>> > > > > > > > DEBUG 4: Updating storm name from "-305" to "-255"
for
>> > > > ML0900022018.
>> > > > > > > > DEBUG 4: Updating storm name from "-255" to "-203"
for
>> > > > ML0900022018.
>> > > > > > > > DEBUG 4: Updating storm name from "-203" to "-205"
for
>> > > > ML0900022018.
>> > > > > > > > DEBUG 4: Updating storm name from "-205" to "-217"
for
>> > > > ML0900022018.
>> > > > > > > > DEBUG 4: Updating storm name from "-217" to "-211"
for
>> > > > ML0900022018.
>> > > > > > > > DEBUG 4: Updating storm name from "-211" to "-182"
for
>> > > > ML0900022018.
>> > > > > > > > DEBUG 4: Updating storm name from "-182" to "-138"
for
>> > > > ML0900022018.
>> > > > > > > > DEBUG 4: [File 1 of 1] Used 9 of 9 lines read from
file
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat"
>> > > > > > > > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
>> > > > > > > > DEBUG 3: Identified 1 track(s).
>> > > > > > > > DEBUG 4: [Track 1 of 1] TrackInfo: StormId =
"ML0900022018",
>> > > > IsBest =
>> > > > > > > > false, IsOper = false, CheckAnly = true, IsAnly =
true,
>> Basin =
>> > > > "ML",
>> > > > > > > > Cyclone = "090002", StormName = "-138",
TechniqueNumber = 3,
>> > > > > Technique
>> > > > > > =
>> > > > > > > > "GFSO", Initials = "-9999", InitTime = NA,
MinValidTime =
>> > > > > > > 20180901_000000,
>> > > > > > > > MaxValidTime = 20180903_000000, NPoints = 9, NAlloc =
100,
>> > > > > NTrackLines
>> > > > > > =
>> > > > > > > 0
>> > > > > > > > DEBUG 2: Found 1 BDECK track(s).
>> > > > > > > > DEBUG 2: Processing 1 ADECK file(s).
>> > > > > > > > DEBUG 4: [File 1 of 1] Used 0 of 21 lines read from
file
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat"
>> > > > > > > > DEBUG 3: Used 0 of 21 lines read from 1 file(s).
>> > > > > > > > DEBUG 3: Identified 0 track(s).
>> > > > > > > > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
>> > > > > > > > DEBUG 2: Finished adding 0 and replacing 0 Interp12
>> track(s).
>> > > > > > > > DEBUG 2: Deriving 0 ADECK consensus model(s).
>> > > > > > > > DEBUG 2: Added 0 ADECK consensus tracks(s).
>> > > > > > > > DEBUG 2: Deriving 0 ADECK lag model(s).
>> > > > > > > > DEBUG 2: Added 0 ADECK lag tracks(s).
>> > > > > > > > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
>> > > > > > > > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
>> > > > > > > > DEBUG 2: Filtering 0 ADECK tracks based on config
file
>> > settings.
>> > > > > > > > DEBUG 3: Total tracks read = 0
>> > > > > > > > DEBUG 3: Total tracks kept = 0
>> > > > > > > > DEBUG 3: Rejected for storm name = 0
>> > > > > > > > DEBUG 3: Rejected for valid time = 0
>> > > > > > > > DEBUG 3: Rejected for required lead times = 0
>> > > > > > > > DEBUG 3: Rejected for init mask = 0
>> > > > > > > > DEBUG 3: Rejected for valid mask = 0
>> > > > > > > > DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
>> > > > > > > > DEBUG 1: Watch/Warning file:
>> > > > /opt/MET7/share/met/tc_data/wwpts_us.txt
>> > > > > > > > DEBUG 1: Output file:
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt.tcst
>> > > > > > > >
>> > > > > > > > On Wed, Nov 21, 2018 at 9:16 AM Rosalyn MacCracken -
NOAA
>> > > > Affiliate <
>> > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
>> > > > > > > >
>> > > > > > > > > Hi John,
>> > > > > > > > >
>> > > > > > > > > Yes, that makes sense, and yes, I am using Guang
Pings'
>> > > modified
>> > > > > > files.
>> > > > > > > > > I'm not sure how to run tc_pairs_wrapper.py,
though. I
>> > think I
>> > > > > need
>> > > > > > > some
>> > > > > > > > > type of *.conf file to pass in the files I need, or
>> something
>> > > > like
>> > > > > > > that.
>> > > > > > > > > I'm in the process of reading through the code now.
I
>> wonder
>> > > if
>> > > > I
>> > > > > > > could
>> > > > > > > > > just pull out the function process_non_atcf, to run
in a
>> > > > > stand-alone
>> > > > > > > > mode.
>> > > > > > > > > I wish there was some sort of documentation with
this
>> code...
>> > > > > > > > >
>> > > > > > > > > And, so, once I can read these files, I can use
tc_pairs
>> to
>> > > > create
>> > > > > > the
>> > > > > > > > > forecast error statistics, and then use tc_stats to
>> aggregate
>> > > the
>> > > > > > files
>> > > > > > > > and
>> > > > > > > > > get the mean, etc, for a period of time, correct?
Was
>> there
>> > > some
>> > > > > > sort
>> > > > > > > of
>> > > > > > > > > plotting scripts for the output files?
>> > > > > > > > >
>> > > > > > > > > So, with tc_pairs, I can define a region, like I
did with
>> > > > > point_stat,
>> > > > > > > > > correct? Can I use the same *.nc files, or do I
use the
>> > > original
>> > > > > > > *.poly
>> > > > > > > > > files?
>> > > > > > > > >
>> > > > > > > > > Roz
>> > > > > > > > >
>> > > > > > > > > On Tue, Nov 20, 2018 at 7:53 PM John Halley Gotway
via RT
>> <
>> > > > > > > > > met_help at ucar.edu> wrote:
>> > > > > > > > >
>> > > > > > > > >> Hi Roz,
>> > > > > > > > >>
>> > > > > > > > >> Yes, I see you’re using the modified ATCF format
that
>> Guang
>> > > Ping
>> > > > > > uses.
>> > > > > > > > >> You’re correct that tc_pairs doesn’t parse this
directly
>> > since
>> > > > it
>> > > > > > > > differs
>> > > > > > > > >> slightly from the ATCF format.
>> > > > > > > > >>
>> > > > > > > > >> When we’ve processed this data in the past, we
first
>> > > > pre-processed
>> > > > > > it
>> > > > > > > > in a
>> > > > > > > > >> shell script to:
>> > > > > > > > >>
>> > > > > > > > >> 1. Replace the -99 value with -9999 for missing
data.
>> > > > > > > > >> 2. Remove the 3rd column
>> > > > > > > > >> 3. Updating the 2nd column to include the year and
>> cyclone
>> > > > number.
>> > > > > > > > >>
>> > > > > > > > >> And that enable tc_pairs to read it.
>> > > > > > > > >>
>> > > > > > > > >> This is actually Done in python by one of the use
cases
>> in
>> > > > > METplus.
>> > > > > > > > >>
>> > > > > > > > >> Take a look at this file:
>> > > > > > > > >>
>> > > > >
>> https://github.com/NCAR/METplus/blob/master/ush/tc_pairs_wrapper.py
>> > > > > > > > >>
>> > > > > > > > >> At the function named:
>> > > > > > > > >> process_non_atcf
>> > > > > > > > >>
>> > > > > > > > >> I haven’t run this using METplus scripts myself,
but all
>> the
>> > > > > pieces
>> > > > > > of
>> > > > > > > > >> logic are there, and it should be possible.
>> > > > > > > > >>
>> > > > > > > > >> Make sense?
>> > > > > > > > >>
>> > > > > > > > >> Thanks
>> > > > > > > > >> John
>> > > > > > > > >>
>> > > > > > > > >> On Tue, Nov 20, 2018 at 1:13 PM Rosalyn MacCracken
- NOAA
>> > > > > Affiliate
>> > > > > > > via
>> > > > > > > > >> RT <
>> > > > > > > > >> met_help at ucar.edu> wrote:
>> > > > > > > > >>
>> > > > > > > > >> >
>> > > > > > > > >> > Tue Nov 20 13:13:05 2018: Request 87885 was
acted upon.
>> > > > > > > > >> > Transaction: Ticket created by
>> > rosalyn.maccracken at noaa.gov
>> > > > > > > > >> > Queue: met_help
>> > > > > > > > >> > Subject: Help with using TC_pairs
>> > > > > > > > >> > Owner: Nobody
>> > > > > > > > >> > Requestors: rosalyn.maccracken at noaa.gov
>> > > > > > > > >> > Status: new
>> > > > > > > > >> > Ticket <URL:
>> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
>> > > > > > > > >
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >> > Hi,
>> > > > > > > > >> >
>> > > > > > > > >> > I'm trying to use TC_pairs for extra-tropical
cyclones
>> > > > > (basin=ML).
>> > > > > > > I
>> > > > > > > > >> think
>> > > > > > > > >> > the format of the ATCF data is the issue, but,
I'm not
>> > sure.
>> > > > I
>> > > > > > have
>> > > > > > > > an
>> > > > > > > > >> > Adeck and a Bdeck file, where a sample of one of
the
>> lines
>> > > is:
>> > > > > > > > >> > ML, 0003, 2018090100_F000_687N_0159W_FOF,
2018090100,
>> 03,
>> > > > GFSO,
>> > > > > > 000,
>> > > > > > > > >> 687N,
>> > > > > > > > >> > 159W, 32, 987, XX, 34, NEQ, 0301, 0338, 0000,
0000,
>> > 988,
>> > > > > 52,
>> > > > > > > > -99,
>> > > > > > > > >> > 330, 86, -99, -9999, -9999, 175, 202, 137,
135,
>> 0081
>> > > > > > > > >> >
>> > > > > > > > >> > This is slightly different from the NRL ATCF
format on
>> > their
>> > > > > > > website,
>> > > > > > > > >> such
>> > > > > > > > >> > as the second column. Anyway, I configured the
config
>> > file
>> > > > very
>> > > > > > > > simply,
>> > > > > > > > >> > (model=GFS, basin=ML), and ran tc_pairs, and got
the
>> > output
>> > > to
>> > > > > the
>> > > > > > > > >> screen
>> > > > > > > > >> > below.
>> > > > > > > > >> >
>> > > > > > > > >> > Is there a way to work around the diferent ATCF
format?
>> > > I've
>> > > > > > > > attached a
>> > > > > > > > >> > sample of my Adeck and Bdeck files.
>> > > > > > > > >> >
>> > > > > > > > >> > Thanks,
>> > > > > > > > >> >
>> > > > > > > > >> > Roz
>> > > > > > > > >> >
>> > > > > > > > >> > Output to the screen:
>> > > > > > > > >> > /opt/MET7/bin/tc_pairs -adeck
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
~/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003
>> > > > > > > > >> > -bdeck
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
~/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003
>> > > > > > > > >> > -config
>> > > > > > > > >> >
>> > > > > > > >
>> > > > >
>> >
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
>> > > > > > > > >> > -out
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt
>> > > > > > > > >> > -log
>> > > > > > > > >>
>> > > > > >
>> >
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
>> > > > > > > > >> > -v 3
>> > > > > > > > >> > DEBUG 1: [Source 1 of 1] ADECK Source:
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003,
>> > > > > > > > >> > Model Suffix: (nul)
>> > > > > > > > >> > DEBUG 1: [Source 1 of 1] BDECK Source:
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003,
>> > > > > > > > >> > Model Suffix: (nul)
>> > > > > > > > >> > DEBUG 1: Config File Default:
>> > > > > > > > >> > /opt/MET7/share/met/config/TCPairsConfig_default
>> > > > > > > > >> > DEBUG 1: Config File User:
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
>> > > > > > > > >> > DEBUG 1: Distance to land file:
>> > /opt/MET7/share/met/tc_data/
>> > > > > > > > >> > dland_global_tenth_degree.nc
>> > > > > > > > >> > DEBUG 2: Processing 1 BDECK file(s).
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude
>> ...
>> > > "000"
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
>> ...
>> > > > "687N"
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude
>> ...
>> > > "000"
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
>> ...
>> > > > "701N"
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude
>> ...
>> > > "000"
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
>> ...
>> > > > "711N"
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude
>> ...
>> > > "000"
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
>> ...
>> > > > "720N"
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude
>> ...
>> > > "000"
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
>> ...
>> > > > "730N"
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude
>> ...
>> > > "000"
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
>> ...
>> > > > "741N"
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude
>> ...
>> > > "000"
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
>> ...
>> > > > "751N"
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude
>> ...
>> > > "000"
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
>> ...
>> > > > "760N"
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude
>> ...
>> > > "000"
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
>> ...
>> > > > "764N"
>> > > > > > > > >> > WARNING:
>> > > > > > > > >> > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
>> > > > > > > > >> > DEBUG 3: Identified 9 track(s).
>> > > > > > > > >> > DEBUG 2: Found 9 BDECK track(s).
>> > > > > > > > >> > DEBUG 2: Processing 1 ADECK file(s).
>> > > > > > > > >> > DEBUG 3: Used 0 of 22 lines read from 1 file(s).
>> > > > > > > > >> > DEBUG 3: Identified 0 track(s).
>> > > > > > > > >> > DEBUG 2: Deriving 12-hour interpolated ADECK
tracks.
>> > > > > > > > >> > DEBUG 2: Finished adding 0 and replacing 0
Interp12
>> > > track(s).
>> > > > > > > > >> > DEBUG 2: Deriving 0 ADECK consensus model(s).
>> > > > > > > > >> > DEBUG 2: Added 0 ADECK consensus tracks(s).
>> > > > > > > > >> > DEBUG 2: Deriving 0 ADECK lag model(s).
>> > > > > > > > >> > DEBUG 2: Added 0 ADECK lag tracks(s).
>> > > > > > > > >> > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline
model(s).
>> > > > > > > > >> > DEBUG 2: Added 0 CLIPER/SHIFOR baseline
track(s).
>> > > > > > > > >> > DEBUG 2: Filtering 0 ADECK tracks based on
config file
>> > > > settings.
>> > > > > > > > >> > DEBUG 3: Total tracks read = 0
>> > > > > > > > >> > DEBUG 3: Total tracks kept = 0
>> > > > > > > > >> > DEBUG 3: Rejected for storm name = 0
>> > > > > > > > >> > DEBUG 3: Rejected for valid time = 0
>> > > > > > > > >> > DEBUG 3: Rejected for required lead times = 0
>> > > > > > > > >> > DEBUG 3: Rejected for init mask = 0
>> > > > > > > > >> > DEBUG 3: Rejected for valid mask = 0
>> > > > > > > > >> > DEBUG 2: Matching 0 ADECK tracks to 9 BDECK
tracks.
>> > > > > > > > >> > DEBUG 1: Watch/Warning file:
>> > > > > > > /opt/MET7/share/met/tc_data/wwpts_us.txt
>> > > > > > > > >> > DEBUG 1: Output file:
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt.tcst
>> > > > > > > > >> >
>> > > > > > > > >> > --
>> > > > > > > > >> > Rosalyn MacCracken
>> > > > > > > > >> > Support Scientist
>> > > > > > > > >> >
>> > > > > > > > >> > Ocean Applications Branch
>> > > > > > > > >> > NOAA/NWS Ocean Prediction Center
>> > > > > > > > >> > NCWCP
>> > > > > > > > >> > 5830 University Research Ct
>> > > > > > > > >> > College Park, MD 20740-3818
>> > > > > > > > >> >
>> > > > > > > > >> > (p) 301-683-1551
>> > > > > > > > >> > rosalyn.maccracken at noaa.gov
>> > > > > > > > >> >
>> > > > > > > > >> >
>> > > > > > > > >>
>> > > > > > > > >>
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > > Rosalyn MacCracken
>> > > > > > > > > Support Scientist
>> > > > > > > > >
>> > > > > > > > > Ocean Applications Branch
>> > > > > > > > > NOAA/NWS Ocean Prediction Center
>> > > > > > > > > NCWCP
>> > > > > > > > > 5830 University Research Ct
>> > > > > > > > > College Park, MD 20740-3818
>> > > > > > > > >
>> > > > > > > > > (p) 301-683-1551
>> > > > > > > > > rosalyn.maccracken at noaa.gov
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > > Rosalyn MacCracken
>> > > > > > > > Support Scientist
>> > > > > > > >
>> > > > > > > > Ocean Applications Branch
>> > > > > > > > NOAA/NWS Ocean Prediction Center
>> > > > > > > > NCWCP
>> > > > > > > > 5830 University Research Ct
>> > > > > > > > College Park, MD 20740-3818
>> > > > > > > >
>> > > > > > > > (p) 301-683-1551
>> > > > > > > > rosalyn.maccracken at noaa.gov
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Rosalyn MacCracken
>> > > > > > Support Scientist
>> > > > > >
>> > > > > > Ocean Applications Branch
>> > > > > > NOAA/NWS Ocean Prediction Center
>> > > > > > NCWCP
>> > > > > > 5830 University Research Ct
>> > > > > > College Park, MD 20740-3818
>> > > > > >
>> > > > > > (p) 301-683-1551
>> > > > > > rosalyn.maccracken at noaa.gov
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > > --
>> > > > Rosalyn MacCracken
>> > > > Support Scientist
>> > > >
>> > > > Ocean Applications Branch
>> > > > NOAA/NWS Ocean Prediction Center
>> > > > NCWCP
>> > > > 5830 University Research Ct
>> > > > College Park, MD 20740-3818
>> > > >
>> > > > (p) 301-683-1551
>> > > > rosalyn.maccracken at noaa.gov
>> > > >
>> > > >
>> > >
>> > >
>> >
>> > --
>> > Rosalyn MacCracken
>> > Support Scientist
>> >
>> > Ocean Applications Branch
>> > NOAA/NWS Ocean Prediction Center
>> > NCWCP
>> > 5830 University Research Ct
>> > College Park, MD 20740-3818
>> >
>> > (p) 301-683-1551
>> > rosalyn.maccracken at noaa.gov
>> >
>> >
>>
>>
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Help with using TC_pairs
From: John Halley Gotway
Time: Thu Dec 06 10:05:26 2018
Roz,
Sure, you can do that with tc_stat. Just use tc_stat to run a "-
filter"
job without actually applying any filtering criteria. And then use
the
"-dump_row" option to write the filtered output to a file.
That will effectively concatenate multiple files into a single output
file.
John
On Thu, Dec 6, 2018 at 9:46 AM Rosalyn MacCracken - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
>
> Hi John,
>
> I got tc_stat to work the way I wanted to, and was able to produce
summary
> statistics. Now that I have matched pairs, I was thinking about
doing some
> other things, unrelated to MET (like data mining, clustering or
> classification of errors). To do this data mining, I bet it would
be
> easier if I had all the data in one big file of all the storm ids,
from
> 201711 - 201809. I could accomplish this with python, opening the
files
> and appending to dataframes, etc. But, what I thought I should do
is ask
> you if there was an easy way to do that in MET. Is there a way to
concat
> all the files with the *tcst ending into one big file?
>
> Thanks!
>
> Roz
>
> On Tue, Nov 27, 2018 at 2:31 PM Rosalyn MacCracken - NOAA Affiliate
<
> rosalyn.maccracken at noaa.gov> wrote:
>
> > Hi John,
> >
> > That worked! Thank you so much for seeing my little mistake. I
think,
> > originally, I had something in those brackets, and took it out,
but, left
> > the quotes inside the brackets. I was hoping it was something
easy like
> > that.
> >
> > So, I have a few more questions.
> > 1) I want to use a domain file, like I did with point_stat. For
that, I
> > created a *.poly file, then, converted it to a *.nc. Which do I
use for
> > tc_pairs?
> > 2) Is there a way to change the wind speed warning categories,
from
> > 34kts, 50kts and 64kts, just in case OPC might want some other
category?
> >
> > Thinking about what I'm going to do with this using tc_stat, I
will need
> > to calculate track errors(in nm) and intensity errors by pressure,
and
> > also, intensity errors by lead time. I imagine I will be able to
do that
> > using -column_thresh, or something else, correct? I saw that on
the
> Online
> > Tutorial page. But, I guess my question about those things would
be, is
> > there anything I need to make sure that is in my adeck/bdeck files
that
> is
> > not included now, that I should include to make those tasks
easier? I
> just
> > want to make sure that I've included everything I need.
> >
> > Roz
> >
> > On Tue, Nov 27, 2018 at 11:43 AM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Roz,
> >>
> >> Thanks for posting your sample data. I found the problem in your
> >> TCPairsConfig file. Change...
> >> FROM: storm_id = [ "" ];
> >> TO: storm_id = [ ];
> >>
> >> The storm_id entry is an array of storm ID's to match. An empty
array
> is
> >> just square brackets with nothing inside ([ ]). Since you had
> >> double-quotes inside, that's an array of length 1 containing an
empty
> >> string.
> >>
> >> Thanks,
> >> John
> >>
> >>
> >> On Tue, Nov 27, 2018 at 4:50 AM Rosalyn MacCracken - NOAA
Affiliate via
> >> RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
> >> >
> >> > Hi John,
> >> >
> >> > I uploaded the original files (Adeck = aml*, Bdeck=bml*,
without the
> >> > *modified.dat ending), the modified files (with the
*modified.dat
> >> ending)
> >> > and the config file into my ftp directory (*cd
> >> > incoming/irap/met_help/maccracken). *Let me know what you
find.
> >> >
> >> > Thanks for your help!
> >> >
> >> > Roz
> >> >
> >> > On Mon, Nov 26, 2018 at 5:22 PM John Halley Gotway via RT <
> >> > met_help at ucar.edu>
> >> > wrote:
> >> >
> >> > > Roz,
> >> > >
> >> > > Yes, the first 8 columns are required. The additional ones
are
> >> optional.
> >> > >
> >> > > Looking at your log output, I see that tc_pairs is skipping
over all
> >> 642
> >> > > input lines. And the question is why? If you send me your
latest
> >> ADeck,
> >> > > BDeck, and TC-Pairs config files, I could run it here and try
to
> >> figure
> >> > out
> >> > > why it isn't defining any ADECK tracks.
> >> > >
> >> > > Thanks,
> >> > > John
> >> > >
> >> > > On Mon, Nov 26, 2018 at 12:23 PM Rosalyn MacCracken - NOAA
Affiliate
> >> via
> >> > RT
> >> > > <met_help at ucar.edu> wrote:
> >> > >
> >> > > >
> >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
> >> > > >
> >> > > > So, can I just have 8 columns for the input file, or, do I
need to
> >> make
> >> > > > columns of -9999 or NaN for all the rest of the columns?
> >> > > >
> >> > > > I experimented, and just wrote out 10 columns for the input
file:
> >> > > > agfso_trackdf_mod =
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
>
agfso_trackdf[['basin_name','storm_id','model_start_date','const','model','fcst_lead_time','vort_lat_cent','vort_lon_cent','vort_10m_max',
> >> > > > 'vort_min_mslp','type','thresh_wspd']]
> >> > > >
> >> > > > because I wanted stats for these variables, and I got the
output
> >> below.
> >> > > > So, maybe it's working, but, the config file needs to be
> different?
> >> > Any
> >> > > > other ideas of what I should try doing/looking at?
> >> > > >
> >> > > > Roz
> >> > > >
> >> > > > Output:
> >> > > > DEBUG 1: [Source 1 of 1] ADECK Source:
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018083118.gfso.0053_modified.dat,
> >> > > > Model Suffix: _EXP
> >> > > > DEBUG 1: [Source 1 of 1] BDECK Source:
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018083118.gfso.0053_modified.dat,
> >> > > > Model Suffix: (nul)
> >> > > > DEBUG 1: Config File Default:
> >> > > > /opt/MET7/share/met/config/TCPairsConfig_default
> >> > > > DEBUG 1: Config File User:
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> >> > > > DEBUG 1: Distance to land file:
/opt/MET7/share/met/tc_data/
> >> > > > dland_global_tenth_degree.nc
> >> > > > DEBUG 4:
> >> > > > DEBUG 4: Latitude/Longitude Grid Data:
> >> > > > DEBUG 4: lat_ll: -90
> >> > > > DEBUG 4: lon_ll: 180
> >> > > > DEBUG 4: delta_lat: 0.1
> >> > > > DEBUG 4: delta_lon: 0.1
> >> > > > DEBUG 4: Nlat: 1801
> >> > > > DEBUG 4: Nlon: 3601
> >> > > > DEBUG 4:
> >> > > > DEBUG 2: Processing 1 BDECK file(s).
> >> > > > DEBUG 4: [File 1 of 1] Used 30 of 30 lines read from file
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018083118.gfso.0053_modified.dat"
> >> > > > DEBUG 3: Used 30 of 30 lines read from 1 file(s).
> >> > > > DEBUG 3: Identified 1 track(s).
> >> > > > DEBUG 4: [Track 1 of 1] TrackInfo: StormId =
"ML0800532018",
> IsBest
> >> =
> >> > > > false, IsOper = false, CheckAnly = true, IsAnly = true,
Basin =
> >> "ML",
> >> > > > Cyclone = "080053", StormName = "", TechniqueNumber = 3,
> Technique =
> >> > > > "GFSO", Initials = "", InitTime = NA, MinValidTime =
> >> 20180805_060000,
> >> > > > MaxValidTime = 20180812_120000, NPoints = 30, NAlloc = 100,
> >> NTrackLines
> >> > > = 0
> >> > > > DEBUG 2: Found 1 BDECK track(s).
> >> > > > DEBUG 2: Processing 1 ADECK file(s).
> >> > > > DEBUG 4: [File 1 of 1] Used 0 of 642 lines read from file
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018083118.gfso.0053_modified.dat"
> >> > > > DEBUG 3: Used 0 of 642 lines read from 1 file(s).
> >> > > > DEBUG 3: Identified 0 track(s).
> >> > > > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> >> > > > DEBUG 2: Finished adding 0 and replacing 0 Interp12
track(s).
> >> > > > DEBUG 2: Deriving 0 ADECK consensus model(s).
> >> > > > DEBUG 2: Added 0 ADECK consensus tracks(s).
> >> > > > DEBUG 2: Deriving 0 ADECK lag model(s).
> >> > > > DEBUG 2: Added 0 ADECK lag tracks(s).
> >> > > > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> >> > > > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> >> > > > DEBUG 2: Filtering 0 ADECK tracks based on config file
settings.
> >> > > > DEBUG 3: Total tracks read = 0
> >> > > > DEBUG 3: Total tracks kept = 0
> >> > > > DEBUG 3: Rejected for storm name = 0
> >> > > > DEBUG 3: Rejected for valid time = 0
> >> > > > DEBUG 3: Rejected for required lead times = 0
> >> > > > DEBUG 3: Rejected for init mask = 0
> >> > > > DEBUG 3: Rejected for valid mask = 0
> >> > > > DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
> >> > > > DEBUG 1: Watch/Warning file:
> >> /opt/MET7/share/met/tc_data/wwpts_us.txt
> >> > > > DEBUG 1: Output file:
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018083118.gfso.0053_test_modified_output.txt.tcst
> >> > > >
> >> > > >
> >> > > > On Mon, Nov 26, 2018 at 2:06 PM John Halley Gotway via RT <
> >> > > > met_help at ucar.edu>
> >> > > > wrote:
> >> > > >
> >> > > > > Roz,
> >> > > > >
> >> > > > > Page 334 of the user's guide (
> >> > > > >
> >> > >
> >>
>
https://dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v8.0.pdf
> >> > > > )
> >> > > > > mentions that only the first 8 ATCF columns are required.
If
> >> > > additional
> >> > > > > columns are present in the input ATCF files, then
tc_pairs will
> >> > > interpret
> >> > > > > them as specified by the ATCF format.
> >> > > > >
> >> > > > > So if the storm name column is present, MET in interpret
those
> >> values
> >> > > as
> >> > > > > the storm name.
> >> > > > >
> >> > > > > For Guang Ping's data, probably best to have that column
empty.
> >> > > > >
> >> > > > > John
> >> > > > >
> >> > > > >
> >> > > > > On Mon, Nov 26, 2018 at 11:51 AM Rosalyn MacCracken -
NOAA
> >> Affiliate
> >> > > via
> >> > > > RT
> >> > > > > <met_help at ucar.edu> wrote:
> >> > > > >
> >> > > > > >
> >> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
> >
> >> > > > > >
> >> > > > > > Hi John,
> >> > > > > >
> >> > > > > > Ok, I will ignore the suffix error. I do have numeric
values
> in
> >> > > > > > STORMNAME. I wonder if in the modified files I have,
this
> >> column
> >> > is
> >> > > > not
> >> > > > > > actually STORMNAME, but, some other variable. Now, I
don't
> have
> >> > the
> >> > > > > number
> >> > > > > > of columns that the NRL website has. Will that be a
problem?
> >> > > Should I
> >> > > > > > create dummy columns?
> >> > > > > >
> >> > > > > > Roz
> >> > > > > >
> >> > > > > > On Mon, Nov 26, 2018 at 1:14 PM John Halley Gotway via
RT <
> >> > > > > > met_help at ucar.edu>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Roz,
> >> > > > > > >
> >> > > > > > > I see that you're wondering about this log message:
> >> > > > > > > *Model Suffix: (nul)*
> >> > > > > > >
> >> > > > > > > Everything is fine, you do not need to be concerned
about
> >> this.
> >> > > > > Here's a
> >> > > > > > > line taken from the usage statement for the tc_pairs
tool:
> >> > > > > > > *Note: The "-adeck", "-edeck", and "-bdeck" options
may
> >> include
> >> > > > > > > "suffix=string" to modify the model names from that
source.*
> >> > > > > > >
> >> > > > > > > Since you didn't specify "suffix=string" on the
command
> line,
> >> the
> >> > > > > suffix
> >> > > > > > > remains the default value of an empty string, i.e.
"(nul)".
> >> So
> >> > why
> >> > > > > would
> >> > > > > > > you ever want to use a suffix? Sometimes data is
messy. We
> >> were
> >> > > > > running
> >> > > > > > > tc_pairs to evaluate track data from multiple sources
and we
> >> > found
> >> > > > that
> >> > > > > > > some model names we reused in both sources. Let's
say the
> >> OFCL
> >> > > > > forecast
> >> > > > > > > showed up in both stream 1.0 and stream 1.5 data.
> Specifying
> >> a
> >> > > > suffix
> >> > > > > > such
> >> > > > > > > as "suffix=_STREAM1.5" changes the model name from
"OFCL" to
> >> > > > > > > "OFCL_STREAM1.5". And that enables us to distinguish
> between
> >> > those
> >> > > > > > tracks
> >> > > > > > > in the output.
> >> > > > > > >
> >> > > > > > > But uou probably don't have any need for this option
in your
> >> > > current
> >> > > > > > work.
> >> > > > > > >
> >> > > > > > > However, I am concerned about these log messages:
> >> > > > > > > DEBUG 4: Updating storm name from "-287" to "-305"
for
> >> > > ML0900022018.
> >> > > > > > > DEBUG 4: Updating storm name from "-305" to "-255"
for
> >> > > ML0900022018.
> >> > > > > > > DEBUG 4: Updating storm name from "-255" to "-203"
for
> >> > > ML0900022018.
> >> > > > > > > DEBUG 4: Updating storm name from "-203" to "-205"
for
> >> > > ML0900022018.
> >> > > > > > > DEBUG 4: Updating storm name from "-205" to "-217"
for
> >> > > ML0900022018.
> >> > > > > > > DEBUG 4: Updating storm name from "-217" to "-211"
for
> >> > > ML0900022018.
> >> > > > > > > DEBUG 4: Updating storm name from "-211" to "-182"
for
> >> > > ML0900022018.
> >> > > > > > > DEBUG 4: Updating storm name from "-182" to "-138"
for
> >> > > ML0900022018.
> >> > > > > > >
> >> > > > > > > You likely have numeric values showing up in the
"STORMNAME"
> >> ATCF
> >> > > > > column:
> >> > > > > > >
> >> > >
https://www.nrlmry.navy.mil/atcf_web/docs/database/new/abdeck.txt
> >> > > > > > >
> >> > > > > > > I'd recommend looking into that.
> >> > > > > > >
> >> > > > > > > Thanks,
> >> > > > > > > John
> >> > > > > > >
> >> > > > > > > On Mon, Nov 26, 2018 at 8:44 AM Rosalyn MacCracken -
NOAA
> >> > Affiliate
> >> > > > via
> >> > > > > > RT
> >> > > > > > > <
> >> > > > > > > met_help at ucar.edu> wrote:
> >> > > > > > >
> >> > > > > > > >
> >> > > > > > > > <URL:
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
> >> > >
> >> > > > > > > >
> >> > > > > > > > Hi John,
> >> > > > > > > >
> >> > > > > > > > I figured out an easy way to read in and write out
Guang
> >> Ping's
> >> > > > > > modified
> >> > > > > > > > ATCF file, so that it:
> >> > > > > > > > 1) Removes the 3rd column
> >> > > > > > > > 2) Replaces -99 with -9999 in column 21 and 24
(after the
> >> 3rd
> >> > > > column
> >> > > > > > is
> >> > > > > > > > removed)
> >> > > > > > > > 3) Updates column 2 to include month+cyclone
number.
> (You
> >> had
> >> > > > said
> >> > > > > to
> >> > > > > > > do
> >> > > > > > > > year+cyclone #, but the tc_pairs_wrapper.py said to
do
> >> > > > > month+cyclone#)
> >> > > > > > > >
> >> > > > > > > > I changed Adeck and Bdeck.
> >> > > > > > > > Just a couple lines of my file:
> >> > > > > > > >
> >> > > > > > > > ML, 090002, 2018090100, 03, GFSO, 000, 511S, 897W,
40,
> >> 976,
> >> > > XX,
> >> > > > > 34,
> >> > > > > > > > NEQ, 0625, 0630, 0000, 0226, 978, 68, -9999,
123, 155,
> >> > -9999,
> >> > > > > -9999,
> >> > > > > > > > -9999, -297, -394, -287, -458, 0073
> >> > > > > > > > ML, 090002, 2018090106, 03, GFSO, 000, 537S, 854W,
44,
> >> 964,
> >> > > XX,
> >> > > > > 34,
> >> > > > > > > > NEQ, 0640, 0645, 0000, 0297, 969, 108, -9999,
119, 147,
> >> > -9999,
> >> > > > > -9999,
> >> > > > > > > > -9999, -403, -648, -305, -394, 0055
> >> > > > > > > >
> >> > > > > > > > So, I've run tc_pairs again and my output is below.
I
> don't
> >> > > > > understand
> >> > > > > > > the
> >> > > > > > > > Model Suffix: (nul). Is the input format still
wrong?
> >> > > > > > > >
> >> > > > > > > > Roz
> >> > > > > > > >
> >> > > > > > > > Output:
> >> > > > > > > > /opt/MET7/bin/tc_pairs -adeck
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat
> >> > > > > > > > -bdeck
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat
> >> > > > > > > > -config
> >> > > > > >
> ~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> >> > > > > > > > -out
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt
> >> > > > > > > > -log
> >> > > > > > >
> >> > >
>
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> >> > > > > > > > -v 4
> >> > > > > > > > DEBUG 1: [Source 1 of 1] ADECK Source:
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat,
> >> > > > > > > > Model Suffix: (nul)
> >> > > > > > > > DEBUG 1: [Source 1 of 1] BDECK Source:
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat,
> >> > > > > > > > Model Suffix: (nul)
> >> > > > > > > > DEBUG 1: Config File Default:
> >> > > > > > > > /opt/MET7/share/met/config/TCPairsConfig_default
> >> > > > > > > > DEBUG 1: Config File User:
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> >> > > > > > > > DEBUG 1: Distance to land file:
> /opt/MET7/share/met/tc_data/
> >> > > > > > > > dland_global_tenth_degree.nc
> >> > > > > > > > DEBUG 4:
> >> > > > > > > > DEBUG 4: Latitude/Longitude Grid Data:
> >> > > > > > > > DEBUG 4: lat_ll: -90
> >> > > > > > > > DEBUG 4: lon_ll: 180
> >> > > > > > > > DEBUG 4: delta_lat: 0.1
> >> > > > > > > > DEBUG 4: delta_lon: 0.1
> >> > > > > > > > DEBUG 4: Nlat: 1801
> >> > > > > > > > DEBUG 4: Nlon: 3601
> >> > > > > > > > DEBUG 4:
> >> > > > > > > > DEBUG 2: Processing 1 BDECK file(s).
> >> > > > > > > > DEBUG 4: Updating storm name from "-287" to "-305"
for
> >> > > > ML0900022018.
> >> > > > > > > > DEBUG 4: Updating storm name from "-305" to "-255"
for
> >> > > > ML0900022018.
> >> > > > > > > > DEBUG 4: Updating storm name from "-255" to "-203"
for
> >> > > > ML0900022018.
> >> > > > > > > > DEBUG 4: Updating storm name from "-203" to "-205"
for
> >> > > > ML0900022018.
> >> > > > > > > > DEBUG 4: Updating storm name from "-205" to "-217"
for
> >> > > > ML0900022018.
> >> > > > > > > > DEBUG 4: Updating storm name from "-217" to "-211"
for
> >> > > > ML0900022018.
> >> > > > > > > > DEBUG 4: Updating storm name from "-211" to "-182"
for
> >> > > > ML0900022018.
> >> > > > > > > > DEBUG 4: Updating storm name from "-182" to "-138"
for
> >> > > > ML0900022018.
> >> > > > > > > > DEBUG 4: [File 1 of 1] Used 9 of 9 lines read from
file
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat"
> >> > > > > > > > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
> >> > > > > > > > DEBUG 3: Identified 1 track(s).
> >> > > > > > > > DEBUG 4: [Track 1 of 1] TrackInfo: StormId =
> "ML0900022018",
> >> > > > IsBest =
> >> > > > > > > > false, IsOper = false, CheckAnly = true, IsAnly =
true,
> >> Basin =
> >> > > > "ML",
> >> > > > > > > > Cyclone = "090002", StormName = "-138",
TechniqueNumber =
> 3,
> >> > > > > Technique
> >> > > > > > =
> >> > > > > > > > "GFSO", Initials = "-9999", InitTime = NA,
MinValidTime =
> >> > > > > > > 20180901_000000,
> >> > > > > > > > MaxValidTime = 20180903_000000, NPoints = 9, NAlloc
= 100,
> >> > > > > NTrackLines
> >> > > > > > =
> >> > > > > > > 0
> >> > > > > > > > DEBUG 2: Found 1 BDECK track(s).
> >> > > > > > > > DEBUG 2: Processing 1 ADECK file(s).
> >> > > > > > > > DEBUG 4: [File 1 of 1] Used 0 of 21 lines read from
file
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat"
> >> > > > > > > > DEBUG 3: Used 0 of 21 lines read from 1 file(s).
> >> > > > > > > > DEBUG 3: Identified 0 track(s).
> >> > > > > > > > DEBUG 2: Deriving 12-hour interpolated ADECK
tracks.
> >> > > > > > > > DEBUG 2: Finished adding 0 and replacing 0 Interp12
> >> track(s).
> >> > > > > > > > DEBUG 2: Deriving 0 ADECK consensus model(s).
> >> > > > > > > > DEBUG 2: Added 0 ADECK consensus tracks(s).
> >> > > > > > > > DEBUG 2: Deriving 0 ADECK lag model(s).
> >> > > > > > > > DEBUG 2: Added 0 ADECK lag tracks(s).
> >> > > > > > > > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline
model(s).
> >> > > > > > > > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> >> > > > > > > > DEBUG 2: Filtering 0 ADECK tracks based on config
file
> >> > settings.
> >> > > > > > > > DEBUG 3: Total tracks read = 0
> >> > > > > > > > DEBUG 3: Total tracks kept = 0
> >> > > > > > > > DEBUG 3: Rejected for storm name = 0
> >> > > > > > > > DEBUG 3: Rejected for valid time = 0
> >> > > > > > > > DEBUG 3: Rejected for required lead times = 0
> >> > > > > > > > DEBUG 3: Rejected for init mask = 0
> >> > > > > > > > DEBUG 3: Rejected for valid mask = 0
> >> > > > > > > > DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
> >> > > > > > > > DEBUG 1: Watch/Warning file:
> >> > > > /opt/MET7/share/met/tc_data/wwpts_us.txt
> >> > > > > > > > DEBUG 1: Output file:
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt.tcst
> >> > > > > > > >
> >> > > > > > > > On Wed, Nov 21, 2018 at 9:16 AM Rosalyn MacCracken
- NOAA
> >> > > > Affiliate <
> >> > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> >> > > > > > > >
> >> > > > > > > > > Hi John,
> >> > > > > > > > >
> >> > > > > > > > > Yes, that makes sense, and yes, I am using Guang
Pings'
> >> > > modified
> >> > > > > > files.
> >> > > > > > > > > I'm not sure how to run tc_pairs_wrapper.py,
though. I
> >> > think I
> >> > > > > need
> >> > > > > > > some
> >> > > > > > > > > type of *.conf file to pass in the files I need,
or
> >> something
> >> > > > like
> >> > > > > > > that.
> >> > > > > > > > > I'm in the process of reading through the code
now. I
> >> wonder
> >> > > if
> >> > > > I
> >> > > > > > > could
> >> > > > > > > > > just pull out the function process_non_atcf, to
run in a
> >> > > > > stand-alone
> >> > > > > > > > mode.
> >> > > > > > > > > I wish there was some sort of documentation with
this
> >> code...
> >> > > > > > > > >
> >> > > > > > > > > And, so, once I can read these files, I can use
tc_pairs
> >> to
> >> > > > create
> >> > > > > > the
> >> > > > > > > > > forecast error statistics, and then use tc_stats
to
> >> aggregate
> >> > > the
> >> > > > > > files
> >> > > > > > > > and
> >> > > > > > > > > get the mean, etc, for a period of time, correct?
Was
> >> there
> >> > > some
> >> > > > > > sort
> >> > > > > > > of
> >> > > > > > > > > plotting scripts for the output files?
> >> > > > > > > > >
> >> > > > > > > > > So, with tc_pairs, I can define a region, like I
did
> with
> >> > > > > point_stat,
> >> > > > > > > > > correct? Can I use the same *.nc files, or do I
use the
> >> > > original
> >> > > > > > > *.poly
> >> > > > > > > > > files?
> >> > > > > > > > >
> >> > > > > > > > > Roz
> >> > > > > > > > >
> >> > > > > > > > > On Tue, Nov 20, 2018 at 7:53 PM John Halley
Gotway via
> RT
> >> <
> >> > > > > > > > > met_help at ucar.edu> wrote:
> >> > > > > > > > >
> >> > > > > > > > >> Hi Roz,
> >> > > > > > > > >>
> >> > > > > > > > >> Yes, I see you’re using the modified ATCF format
that
> >> Guang
> >> > > Ping
> >> > > > > > uses.
> >> > > > > > > > >> You’re correct that tc_pairs doesn’t parse this
> directly
> >> > since
> >> > > > it
> >> > > > > > > > differs
> >> > > > > > > > >> slightly from the ATCF format.
> >> > > > > > > > >>
> >> > > > > > > > >> When we’ve processed this data in the past, we
first
> >> > > > pre-processed
> >> > > > > > it
> >> > > > > > > > in a
> >> > > > > > > > >> shell script to:
> >> > > > > > > > >>
> >> > > > > > > > >> 1. Replace the -99 value with -9999 for missing
data.
> >> > > > > > > > >> 2. Remove the 3rd column
> >> > > > > > > > >> 3. Updating the 2nd column to include the year
and
> >> cyclone
> >> > > > number.
> >> > > > > > > > >>
> >> > > > > > > > >> And that enable tc_pairs to read it.
> >> > > > > > > > >>
> >> > > > > > > > >> This is actually Done in python by one of the
use cases
> >> in
> >> > > > > METplus.
> >> > > > > > > > >>
> >> > > > > > > > >> Take a look at this file:
> >> > > > > > > > >>
> >> > > > >
> >>
https://github.com/NCAR/METplus/blob/master/ush/tc_pairs_wrapper.py
> >> > > > > > > > >>
> >> > > > > > > > >> At the function named:
> >> > > > > > > > >> process_non_atcf
> >> > > > > > > > >>
> >> > > > > > > > >> I haven’t run this using METplus scripts myself,
but
> all
> >> the
> >> > > > > pieces
> >> > > > > > of
> >> > > > > > > > >> logic are there, and it should be possible.
> >> > > > > > > > >>
> >> > > > > > > > >> Make sense?
> >> > > > > > > > >>
> >> > > > > > > > >> Thanks
> >> > > > > > > > >> John
> >> > > > > > > > >>
> >> > > > > > > > >> On Tue, Nov 20, 2018 at 1:13 PM Rosalyn
MacCracken -
> NOAA
> >> > > > > Affiliate
> >> > > > > > > via
> >> > > > > > > > >> RT <
> >> > > > > > > > >> met_help at ucar.edu> wrote:
> >> > > > > > > > >>
> >> > > > > > > > >> >
> >> > > > > > > > >> > Tue Nov 20 13:13:05 2018: Request 87885 was
acted
> upon.
> >> > > > > > > > >> > Transaction: Ticket created by
> >> > rosalyn.maccracken at noaa.gov
> >> > > > > > > > >> > Queue: met_help
> >> > > > > > > > >> > Subject: Help with using TC_pairs
> >> > > > > > > > >> > Owner: Nobody
> >> > > > > > > > >> > Requestors: rosalyn.maccracken at noaa.gov
> >> > > > > > > > >> > Status: new
> >> > > > > > > > >> > Ticket <URL:
> >> > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
> >> > > > > > > > >
> >> > > > > > > > >> >
> >> > > > > > > > >> >
> >> > > > > > > > >> > Hi,
> >> > > > > > > > >> >
> >> > > > > > > > >> > I'm trying to use TC_pairs for extra-tropical
> cyclones
> >> > > > > (basin=ML).
> >> > > > > > > I
> >> > > > > > > > >> think
> >> > > > > > > > >> > the format of the ATCF data is the issue, but,
I'm
> not
> >> > sure.
> >> > > > I
> >> > > > > > have
> >> > > > > > > > an
> >> > > > > > > > >> > Adeck and a Bdeck file, where a sample of one
of the
> >> lines
> >> > > is:
> >> > > > > > > > >> > ML, 0003, 2018090100_F000_687N_0159W_FOF,
2018090100,
> >> 03,
> >> > > > GFSO,
> >> > > > > > 000,
> >> > > > > > > > >> 687N,
> >> > > > > > > > >> > 159W, 32, 987, XX, 34, NEQ, 0301, 0338,
0000,
> 0000,
> >> > 988,
> >> > > > > 52,
> >> > > > > > > > -99,
> >> > > > > > > > >> > 330, 86, -99, -9999, -9999, 175, 202, 137,
135,
> >> 0081
> >> > > > > > > > >> >
> >> > > > > > > > >> > This is slightly different from the NRL ATCF
format
> on
> >> > their
> >> > > > > > > website,
> >> > > > > > > > >> such
> >> > > > > > > > >> > as the second column. Anyway, I configured
the
> config
> >> > file
> >> > > > very
> >> > > > > > > > simply,
> >> > > > > > > > >> > (model=GFS, basin=ML), and ran tc_pairs, and
got the
> >> > output
> >> > > to
> >> > > > > the
> >> > > > > > > > >> screen
> >> > > > > > > > >> > below.
> >> > > > > > > > >> >
> >> > > > > > > > >> > Is there a way to work around the diferent
ATCF
> format?
> >> > > I've
> >> > > > > > > > attached a
> >> > > > > > > > >> > sample of my Adeck and Bdeck files.
> >> > > > > > > > >> >
> >> > > > > > > > >> > Thanks,
> >> > > > > > > > >> >
> >> > > > > > > > >> > Roz
> >> > > > > > > > >> >
> >> > > > > > > > >> > Output to the screen:
> >> > > > > > > > >> > /opt/MET7/bin/tc_pairs -adeck
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
~/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003
> >> > > > > > > > >> > -bdeck
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
~/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003
> >> > > > > > > > >> > -config
> >> > > > > > > > >> >
> >> > > > > > > >
> >> > > > >
> >> >
>
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> >> > > > > > > > >> > -out
> >> > > > > > > > >> >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt
> >> > > > > > > > >> > -log
> >> > > > > > > > >>
> >> > > > > >
> >> >
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> >> > > > > > > > >> > -v 3
> >> > > > > > > > >> > DEBUG 1: [Source 1 of 1] ADECK Source:
> >> > > > > > > > >> >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003,
> >> > > > > > > > >> > Model Suffix: (nul)
> >> > > > > > > > >> > DEBUG 1: [Source 1 of 1] BDECK Source:
> >> > > > > > > > >> >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003,
> >> > > > > > > > >> > Model Suffix: (nul)
> >> > > > > > > > >> > DEBUG 1: Config File Default:
> >> > > > > > > > >> >
/opt/MET7/share/met/config/TCPairsConfig_default
> >> > > > > > > > >> > DEBUG 1: Config File User:
> >> > > > > > > > >> >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> >> > > > > > > > >> > DEBUG 1: Distance to land file:
> >> > /opt/MET7/share/met/tc_data/
> >> > > > > > > > >> > dland_global_tenth_degree.nc
> >> > > > > > > > >> > DEBUG 2: Processing 1 BDECK file(s).
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude
> >> ...
> >> > > "000"
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
> >> ...
> >> > > > "687N"
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude
> >> ...
> >> > > "000"
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
> >> ...
> >> > > > "701N"
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude
> >> ...
> >> > > "000"
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
> >> ...
> >> > > > "711N"
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude
> >> ...
> >> > > "000"
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
> >> ...
> >> > > > "720N"
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude
> >> ...
> >> > > "000"
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
> >> ...
> >> > > > "730N"
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude
> >> ...
> >> > > "000"
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
> >> ...
> >> > > > "741N"
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude
> >> ...
> >> > > "000"
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
> >> ...
> >> > > > "751N"
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude
> >> ...
> >> > > "000"
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
> >> ...
> >> > > > "760N"
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
latitude
> >> ...
> >> > > "000"
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
longitude
> >> ...
> >> > > > "764N"
> >> > > > > > > > >> > WARNING:
> >> > > > > > > > >> > DEBUG 3: Used 9 of 9 lines read from 1
file(s).
> >> > > > > > > > >> > DEBUG 3: Identified 9 track(s).
> >> > > > > > > > >> > DEBUG 2: Found 9 BDECK track(s).
> >> > > > > > > > >> > DEBUG 2: Processing 1 ADECK file(s).
> >> > > > > > > > >> > DEBUG 3: Used 0 of 22 lines read from 1
file(s).
> >> > > > > > > > >> > DEBUG 3: Identified 0 track(s).
> >> > > > > > > > >> > DEBUG 2: Deriving 12-hour interpolated ADECK
tracks.
> >> > > > > > > > >> > DEBUG 2: Finished adding 0 and replacing 0
Interp12
> >> > > track(s).
> >> > > > > > > > >> > DEBUG 2: Deriving 0 ADECK consensus model(s).
> >> > > > > > > > >> > DEBUG 2: Added 0 ADECK consensus tracks(s).
> >> > > > > > > > >> > DEBUG 2: Deriving 0 ADECK lag model(s).
> >> > > > > > > > >> > DEBUG 2: Added 0 ADECK lag tracks(s).
> >> > > > > > > > >> > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline
model(s).
> >> > > > > > > > >> > DEBUG 2: Added 0 CLIPER/SHIFOR baseline
track(s).
> >> > > > > > > > >> > DEBUG 2: Filtering 0 ADECK tracks based on
config
> file
> >> > > > settings.
> >> > > > > > > > >> > DEBUG 3: Total tracks read = 0
> >> > > > > > > > >> > DEBUG 3: Total tracks kept = 0
> >> > > > > > > > >> > DEBUG 3: Rejected for storm name = 0
> >> > > > > > > > >> > DEBUG 3: Rejected for valid time = 0
> >> > > > > > > > >> > DEBUG 3: Rejected for required lead times = 0
> >> > > > > > > > >> > DEBUG 3: Rejected for init mask = 0
> >> > > > > > > > >> > DEBUG 3: Rejected for valid mask = 0
> >> > > > > > > > >> > DEBUG 2: Matching 0 ADECK tracks to 9 BDECK
tracks.
> >> > > > > > > > >> > DEBUG 1: Watch/Warning file:
> >> > > > > > > /opt/MET7/share/met/tc_data/wwpts_us.txt
> >> > > > > > > > >> > DEBUG 1: Output file:
> >> > > > > > > > >> >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt.tcst
> >> > > > > > > > >> >
> >> > > > > > > > >> > --
> >> > > > > > > > >> > Rosalyn MacCracken
> >> > > > > > > > >> > Support Scientist
> >> > > > > > > > >> >
> >> > > > > > > > >> > Ocean Applications Branch
> >> > > > > > > > >> > NOAA/NWS Ocean Prediction Center
> >> > > > > > > > >> > NCWCP
> >> > > > > > > > >> > 5830 University Research Ct
> >> > > > > > > > >> > College Park, MD 20740-3818
> >> > > > > > > > >> >
> >> > > > > > > > >> > (p) 301-683-1551
> >> > > > > > > > >> > rosalyn.maccracken at noaa.gov
> >> > > > > > > > >> >
> >> > > > > > > > >> >
> >> > > > > > > > >>
> >> > > > > > > > >>
> >> > > > > > > > >
> >> > > > > > > > > --
> >> > > > > > > > > Rosalyn MacCracken
> >> > > > > > > > > Support Scientist
> >> > > > > > > > >
> >> > > > > > > > > Ocean Applications Branch
> >> > > > > > > > > NOAA/NWS Ocean Prediction Center
> >> > > > > > > > > NCWCP
> >> > > > > > > > > 5830 University Research Ct
> >> > > > > > > > > College Park, MD 20740-3818
> >> > > > > > > > >
> >> > > > > > > > > (p) 301-683-1551
> >> > > > > > > > > rosalyn.maccracken at noaa.gov
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > --
> >> > > > > > > > Rosalyn MacCracken
> >> > > > > > > > Support Scientist
> >> > > > > > > >
> >> > > > > > > > Ocean Applications Branch
> >> > > > > > > > NOAA/NWS Ocean Prediction Center
> >> > > > > > > > NCWCP
> >> > > > > > > > 5830 University Research Ct
> >> > > > > > > > College Park, MD 20740-3818
> >> > > > > > > >
> >> > > > > > > > (p) 301-683-1551
> >> > > > > > > > rosalyn.maccracken at noaa.gov
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > > > --
> >> > > > > > Rosalyn MacCracken
> >> > > > > > Support Scientist
> >> > > > > >
> >> > > > > > Ocean Applications Branch
> >> > > > > > NOAA/NWS Ocean Prediction Center
> >> > > > > > NCWCP
> >> > > > > > 5830 University Research Ct
> >> > > > > > College Park, MD 20740-3818
> >> > > > > >
> >> > > > > > (p) 301-683-1551
> >> > > > > > rosalyn.maccracken at noaa.gov
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > > > --
> >> > > > Rosalyn MacCracken
> >> > > > Support Scientist
> >> > > >
> >> > > > Ocean Applications Branch
> >> > > > NOAA/NWS Ocean Prediction Center
> >> > > > NCWCP
> >> > > > 5830 University Research Ct
> >> > > > College Park, MD 20740-3818
> >> > > >
> >> > > > (p) 301-683-1551
> >> > > > rosalyn.maccracken at noaa.gov
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> >
> >> > --
> >> > Rosalyn MacCracken
> >> > Support Scientist
> >> >
> >> > Ocean Applications Branch
> >> > NOAA/NWS Ocean Prediction Center
> >> > NCWCP
> >> > 5830 University Research Ct
> >> > College Park, MD 20740-3818
> >> >
> >> > (p) 301-683-1551
> >> > rosalyn.maccracken at noaa.gov
> >> >
> >> >
> >>
> >>
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applications Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
>
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
>
------------------------------------------------
Subject: Help with using TC_pairs
From: Rosalyn MacCracken - NOAA Affiliate
Time: Thu Dec 06 10:07:49 2018
Hi John,
Oh, thanks! Is this better than just doing a "cat * > merged_file" in
linux, or does it essentially do the same thing?
Roz
On Thu, Dec 6, 2018 at 12:05 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Roz,
>
> Sure, you can do that with tc_stat. Just use tc_stat to run a "-
filter"
> job without actually applying any filtering criteria. And then use
the
> "-dump_row" option to write the filtered output to a file.
>
> That will effectively concatenate multiple files into a single
output file.
>
> John
>
> On Thu, Dec 6, 2018 at 9:46 AM Rosalyn MacCracken - NOAA Affiliate
via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
> >
> > Hi John,
> >
> > I got tc_stat to work the way I wanted to, and was able to produce
> summary
> > statistics. Now that I have matched pairs, I was thinking about
doing
> some
> > other things, unrelated to MET (like data mining, clustering or
> > classification of errors). To do this data mining, I bet it would
be
> > easier if I had all the data in one big file of all the storm ids,
from
> > 201711 - 201809. I could accomplish this with python, opening the
files
> > and appending to dataframes, etc. But, what I thought I should do
is ask
> > you if there was an easy way to do that in MET. Is there a way to
concat
> > all the files with the *tcst ending into one big file?
> >
> > Thanks!
> >
> > Roz
> >
> > On Tue, Nov 27, 2018 at 2:31 PM Rosalyn MacCracken - NOAA
Affiliate <
> > rosalyn.maccracken at noaa.gov> wrote:
> >
> > > Hi John,
> > >
> > > That worked! Thank you so much for seeing my little mistake. I
think,
> > > originally, I had something in those brackets, and took it out,
but,
> left
> > > the quotes inside the brackets. I was hoping it was something
easy
> like
> > > that.
> > >
> > > So, I have a few more questions.
> > > 1) I want to use a domain file, like I did with point_stat.
For
> that, I
> > > created a *.poly file, then, converted it to a *.nc. Which do I
use
> for
> > > tc_pairs?
> > > 2) Is there a way to change the wind speed warning categories,
from
> > > 34kts, 50kts and 64kts, just in case OPC might want some other
> category?
> > >
> > > Thinking about what I'm going to do with this using tc_stat, I
will
> need
> > > to calculate track errors(in nm) and intensity errors by
pressure, and
> > > also, intensity errors by lead time. I imagine I will be able
to do
> that
> > > using -column_thresh, or something else, correct? I saw that on
the
> > Online
> > > Tutorial page. But, I guess my question about those things
would be,
> is
> > > there anything I need to make sure that is in my adeck/bdeck
files that
> > is
> > > not included now, that I should include to make those tasks
easier? I
> > just
> > > want to make sure that I've included everything I need.
> > >
> > > Roz
> > >
> > > On Tue, Nov 27, 2018 at 11:43 AM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >> Roz,
> > >>
> > >> Thanks for posting your sample data. I found the problem in
your
> > >> TCPairsConfig file. Change...
> > >> FROM: storm_id = [ "" ];
> > >> TO: storm_id = [ ];
> > >>
> > >> The storm_id entry is an array of storm ID's to match. An
empty array
> > is
> > >> just square brackets with nothing inside ([ ]). Since you had
> > >> double-quotes inside, that's an array of length 1 containing an
empty
> > >> string.
> > >>
> > >> Thanks,
> > >> John
> > >>
> > >>
> > >> On Tue, Nov 27, 2018 at 4:50 AM Rosalyn MacCracken - NOAA
Affiliate
> via
> > >> RT <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >> >
> > >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
>
> > >> >
> > >> > Hi John,
> > >> >
> > >> > I uploaded the original files (Adeck = aml*, Bdeck=bml*,
without the
> > >> > *modified.dat ending), the modified files (with the
*modified.dat
> > >> ending)
> > >> > and the config file into my ftp directory (*cd
> > >> > incoming/irap/met_help/maccracken). *Let me know what you
find.
> > >> >
> > >> > Thanks for your help!
> > >> >
> > >> > Roz
> > >> >
> > >> > On Mon, Nov 26, 2018 at 5:22 PM John Halley Gotway via RT <
> > >> > met_help at ucar.edu>
> > >> > wrote:
> > >> >
> > >> > > Roz,
> > >> > >
> > >> > > Yes, the first 8 columns are required. The additional ones
are
> > >> optional.
> > >> > >
> > >> > > Looking at your log output, I see that tc_pairs is skipping
over
> all
> > >> 642
> > >> > > input lines. And the question is why? If you send me your
latest
> > >> ADeck,
> > >> > > BDeck, and TC-Pairs config files, I could run it here and
try to
> > >> figure
> > >> > out
> > >> > > why it isn't defining any ADECK tracks.
> > >> > >
> > >> > > Thanks,
> > >> > > John
> > >> > >
> > >> > > On Mon, Nov 26, 2018 at 12:23 PM Rosalyn MacCracken - NOAA
> Affiliate
> > >> via
> > >> > RT
> > >> > > <met_help at ucar.edu> wrote:
> > >> > >
> > >> > > >
> > >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
> > >> > > >
> > >> > > > So, can I just have 8 columns for the input file, or, do
I need
> to
> > >> make
> > >> > > > columns of -9999 or NaN for all the rest of the columns?
> > >> > > >
> > >> > > > I experimented, and just wrote out 10 columns for the
input
> file:
> > >> > > > agfso_trackdf_mod =
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
agfso_trackdf[['basin_name','storm_id','model_start_date','const','model','fcst_lead_time','vort_lat_cent','vort_lon_cent','vort_10m_max',
> > >> > > > 'vort_min_mslp','type','thresh_wspd']]
> > >> > > >
> > >> > > > because I wanted stats for these variables, and I got the
output
> > >> below.
> > >> > > > So, maybe it's working, but, the config file needs to be
> > different?
> > >> > Any
> > >> > > > other ideas of what I should try doing/looking at?
> > >> > > >
> > >> > > > Roz
> > >> > > >
> > >> > > > Output:
> > >> > > > DEBUG 1: [Source 1 of 1] ADECK Source:
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018083118.gfso.0053_modified.dat,
> > >> > > > Model Suffix: _EXP
> > >> > > > DEBUG 1: [Source 1 of 1] BDECK Source:
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018083118.gfso.0053_modified.dat,
> > >> > > > Model Suffix: (nul)
> > >> > > > DEBUG 1: Config File Default:
> > >> > > > /opt/MET7/share/met/config/TCPairsConfig_default
> > >> > > > DEBUG 1: Config File User:
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > >> > > > DEBUG 1: Distance to land file:
/opt/MET7/share/met/tc_data/
> > >> > > > dland_global_tenth_degree.nc
> > >> > > > DEBUG 4:
> > >> > > > DEBUG 4: Latitude/Longitude Grid Data:
> > >> > > > DEBUG 4: lat_ll: -90
> > >> > > > DEBUG 4: lon_ll: 180
> > >> > > > DEBUG 4: delta_lat: 0.1
> > >> > > > DEBUG 4: delta_lon: 0.1
> > >> > > > DEBUG 4: Nlat: 1801
> > >> > > > DEBUG 4: Nlon: 3601
> > >> > > > DEBUG 4:
> > >> > > > DEBUG 2: Processing 1 BDECK file(s).
> > >> > > > DEBUG 4: [File 1 of 1] Used 30 of 30 lines read from file
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018083118.gfso.0053_modified.dat"
> > >> > > > DEBUG 3: Used 30 of 30 lines read from 1 file(s).
> > >> > > > DEBUG 3: Identified 1 track(s).
> > >> > > > DEBUG 4: [Track 1 of 1] TrackInfo: StormId =
"ML0800532018",
> > IsBest
> > >> =
> > >> > > > false, IsOper = false, CheckAnly = true, IsAnly = true,
Basin =
> > >> "ML",
> > >> > > > Cyclone = "080053", StormName = "", TechniqueNumber = 3,
> > Technique =
> > >> > > > "GFSO", Initials = "", InitTime = NA, MinValidTime =
> > >> 20180805_060000,
> > >> > > > MaxValidTime = 20180812_120000, NPoints = 30, NAlloc =
100,
> > >> NTrackLines
> > >> > > = 0
> > >> > > > DEBUG 2: Found 1 BDECK track(s).
> > >> > > > DEBUG 2: Processing 1 ADECK file(s).
> > >> > > > DEBUG 4: [File 1 of 1] Used 0 of 642 lines read from file
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018083118.gfso.0053_modified.dat"
> > >> > > > DEBUG 3: Used 0 of 642 lines read from 1 file(s).
> > >> > > > DEBUG 3: Identified 0 track(s).
> > >> > > > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> > >> > > > DEBUG 2: Finished adding 0 and replacing 0 Interp12
track(s).
> > >> > > > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > >> > > > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > >> > > > DEBUG 2: Deriving 0 ADECK lag model(s).
> > >> > > > DEBUG 2: Added 0 ADECK lag tracks(s).
> > >> > > > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> > >> > > > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> > >> > > > DEBUG 2: Filtering 0 ADECK tracks based on config file
settings.
> > >> > > > DEBUG 3: Total tracks read = 0
> > >> > > > DEBUG 3: Total tracks kept = 0
> > >> > > > DEBUG 3: Rejected for storm name = 0
> > >> > > > DEBUG 3: Rejected for valid time = 0
> > >> > > > DEBUG 3: Rejected for required lead times = 0
> > >> > > > DEBUG 3: Rejected for init mask = 0
> > >> > > > DEBUG 3: Rejected for valid mask = 0
> > >> > > > DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
> > >> > > > DEBUG 1: Watch/Warning file:
> > >> /opt/MET7/share/met/tc_data/wwpts_us.txt
> > >> > > > DEBUG 1: Output file:
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018083118.gfso.0053_test_modified_output.txt.tcst
> > >> > > >
> > >> > > >
> > >> > > > On Mon, Nov 26, 2018 at 2:06 PM John Halley Gotway via RT
<
> > >> > > > met_help at ucar.edu>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Roz,
> > >> > > > >
> > >> > > > > Page 334 of the user's guide (
> > >> > > > >
> > >> > >
> > >>
> >
https://dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v8.0.pdf
> > >> > > > )
> > >> > > > > mentions that only the first 8 ATCF columns are
required. If
> > >> > > additional
> > >> > > > > columns are present in the input ATCF files, then
tc_pairs
> will
> > >> > > interpret
> > >> > > > > them as specified by the ATCF format.
> > >> > > > >
> > >> > > > > So if the storm name column is present, MET in
interpret those
> > >> values
> > >> > > as
> > >> > > > > the storm name.
> > >> > > > >
> > >> > > > > For Guang Ping's data, probably best to have that
column
> empty.
> > >> > > > >
> > >> > > > > John
> > >> > > > >
> > >> > > > >
> > >> > > > > On Mon, Nov 26, 2018 at 11:51 AM Rosalyn MacCracken -
NOAA
> > >> Affiliate
> > >> > > via
> > >> > > > RT
> > >> > > > > <met_help at ucar.edu> wrote:
> > >> > > > >
> > >> > > > > >
> > >> > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
> > >
> > >> > > > > >
> > >> > > > > > Hi John,
> > >> > > > > >
> > >> > > > > > Ok, I will ignore the suffix error. I do have
numeric
> values
> > in
> > >> > > > > > STORMNAME. I wonder if in the modified files I have,
this
> > >> column
> > >> > is
> > >> > > > not
> > >> > > > > > actually STORMNAME, but, some other variable. Now, I
don't
> > have
> > >> > the
> > >> > > > > number
> > >> > > > > > of columns that the NRL website has. Will that be a
> problem?
> > >> > > Should I
> > >> > > > > > create dummy columns?
> > >> > > > > >
> > >> > > > > > Roz
> > >> > > > > >
> > >> > > > > > On Mon, Nov 26, 2018 at 1:14 PM John Halley Gotway
via RT <
> > >> > > > > > met_help at ucar.edu>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Roz,
> > >> > > > > > >
> > >> > > > > > > I see that you're wondering about this log message:
> > >> > > > > > > *Model Suffix: (nul)*
> > >> > > > > > >
> > >> > > > > > > Everything is fine, you do not need to be concerned
about
> > >> this.
> > >> > > > > Here's a
> > >> > > > > > > line taken from the usage statement for the
tc_pairs tool:
> > >> > > > > > > *Note: The "-adeck", "-edeck", and "-bdeck" options
may
> > >> include
> > >> > > > > > > "suffix=string" to modify the model names from that
> source.*
> > >> > > > > > >
> > >> > > > > > > Since you didn't specify "suffix=string" on the
command
> > line,
> > >> the
> > >> > > > > suffix
> > >> > > > > > > remains the default value of an empty string, i.e.
> "(nul)".
> > >> So
> > >> > why
> > >> > > > > would
> > >> > > > > > > you ever want to use a suffix? Sometimes data is
messy.
> We
> > >> were
> > >> > > > > running
> > >> > > > > > > tc_pairs to evaluate track data from multiple
sources and
> we
> > >> > found
> > >> > > > that
> > >> > > > > > > some model names we reused in both sources. Let's
say the
> > >> OFCL
> > >> > > > > forecast
> > >> > > > > > > showed up in both stream 1.0 and stream 1.5 data.
> > Specifying
> > >> a
> > >> > > > suffix
> > >> > > > > > such
> > >> > > > > > > as "suffix=_STREAM1.5" changes the model name from
"OFCL"
> to
> > >> > > > > > > "OFCL_STREAM1.5". And that enables us to
distinguish
> > between
> > >> > those
> > >> > > > > > tracks
> > >> > > > > > > in the output.
> > >> > > > > > >
> > >> > > > > > > But uou probably don't have any need for this
option in
> your
> > >> > > current
> > >> > > > > > work.
> > >> > > > > > >
> > >> > > > > > > However, I am concerned about these log messages:
> > >> > > > > > > DEBUG 4: Updating storm name from "-287" to "-305"
for
> > >> > > ML0900022018.
> > >> > > > > > > DEBUG 4: Updating storm name from "-305" to "-255"
for
> > >> > > ML0900022018.
> > >> > > > > > > DEBUG 4: Updating storm name from "-255" to "-203"
for
> > >> > > ML0900022018.
> > >> > > > > > > DEBUG 4: Updating storm name from "-203" to "-205"
for
> > >> > > ML0900022018.
> > >> > > > > > > DEBUG 4: Updating storm name from "-205" to "-217"
for
> > >> > > ML0900022018.
> > >> > > > > > > DEBUG 4: Updating storm name from "-217" to "-211"
for
> > >> > > ML0900022018.
> > >> > > > > > > DEBUG 4: Updating storm name from "-211" to "-182"
for
> > >> > > ML0900022018.
> > >> > > > > > > DEBUG 4: Updating storm name from "-182" to "-138"
for
> > >> > > ML0900022018.
> > >> > > > > > >
> > >> > > > > > > You likely have numeric values showing up in the
> "STORMNAME"
> > >> ATCF
> > >> > > > > column:
> > >> > > > > > >
> > >> > >
https://www.nrlmry.navy.mil/atcf_web/docs/database/new/abdeck.txt
> > >> > > > > > >
> > >> > > > > > > I'd recommend looking into that.
> > >> > > > > > >
> > >> > > > > > > Thanks,
> > >> > > > > > > John
> > >> > > > > > >
> > >> > > > > > > On Mon, Nov 26, 2018 at 8:44 AM Rosalyn MacCracken
- NOAA
> > >> > Affiliate
> > >> > > > via
> > >> > > > > > RT
> > >> > > > > > > <
> > >> > > > > > > met_help at ucar.edu> wrote:
> > >> > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > <URL:
> > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
> > >> > >
> > >> > > > > > > >
> > >> > > > > > > > Hi John,
> > >> > > > > > > >
> > >> > > > > > > > I figured out an easy way to read in and write
out Guang
> > >> Ping's
> > >> > > > > > modified
> > >> > > > > > > > ATCF file, so that it:
> > >> > > > > > > > 1) Removes the 3rd column
> > >> > > > > > > > 2) Replaces -99 with -9999 in column 21 and 24
(after
> the
> > >> 3rd
> > >> > > > column
> > >> > > > > > is
> > >> > > > > > > > removed)
> > >> > > > > > > > 3) Updates column 2 to include month+cyclone
number.
> > (You
> > >> had
> > >> > > > said
> > >> > > > > to
> > >> > > > > > > do
> > >> > > > > > > > year+cyclone #, but the tc_pairs_wrapper.py said
to do
> > >> > > > > month+cyclone#)
> > >> > > > > > > >
> > >> > > > > > > > I changed Adeck and Bdeck.
> > >> > > > > > > > Just a couple lines of my file:
> > >> > > > > > > >
> > >> > > > > > > > ML, 090002, 2018090100, 03, GFSO, 000, 511S,
897W, 40,
> > >> 976,
> > >> > > XX,
> > >> > > > > 34,
> > >> > > > > > > > NEQ, 0625, 0630, 0000, 0226, 978, 68, -9999,
123,
> 155,
> > >> > -9999,
> > >> > > > > -9999,
> > >> > > > > > > > -9999, -297, -394, -287, -458, 0073
> > >> > > > > > > > ML, 090002, 2018090106, 03, GFSO, 000, 537S,
854W, 44,
> > >> 964,
> > >> > > XX,
> > >> > > > > 34,
> > >> > > > > > > > NEQ, 0640, 0645, 0000, 0297, 969, 108, -9999,
119,
> 147,
> > >> > -9999,
> > >> > > > > -9999,
> > >> > > > > > > > -9999, -403, -648, -305, -394, 0055
> > >> > > > > > > >
> > >> > > > > > > > So, I've run tc_pairs again and my output is
below. I
> > don't
> > >> > > > > understand
> > >> > > > > > > the
> > >> > > > > > > > Model Suffix: (nul). Is the input format still
wrong?
> > >> > > > > > > >
> > >> > > > > > > > Roz
> > >> > > > > > > >
> > >> > > > > > > > Output:
> > >> > > > > > > > /opt/MET7/bin/tc_pairs -adeck
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat
> > >> > > > > > > > -bdeck
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat
> > >> > > > > > > > -config
> > >> > > > > >
> > ~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > >> > > > > > > > -out
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt
> > >> > > > > > > > -log
> > >> > > > > > >
> > >> > >
> >
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> > >> > > > > > > > -v 4
> > >> > > > > > > > DEBUG 1: [Source 1 of 1] ADECK Source:
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat,
> > >> > > > > > > > Model Suffix: (nul)
> > >> > > > > > > > DEBUG 1: [Source 1 of 1] BDECK Source:
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat,
> > >> > > > > > > > Model Suffix: (nul)
> > >> > > > > > > > DEBUG 1: Config File Default:
> > >> > > > > > > > /opt/MET7/share/met/config/TCPairsConfig_default
> > >> > > > > > > > DEBUG 1: Config File User:
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > >> > > > > > > > DEBUG 1: Distance to land file:
> > /opt/MET7/share/met/tc_data/
> > >> > > > > > > > dland_global_tenth_degree.nc
> > >> > > > > > > > DEBUG 4:
> > >> > > > > > > > DEBUG 4: Latitude/Longitude Grid Data:
> > >> > > > > > > > DEBUG 4: lat_ll: -90
> > >> > > > > > > > DEBUG 4: lon_ll: 180
> > >> > > > > > > > DEBUG 4: delta_lat: 0.1
> > >> > > > > > > > DEBUG 4: delta_lon: 0.1
> > >> > > > > > > > DEBUG 4: Nlat: 1801
> > >> > > > > > > > DEBUG 4: Nlon: 3601
> > >> > > > > > > > DEBUG 4:
> > >> > > > > > > > DEBUG 2: Processing 1 BDECK file(s).
> > >> > > > > > > > DEBUG 4: Updating storm name from "-287" to "-
305" for
> > >> > > > ML0900022018.
> > >> > > > > > > > DEBUG 4: Updating storm name from "-305" to "-
255" for
> > >> > > > ML0900022018.
> > >> > > > > > > > DEBUG 4: Updating storm name from "-255" to "-
203" for
> > >> > > > ML0900022018.
> > >> > > > > > > > DEBUG 4: Updating storm name from "-203" to "-
205" for
> > >> > > > ML0900022018.
> > >> > > > > > > > DEBUG 4: Updating storm name from "-205" to "-
217" for
> > >> > > > ML0900022018.
> > >> > > > > > > > DEBUG 4: Updating storm name from "-217" to "-
211" for
> > >> > > > ML0900022018.
> > >> > > > > > > > DEBUG 4: Updating storm name from "-211" to "-
182" for
> > >> > > > ML0900022018.
> > >> > > > > > > > DEBUG 4: Updating storm name from "-182" to "-
138" for
> > >> > > > ML0900022018.
> > >> > > > > > > > DEBUG 4: [File 1 of 1] Used 9 of 9 lines read
from file
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat"
> > >> > > > > > > > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
> > >> > > > > > > > DEBUG 3: Identified 1 track(s).
> > >> > > > > > > > DEBUG 4: [Track 1 of 1] TrackInfo: StormId =
> > "ML0900022018",
> > >> > > > IsBest =
> > >> > > > > > > > false, IsOper = false, CheckAnly = true, IsAnly =
true,
> > >> Basin =
> > >> > > > "ML",
> > >> > > > > > > > Cyclone = "090002", StormName = "-138",
TechniqueNumber
> =
> > 3,
> > >> > > > > Technique
> > >> > > > > > =
> > >> > > > > > > > "GFSO", Initials = "-9999", InitTime = NA,
MinValidTime
> =
> > >> > > > > > > 20180901_000000,
> > >> > > > > > > > MaxValidTime = 20180903_000000, NPoints = 9,
NAlloc =
> 100,
> > >> > > > > NTrackLines
> > >> > > > > > =
> > >> > > > > > > 0
> > >> > > > > > > > DEBUG 2: Found 1 BDECK track(s).
> > >> > > > > > > > DEBUG 2: Processing 1 ADECK file(s).
> > >> > > > > > > > DEBUG 4: [File 1 of 1] Used 0 of 21 lines read
from file
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat"
> > >> > > > > > > > DEBUG 3: Used 0 of 21 lines read from 1 file(s).
> > >> > > > > > > > DEBUG 3: Identified 0 track(s).
> > >> > > > > > > > DEBUG 2: Deriving 12-hour interpolated ADECK
tracks.
> > >> > > > > > > > DEBUG 2: Finished adding 0 and replacing 0
Interp12
> > >> track(s).
> > >> > > > > > > > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > >> > > > > > > > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > >> > > > > > > > DEBUG 2: Deriving 0 ADECK lag model(s).
> > >> > > > > > > > DEBUG 2: Added 0 ADECK lag tracks(s).
> > >> > > > > > > > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline
model(s).
> > >> > > > > > > > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> > >> > > > > > > > DEBUG 2: Filtering 0 ADECK tracks based on config
file
> > >> > settings.
> > >> > > > > > > > DEBUG 3: Total tracks read = 0
> > >> > > > > > > > DEBUG 3: Total tracks kept = 0
> > >> > > > > > > > DEBUG 3: Rejected for storm name = 0
> > >> > > > > > > > DEBUG 3: Rejected for valid time = 0
> > >> > > > > > > > DEBUG 3: Rejected for required lead times = 0
> > >> > > > > > > > DEBUG 3: Rejected for init mask = 0
> > >> > > > > > > > DEBUG 3: Rejected for valid mask = 0
> > >> > > > > > > > DEBUG 2: Matching 0 ADECK tracks to 1 BDECK
tracks.
> > >> > > > > > > > DEBUG 1: Watch/Warning file:
> > >> > > > /opt/MET7/share/met/tc_data/wwpts_us.txt
> > >> > > > > > > > DEBUG 1: Output file:
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt.tcst
> > >> > > > > > > >
> > >> > > > > > > > On Wed, Nov 21, 2018 at 9:16 AM Rosalyn
MacCracken -
> NOAA
> > >> > > > Affiliate <
> > >> > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > >> > > > > > > >
> > >> > > > > > > > > Hi John,
> > >> > > > > > > > >
> > >> > > > > > > > > Yes, that makes sense, and yes, I am using
Guang
> Pings'
> > >> > > modified
> > >> > > > > > files.
> > >> > > > > > > > > I'm not sure how to run tc_pairs_wrapper.py,
though.
> I
> > >> > think I
> > >> > > > > need
> > >> > > > > > > some
> > >> > > > > > > > > type of *.conf file to pass in the files I
need, or
> > >> something
> > >> > > > like
> > >> > > > > > > that.
> > >> > > > > > > > > I'm in the process of reading through the code
now. I
> > >> wonder
> > >> > > if
> > >> > > > I
> > >> > > > > > > could
> > >> > > > > > > > > just pull out the function process_non_atcf, to
run
> in a
> > >> > > > > stand-alone
> > >> > > > > > > > mode.
> > >> > > > > > > > > I wish there was some sort of documentation
with this
> > >> code...
> > >> > > > > > > > >
> > >> > > > > > > > > And, so, once I can read these files, I can use
> tc_pairs
> > >> to
> > >> > > > create
> > >> > > > > > the
> > >> > > > > > > > > forecast error statistics, and then use
tc_stats to
> > >> aggregate
> > >> > > the
> > >> > > > > > files
> > >> > > > > > > > and
> > >> > > > > > > > > get the mean, etc, for a period of time,
correct? Was
> > >> there
> > >> > > some
> > >> > > > > > sort
> > >> > > > > > > of
> > >> > > > > > > > > plotting scripts for the output files?
> > >> > > > > > > > >
> > >> > > > > > > > > So, with tc_pairs, I can define a region, like
I did
> > with
> > >> > > > > point_stat,
> > >> > > > > > > > > correct? Can I use the same *.nc files, or do
I use
> the
> > >> > > original
> > >> > > > > > > *.poly
> > >> > > > > > > > > files?
> > >> > > > > > > > >
> > >> > > > > > > > > Roz
> > >> > > > > > > > >
> > >> > > > > > > > > On Tue, Nov 20, 2018 at 7:53 PM John Halley
Gotway via
> > RT
> > >> <
> > >> > > > > > > > > met_help at ucar.edu> wrote:
> > >> > > > > > > > >
> > >> > > > > > > > >> Hi Roz,
> > >> > > > > > > > >>
> > >> > > > > > > > >> Yes, I see you’re using the modified ATCF
format that
> > >> Guang
> > >> > > Ping
> > >> > > > > > uses.
> > >> > > > > > > > >> You’re correct that tc_pairs doesn’t parse
this
> > directly
> > >> > since
> > >> > > > it
> > >> > > > > > > > differs
> > >> > > > > > > > >> slightly from the ATCF format.
> > >> > > > > > > > >>
> > >> > > > > > > > >> When we’ve processed this data in the past, we
first
> > >> > > > pre-processed
> > >> > > > > > it
> > >> > > > > > > > in a
> > >> > > > > > > > >> shell script to:
> > >> > > > > > > > >>
> > >> > > > > > > > >> 1. Replace the -99 value with -9999 for
missing data.
> > >> > > > > > > > >> 2. Remove the 3rd column
> > >> > > > > > > > >> 3. Updating the 2nd column to include the year
and
> > >> cyclone
> > >> > > > number.
> > >> > > > > > > > >>
> > >> > > > > > > > >> And that enable tc_pairs to read it.
> > >> > > > > > > > >>
> > >> > > > > > > > >> This is actually Done in python by one of the
use
> cases
> > >> in
> > >> > > > > METplus.
> > >> > > > > > > > >>
> > >> > > > > > > > >> Take a look at this file:
> > >> > > > > > > > >>
> > >> > > > >
> > >>
https://github.com/NCAR/METplus/blob/master/ush/tc_pairs_wrapper.py
> > >> > > > > > > > >>
> > >> > > > > > > > >> At the function named:
> > >> > > > > > > > >> process_non_atcf
> > >> > > > > > > > >>
> > >> > > > > > > > >> I haven’t run this using METplus scripts
myself, but
> > all
> > >> the
> > >> > > > > pieces
> > >> > > > > > of
> > >> > > > > > > > >> logic are there, and it should be possible.
> > >> > > > > > > > >>
> > >> > > > > > > > >> Make sense?
> > >> > > > > > > > >>
> > >> > > > > > > > >> Thanks
> > >> > > > > > > > >> John
> > >> > > > > > > > >>
> > >> > > > > > > > >> On Tue, Nov 20, 2018 at 1:13 PM Rosalyn
MacCracken -
> > NOAA
> > >> > > > > Affiliate
> > >> > > > > > > via
> > >> > > > > > > > >> RT <
> > >> > > > > > > > >> met_help at ucar.edu> wrote:
> > >> > > > > > > > >>
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Tue Nov 20 13:13:05 2018: Request 87885 was
acted
> > upon.
> > >> > > > > > > > >> > Transaction: Ticket created by
> > >> > rosalyn.maccracken at noaa.gov
> > >> > > > > > > > >> > Queue: met_help
> > >> > > > > > > > >> > Subject: Help with using TC_pairs
> > >> > > > > > > > >> > Owner: Nobody
> > >> > > > > > > > >> > Requestors: rosalyn.maccracken at noaa.gov
> > >> > > > > > > > >> > Status: new
> > >> > > > > > > > >> > Ticket <URL:
> > >> > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
> > >> > > > > > > > >
> > >> > > > > > > > >> >
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Hi,
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > I'm trying to use TC_pairs for extra-
tropical
> > cyclones
> > >> > > > > (basin=ML).
> > >> > > > > > > I
> > >> > > > > > > > >> think
> > >> > > > > > > > >> > the format of the ATCF data is the issue,
but, I'm
> > not
> > >> > sure.
> > >> > > > I
> > >> > > > > > have
> > >> > > > > > > > an
> > >> > > > > > > > >> > Adeck and a Bdeck file, where a sample of
one of
> the
> > >> lines
> > >> > > is:
> > >> > > > > > > > >> > ML, 0003, 2018090100_F000_687N_0159W_FOF,
> 2018090100,
> > >> 03,
> > >> > > > GFSO,
> > >> > > > > > 000,
> > >> > > > > > > > >> 687N,
> > >> > > > > > > > >> > 159W, 32, 987, XX, 34, NEQ, 0301, 0338,
0000,
> > 0000,
> > >> > 988,
> > >> > > > > 52,
> > >> > > > > > > > -99,
> > >> > > > > > > > >> > 330, 86, -99, -9999, -9999, 175, 202,
137,
> 135,
> > >> 0081
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > This is slightly different from the NRL ATCF
format
> > on
> > >> > their
> > >> > > > > > > website,
> > >> > > > > > > > >> such
> > >> > > > > > > > >> > as the second column. Anyway, I configured
the
> > config
> > >> > file
> > >> > > > very
> > >> > > > > > > > simply,
> > >> > > > > > > > >> > (model=GFS, basin=ML), and ran tc_pairs, and
got
> the
> > >> > output
> > >> > > to
> > >> > > > > the
> > >> > > > > > > > >> screen
> > >> > > > > > > > >> > below.
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Is there a way to work around the diferent
ATCF
> > format?
> > >> > > I've
> > >> > > > > > > > attached a
> > >> > > > > > > > >> > sample of my Adeck and Bdeck files.
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Thanks,
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Roz
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Output to the screen:
> > >> > > > > > > > >> > /opt/MET7/bin/tc_pairs -adeck
> > >> > > > > > > > >> >
> > >> > > > > > > > >>
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
~/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003
> > >> > > > > > > > >> > -bdeck
> > >> > > > > > > > >> >
> > >> > > > > > > > >>
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
~/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003
> > >> > > > > > > > >> > -config
> > >> > > > > > > > >> >
> > >> > > > > > > >
> > >> > > > >
> > >> >
> >
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> > >> > > > > > > > >> > -out
> > >> > > > > > > > >> >
> > >> > > > > > > > >> >
> > >> > > > > > > > >>
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt
> > >> > > > > > > > >> > -log
> > >> > > > > > > > >>
> > >> > > > > >
> > >> >
>
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> > >> > > > > > > > >> > -v 3
> > >> > > > > > > > >> > DEBUG 1: [Source 1 of 1] ADECK Source:
> > >> > > > > > > > >> >
> > >> > > > > > > > >> >
> > >> > > > > > > > >>
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003,
> > >> > > > > > > > >> > Model Suffix: (nul)
> > >> > > > > > > > >> > DEBUG 1: [Source 1 of 1] BDECK Source:
> > >> > > > > > > > >> >
> > >> > > > > > > > >> >
> > >> > > > > > > > >>
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003,
> > >> > > > > > > > >> > Model Suffix: (nul)
> > >> > > > > > > > >> > DEBUG 1: Config File Default:
> > >> > > > > > > > >> >
/opt/MET7/share/met/config/TCPairsConfig_default
> > >> > > > > > > > >> > DEBUG 1: Config File User:
> > >> > > > > > > > >> >
> > >> > > > > > > > >> >
> > >> > > > > > > > >>
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> > >> > > > > > > > >> > DEBUG 1: Distance to land file:
> > >> > /opt/MET7/share/met/tc_data/
> > >> > > > > > > > >> > dland_global_tenth_degree.nc
> > >> > > > > > > > >> > DEBUG 2: Processing 1 BDECK file(s).
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
> latitude
> > >> ...
> > >> > > "000"
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
> longitude
> > >> ...
> > >> > > > "687N"
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
> latitude
> > >> ...
> > >> > > "000"
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
> longitude
> > >> ...
> > >> > > > "701N"
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
> latitude
> > >> ...
> > >> > > "000"
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
> longitude
> > >> ...
> > >> > > > "711N"
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
> latitude
> > >> ...
> > >> > > "000"
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
> longitude
> > >> ...
> > >> > > > "720N"
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
> latitude
> > >> ...
> > >> > > "000"
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
> longitude
> > >> ...
> > >> > > > "730N"
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
> latitude
> > >> ...
> > >> > > "000"
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
> longitude
> > >> ...
> > >> > > > "741N"
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
> latitude
> > >> ...
> > >> > > "000"
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
> longitude
> > >> ...
> > >> > > > "751N"
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
> latitude
> > >> ...
> > >> > > "000"
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
> longitude
> > >> ...
> > >> > > > "760N"
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
> latitude
> > >> ...
> > >> > > "000"
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
> longitude
> > >> ...
> > >> > > > "764N"
> > >> > > > > > > > >> > WARNING:
> > >> > > > > > > > >> > DEBUG 3: Used 9 of 9 lines read from 1
file(s).
> > >> > > > > > > > >> > DEBUG 3: Identified 9 track(s).
> > >> > > > > > > > >> > DEBUG 2: Found 9 BDECK track(s).
> > >> > > > > > > > >> > DEBUG 2: Processing 1 ADECK file(s).
> > >> > > > > > > > >> > DEBUG 3: Used 0 of 22 lines read from 1
file(s).
> > >> > > > > > > > >> > DEBUG 3: Identified 0 track(s).
> > >> > > > > > > > >> > DEBUG 2: Deriving 12-hour interpolated ADECK
> tracks.
> > >> > > > > > > > >> > DEBUG 2: Finished adding 0 and replacing 0
Interp12
> > >> > > track(s).
> > >> > > > > > > > >> > DEBUG 2: Deriving 0 ADECK consensus
model(s).
> > >> > > > > > > > >> > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > >> > > > > > > > >> > DEBUG 2: Deriving 0 ADECK lag model(s).
> > >> > > > > > > > >> > DEBUG 2: Added 0 ADECK lag tracks(s).
> > >> > > > > > > > >> > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline
> model(s).
> > >> > > > > > > > >> > DEBUG 2: Added 0 CLIPER/SHIFOR baseline
track(s).
> > >> > > > > > > > >> > DEBUG 2: Filtering 0 ADECK tracks based on
config
> > file
> > >> > > > settings.
> > >> > > > > > > > >> > DEBUG 3: Total tracks read =
0
> > >> > > > > > > > >> > DEBUG 3: Total tracks kept =
0
> > >> > > > > > > > >> > DEBUG 3: Rejected for storm name =
0
> > >> > > > > > > > >> > DEBUG 3: Rejected for valid time =
0
> > >> > > > > > > > >> > DEBUG 3: Rejected for required lead times =
0
> > >> > > > > > > > >> > DEBUG 3: Rejected for init mask =
0
> > >> > > > > > > > >> > DEBUG 3: Rejected for valid mask =
0
> > >> > > > > > > > >> > DEBUG 2: Matching 0 ADECK tracks to 9 BDECK
tracks.
> > >> > > > > > > > >> > DEBUG 1: Watch/Warning file:
> > >> > > > > > > /opt/MET7/share/met/tc_data/wwpts_us.txt
> > >> > > > > > > > >> > DEBUG 1: Output file:
> > >> > > > > > > > >> >
> > >> > > > > > > > >> >
> > >> > > > > > > > >>
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt.tcst
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > --
> > >> > > > > > > > >> > Rosalyn MacCracken
> > >> > > > > > > > >> > Support Scientist
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Ocean Applications Branch
> > >> > > > > > > > >> > NOAA/NWS Ocean Prediction Center
> > >> > > > > > > > >> > NCWCP
> > >> > > > > > > > >> > 5830 University Research Ct
> > >> > > > > > > > >> > College Park, MD 20740-3818
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > (p) 301-683-1551
> > >> > > > > > > > >> > rosalyn.maccracken at noaa.gov
> > >> > > > > > > > >> >
> > >> > > > > > > > >> >
> > >> > > > > > > > >>
> > >> > > > > > > > >>
> > >> > > > > > > > >
> > >> > > > > > > > > --
> > >> > > > > > > > > Rosalyn MacCracken
> > >> > > > > > > > > Support Scientist
> > >> > > > > > > > >
> > >> > > > > > > > > Ocean Applications Branch
> > >> > > > > > > > > NOAA/NWS Ocean Prediction Center
> > >> > > > > > > > > NCWCP
> > >> > > > > > > > > 5830 University Research Ct
> > >> > > > > > > > > College Park, MD 20740-3818
> > >> > > > > > > > >
> > >> > > > > > > > > (p) 301-683-1551
> > >> > > > > > > > > rosalyn.maccracken at noaa.gov
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > --
> > >> > > > > > > > Rosalyn MacCracken
> > >> > > > > > > > Support Scientist
> > >> > > > > > > >
> > >> > > > > > > > Ocean Applications Branch
> > >> > > > > > > > NOAA/NWS Ocean Prediction Center
> > >> > > > > > > > NCWCP
> > >> > > > > > > > 5830 University Research Ct
> > >> > > > > > > > College Park, MD 20740-3818
> > >> > > > > > > >
> > >> > > > > > > > (p) 301-683-1551
> > >> > > > > > > > rosalyn.maccracken at noaa.gov
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > > > --
> > >> > > > > > Rosalyn MacCracken
> > >> > > > > > Support Scientist
> > >> > > > > >
> > >> > > > > > Ocean Applications Branch
> > >> > > > > > NOAA/NWS Ocean Prediction Center
> > >> > > > > > NCWCP
> > >> > > > > > 5830 University Research Ct
> > >> > > > > > College Park, MD 20740-3818
> > >> > > > > >
> > >> > > > > > (p) 301-683-1551
> > >> > > > > > rosalyn.maccracken at noaa.gov
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > > > --
> > >> > > > Rosalyn MacCracken
> > >> > > > Support Scientist
> > >> > > >
> > >> > > > Ocean Applications Branch
> > >> > > > NOAA/NWS Ocean Prediction Center
> > >> > > > NCWCP
> > >> > > > 5830 University Research Ct
> > >> > > > College Park, MD 20740-3818
> > >> > > >
> > >> > > > (p) 301-683-1551
> > >> > > > rosalyn.maccracken at noaa.gov
> > >> > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> >
> > >> > --
> > >> > Rosalyn MacCracken
> > >> > Support Scientist
> > >> >
> > >> > Ocean Applications Branch
> > >> > NOAA/NWS Ocean Prediction Center
> > >> > NCWCP
> > >> > 5830 University Research Ct
> > >> > College Park, MD 20740-3818
> > >> >
> > >> > (p) 301-683-1551
> > >> > rosalyn.maccracken at noaa.gov
> > >> >
> > >> >
> > >>
> > >>
> > >
> > > --
> > > Rosalyn MacCracken
> > > Support Scientist
> > >
> > > Ocean Applications Branch
> > > NOAA/NWS Ocean Prediction Center
> > > NCWCP
> > > 5830 University Research Ct
> > > College Park, MD 20740-3818
> > >
> > > (p) 301-683-1551
> > > rosalyn.maccracken at noaa.gov
> > >
> >
> >
> > --
> > Rosalyn MacCracken
> > Support Scientist
> >
> > Ocean Applications Branch
> > NOAA/NWS Ocean Prediction Center
> > NCWCP
> > 5830 University Research Ct
> > College Park, MD 20740-3818
> >
> > (p) 301-683-1551
> > rosalyn.maccracken at noaa.gov
> >
> >
>
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Help with using TC_pairs
From: Rosalyn MacCracken - NOAA Affiliate
Time: Thu Dec 06 10:10:23 2018
Oh, no, it wouldn't be the same thing. Doing cat on the linux command
line
would keep all the column headers, and the filter/dump row would only
have
one column header, right?
Roz
On Thu, Dec 6, 2018 at 12:07 PM Rosalyn MacCracken - NOAA Affiliate <
rosalyn.maccracken at noaa.gov> wrote:
> Hi John,
>
> Oh, thanks! Is this better than just doing a "cat * > merged_file"
in
> linux, or does it essentially do the same thing?
>
> Roz
>
> On Thu, Dec 6, 2018 at 12:05 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Roz,
>>
>> Sure, you can do that with tc_stat. Just use tc_stat to run a "-
filter"
>> job without actually applying any filtering criteria. And then use
the
>> "-dump_row" option to write the filtered output to a file.
>>
>> That will effectively concatenate multiple files into a single
output
>> file.
>>
>> John
>>
>> On Thu, Dec 6, 2018 at 9:46 AM Rosalyn MacCracken - NOAA Affiliate
via RT
>> <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
>> >
>> > Hi John,
>> >
>> > I got tc_stat to work the way I wanted to, and was able to
produce
>> summary
>> > statistics. Now that I have matched pairs, I was thinking about
doing
>> some
>> > other things, unrelated to MET (like data mining, clustering or
>> > classification of errors). To do this data mining, I bet it
would be
>> > easier if I had all the data in one big file of all the storm
ids, from
>> > 201711 - 201809. I could accomplish this with python, opening
the files
>> > and appending to dataframes, etc. But, what I thought I should
do is
>> ask
>> > you if there was an easy way to do that in MET. Is there a way
to
>> concat
>> > all the files with the *tcst ending into one big file?
>> >
>> > Thanks!
>> >
>> > Roz
>> >
>> > On Tue, Nov 27, 2018 at 2:31 PM Rosalyn MacCracken - NOAA
Affiliate <
>> > rosalyn.maccracken at noaa.gov> wrote:
>> >
>> > > Hi John,
>> > >
>> > > That worked! Thank you so much for seeing my little mistake.
I
>> think,
>> > > originally, I had something in those brackets, and took it out,
but,
>> left
>> > > the quotes inside the brackets. I was hoping it was something
easy
>> like
>> > > that.
>> > >
>> > > So, I have a few more questions.
>> > > 1) I want to use a domain file, like I did with point_stat.
For
>> that, I
>> > > created a *.poly file, then, converted it to a *.nc. Which do
I use
>> for
>> > > tc_pairs?
>> > > 2) Is there a way to change the wind speed warning categories,
from
>> > > 34kts, 50kts and 64kts, just in case OPC might want some other
>> category?
>> > >
>> > > Thinking about what I'm going to do with this using tc_stat, I
will
>> need
>> > > to calculate track errors(in nm) and intensity errors by
pressure, and
>> > > also, intensity errors by lead time. I imagine I will be able
to do
>> that
>> > > using -column_thresh, or something else, correct? I saw that
on the
>> > Online
>> > > Tutorial page. But, I guess my question about those things
would be,
>> is
>> > > there anything I need to make sure that is in my adeck/bdeck
files
>> that
>> > is
>> > > not included now, that I should include to make those tasks
easier? I
>> > just
>> > > want to make sure that I've included everything I need.
>> > >
>> > > Roz
>> > >
>> > > On Tue, Nov 27, 2018 at 11:43 AM John Halley Gotway via RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > >> Roz,
>> > >>
>> > >> Thanks for posting your sample data. I found the problem in
your
>> > >> TCPairsConfig file. Change...
>> > >> FROM: storm_id = [ "" ];
>> > >> TO: storm_id = [ ];
>> > >>
>> > >> The storm_id entry is an array of storm ID's to match. An
empty
>> array
>> > is
>> > >> just square brackets with nothing inside ([ ]). Since you had
>> > >> double-quotes inside, that's an array of length 1 containing
an empty
>> > >> string.
>> > >>
>> > >> Thanks,
>> > >> John
>> > >>
>> > >>
>> > >> On Tue, Nov 27, 2018 at 4:50 AM Rosalyn MacCracken - NOAA
Affiliate
>> via
>> > >> RT <
>> > >> met_help at ucar.edu> wrote:
>> > >>
>> > >> >
>> > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
>> > >> >
>> > >> > Hi John,
>> > >> >
>> > >> > I uploaded the original files (Adeck = aml*, Bdeck=bml*,
without
>> the
>> > >> > *modified.dat ending), the modified files (with the
*modified.dat
>> > >> ending)
>> > >> > and the config file into my ftp directory (*cd
>> > >> > incoming/irap/met_help/maccracken). *Let me know what you
find.
>> > >> >
>> > >> > Thanks for your help!
>> > >> >
>> > >> > Roz
>> > >> >
>> > >> > On Mon, Nov 26, 2018 at 5:22 PM John Halley Gotway via RT <
>> > >> > met_help at ucar.edu>
>> > >> > wrote:
>> > >> >
>> > >> > > Roz,
>> > >> > >
>> > >> > > Yes, the first 8 columns are required. The additional
ones are
>> > >> optional.
>> > >> > >
>> > >> > > Looking at your log output, I see that tc_pairs is
skipping over
>> all
>> > >> 642
>> > >> > > input lines. And the question is why? If you send me
your
>> latest
>> > >> ADeck,
>> > >> > > BDeck, and TC-Pairs config files, I could run it here and
try to
>> > >> figure
>> > >> > out
>> > >> > > why it isn't defining any ADECK tracks.
>> > >> > >
>> > >> > > Thanks,
>> > >> > > John
>> > >> > >
>> > >> > > On Mon, Nov 26, 2018 at 12:23 PM Rosalyn MacCracken - NOAA
>> Affiliate
>> > >> via
>> > >> > RT
>> > >> > > <met_help at ucar.edu> wrote:
>> > >> > >
>> > >> > > >
>> > >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
>> >
>> > >> > > >
>> > >> > > > So, can I just have 8 columns for the input file, or, do
I
>> need to
>> > >> make
>> > >> > > > columns of -9999 or NaN for all the rest of the columns?
>> > >> > > >
>> > >> > > > I experimented, and just wrote out 10 columns for the
input
>> file:
>> > >> > > > agfso_trackdf_mod =
>> > >> > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
agfso_trackdf[['basin_name','storm_id','model_start_date','const','model','fcst_lead_time','vort_lat_cent','vort_lon_cent','vort_10m_max',
>> > >> > > > 'vort_min_mslp','type','thresh_wspd']]
>> > >> > > >
>> > >> > > > because I wanted stats for these variables, and I got
the
>> output
>> > >> below.
>> > >> > > > So, maybe it's working, but, the config file needs to be
>> > different?
>> > >> > Any
>> > >> > > > other ideas of what I should try doing/looking at?
>> > >> > > >
>> > >> > > > Roz
>> > >> > > >
>> > >> > > > Output:
>> > >> > > > DEBUG 1: [Source 1 of 1] ADECK Source:
>> > >> > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018083118.gfso.0053_modified.dat,
>> > >> > > > Model Suffix: _EXP
>> > >> > > > DEBUG 1: [Source 1 of 1] BDECK Source:
>> > >> > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018083118.gfso.0053_modified.dat,
>> > >> > > > Model Suffix: (nul)
>> > >> > > > DEBUG 1: Config File Default:
>> > >> > > > /opt/MET7/share/met/config/TCPairsConfig_default
>> > >> > > > DEBUG 1: Config File User:
>> > >> > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
>> > >> > > > DEBUG 1: Distance to land file:
/opt/MET7/share/met/tc_data/
>> > >> > > > dland_global_tenth_degree.nc
>> > >> > > > DEBUG 4:
>> > >> > > > DEBUG 4: Latitude/Longitude Grid Data:
>> > >> > > > DEBUG 4: lat_ll: -90
>> > >> > > > DEBUG 4: lon_ll: 180
>> > >> > > > DEBUG 4: delta_lat: 0.1
>> > >> > > > DEBUG 4: delta_lon: 0.1
>> > >> > > > DEBUG 4: Nlat: 1801
>> > >> > > > DEBUG 4: Nlon: 3601
>> > >> > > > DEBUG 4:
>> > >> > > > DEBUG 2: Processing 1 BDECK file(s).
>> > >> > > > DEBUG 4: [File 1 of 1] Used 30 of 30 lines read from
file
>> > >> > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018083118.gfso.0053_modified.dat"
>> > >> > > > DEBUG 3: Used 30 of 30 lines read from 1 file(s).
>> > >> > > > DEBUG 3: Identified 1 track(s).
>> > >> > > > DEBUG 4: [Track 1 of 1] TrackInfo: StormId =
"ML0800532018",
>> > IsBest
>> > >> =
>> > >> > > > false, IsOper = false, CheckAnly = true, IsAnly = true,
Basin =
>> > >> "ML",
>> > >> > > > Cyclone = "080053", StormName = "", TechniqueNumber = 3,
>> > Technique =
>> > >> > > > "GFSO", Initials = "", InitTime = NA, MinValidTime =
>> > >> 20180805_060000,
>> > >> > > > MaxValidTime = 20180812_120000, NPoints = 30, NAlloc =
100,
>> > >> NTrackLines
>> > >> > > = 0
>> > >> > > > DEBUG 2: Found 1 BDECK track(s).
>> > >> > > > DEBUG 2: Processing 1 ADECK file(s).
>> > >> > > > DEBUG 4: [File 1 of 1] Used 0 of 642 lines read from
file
>> > >> > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018083118.gfso.0053_modified.dat"
>> > >> > > > DEBUG 3: Used 0 of 642 lines read from 1 file(s).
>> > >> > > > DEBUG 3: Identified 0 track(s).
>> > >> > > > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
>> > >> > > > DEBUG 2: Finished adding 0 and replacing 0 Interp12
track(s).
>> > >> > > > DEBUG 2: Deriving 0 ADECK consensus model(s).
>> > >> > > > DEBUG 2: Added 0 ADECK consensus tracks(s).
>> > >> > > > DEBUG 2: Deriving 0 ADECK lag model(s).
>> > >> > > > DEBUG 2: Added 0 ADECK lag tracks(s).
>> > >> > > > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
>> > >> > > > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
>> > >> > > > DEBUG 2: Filtering 0 ADECK tracks based on config file
>> settings.
>> > >> > > > DEBUG 3: Total tracks read = 0
>> > >> > > > DEBUG 3: Total tracks kept = 0
>> > >> > > > DEBUG 3: Rejected for storm name = 0
>> > >> > > > DEBUG 3: Rejected for valid time = 0
>> > >> > > > DEBUG 3: Rejected for required lead times = 0
>> > >> > > > DEBUG 3: Rejected for init mask = 0
>> > >> > > > DEBUG 3: Rejected for valid mask = 0
>> > >> > > > DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
>> > >> > > > DEBUG 1: Watch/Warning file:
>> > >> /opt/MET7/share/met/tc_data/wwpts_us.txt
>> > >> > > > DEBUG 1: Output file:
>> > >> > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018083118.gfso.0053_test_modified_output.txt.tcst
>> > >> > > >
>> > >> > > >
>> > >> > > > On Mon, Nov 26, 2018 at 2:06 PM John Halley Gotway via
RT <
>> > >> > > > met_help at ucar.edu>
>> > >> > > > wrote:
>> > >> > > >
>> > >> > > > > Roz,
>> > >> > > > >
>> > >> > > > > Page 334 of the user's guide (
>> > >> > > > >
>> > >> > >
>> > >>
>> >
>>
https://dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v8.0.pdf
>> > >> > > > )
>> > >> > > > > mentions that only the first 8 ATCF columns are
required. If
>> > >> > > additional
>> > >> > > > > columns are present in the input ATCF files, then
tc_pairs
>> will
>> > >> > > interpret
>> > >> > > > > them as specified by the ATCF format.
>> > >> > > > >
>> > >> > > > > So if the storm name column is present, MET in
interpret
>> those
>> > >> values
>> > >> > > as
>> > >> > > > > the storm name.
>> > >> > > > >
>> > >> > > > > For Guang Ping's data, probably best to have that
column
>> empty.
>> > >> > > > >
>> > >> > > > > John
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > On Mon, Nov 26, 2018 at 11:51 AM Rosalyn MacCracken -
NOAA
>> > >> Affiliate
>> > >> > > via
>> > >> > > > RT
>> > >> > > > > <met_help at ucar.edu> wrote:
>> > >> > > > >
>> > >> > > > > >
>> > >> > > > > > <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
>> > >
>> > >> > > > > >
>> > >> > > > > > Hi John,
>> > >> > > > > >
>> > >> > > > > > Ok, I will ignore the suffix error. I do have
numeric
>> values
>> > in
>> > >> > > > > > STORMNAME. I wonder if in the modified files I
have, this
>> > >> column
>> > >> > is
>> > >> > > > not
>> > >> > > > > > actually STORMNAME, but, some other variable. Now,
I don't
>> > have
>> > >> > the
>> > >> > > > > number
>> > >> > > > > > of columns that the NRL website has. Will that be a
>> problem?
>> > >> > > Should I
>> > >> > > > > > create dummy columns?
>> > >> > > > > >
>> > >> > > > > > Roz
>> > >> > > > > >
>> > >> > > > > > On Mon, Nov 26, 2018 at 1:14 PM John Halley Gotway
via RT <
>> > >> > > > > > met_help at ucar.edu>
>> > >> > > > > > wrote:
>> > >> > > > > >
>> > >> > > > > > > Roz,
>> > >> > > > > > >
>> > >> > > > > > > I see that you're wondering about this log
message:
>> > >> > > > > > > *Model Suffix: (nul)*
>> > >> > > > > > >
>> > >> > > > > > > Everything is fine, you do not need to be
concerned about
>> > >> this.
>> > >> > > > > Here's a
>> > >> > > > > > > line taken from the usage statement for the
tc_pairs
>> tool:
>> > >> > > > > > > *Note: The "-adeck", "-edeck", and "-bdeck"
options may
>> > >> include
>> > >> > > > > > > "suffix=string" to modify the model names from
that
>> source.*
>> > >> > > > > > >
>> > >> > > > > > > Since you didn't specify "suffix=string" on the
command
>> > line,
>> > >> the
>> > >> > > > > suffix
>> > >> > > > > > > remains the default value of an empty string, i.e.
>> "(nul)".
>> > >> So
>> > >> > why
>> > >> > > > > would
>> > >> > > > > > > you ever want to use a suffix? Sometimes data is
>> messy. We
>> > >> were
>> > >> > > > > running
>> > >> > > > > > > tc_pairs to evaluate track data from multiple
sources
>> and we
>> > >> > found
>> > >> > > > that
>> > >> > > > > > > some model names we reused in both sources. Let's
say
>> the
>> > >> OFCL
>> > >> > > > > forecast
>> > >> > > > > > > showed up in both stream 1.0 and stream 1.5 data.
>> > Specifying
>> > >> a
>> > >> > > > suffix
>> > >> > > > > > such
>> > >> > > > > > > as "suffix=_STREAM1.5" changes the model name from
>> "OFCL" to
>> > >> > > > > > > "OFCL_STREAM1.5". And that enables us to
distinguish
>> > between
>> > >> > those
>> > >> > > > > > tracks
>> > >> > > > > > > in the output.
>> > >> > > > > > >
>> > >> > > > > > > But uou probably don't have any need for this
option in
>> your
>> > >> > > current
>> > >> > > > > > work.
>> > >> > > > > > >
>> > >> > > > > > > However, I am concerned about these log messages:
>> > >> > > > > > > DEBUG 4: Updating storm name from "-287" to "-305"
for
>> > >> > > ML0900022018.
>> > >> > > > > > > DEBUG 4: Updating storm name from "-305" to "-255"
for
>> > >> > > ML0900022018.
>> > >> > > > > > > DEBUG 4: Updating storm name from "-255" to "-203"
for
>> > >> > > ML0900022018.
>> > >> > > > > > > DEBUG 4: Updating storm name from "-203" to "-205"
for
>> > >> > > ML0900022018.
>> > >> > > > > > > DEBUG 4: Updating storm name from "-205" to "-217"
for
>> > >> > > ML0900022018.
>> > >> > > > > > > DEBUG 4: Updating storm name from "-217" to "-211"
for
>> > >> > > ML0900022018.
>> > >> > > > > > > DEBUG 4: Updating storm name from "-211" to "-182"
for
>> > >> > > ML0900022018.
>> > >> > > > > > > DEBUG 4: Updating storm name from "-182" to "-138"
for
>> > >> > > ML0900022018.
>> > >> > > > > > >
>> > >> > > > > > > You likely have numeric values showing up in the
>> "STORMNAME"
>> > >> ATCF
>> > >> > > > > column:
>> > >> > > > > > >
>> > >> > >
>> https://www.nrlmry.navy.mil/atcf_web/docs/database/new/abdeck.txt
>> > >> > > > > > >
>> > >> > > > > > > I'd recommend looking into that.
>> > >> > > > > > >
>> > >> > > > > > > Thanks,
>> > >> > > > > > > John
>> > >> > > > > > >
>> > >> > > > > > > On Mon, Nov 26, 2018 at 8:44 AM Rosalyn MacCracken
- NOAA
>> > >> > Affiliate
>> > >> > > > via
>> > >> > > > > > RT
>> > >> > > > > > > <
>> > >> > > > > > > met_help at ucar.edu> wrote:
>> > >> > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > > > <URL:
>> > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
>> > >> > >
>> > >> > > > > > > >
>> > >> > > > > > > > Hi John,
>> > >> > > > > > > >
>> > >> > > > > > > > I figured out an easy way to read in and write
out
>> Guang
>> > >> Ping's
>> > >> > > > > > modified
>> > >> > > > > > > > ATCF file, so that it:
>> > >> > > > > > > > 1) Removes the 3rd column
>> > >> > > > > > > > 2) Replaces -99 with -9999 in column 21 and 24
(after
>> the
>> > >> 3rd
>> > >> > > > column
>> > >> > > > > > is
>> > >> > > > > > > > removed)
>> > >> > > > > > > > 3) Updates column 2 to include month+cyclone
number.
>> > (You
>> > >> had
>> > >> > > > said
>> > >> > > > > to
>> > >> > > > > > > do
>> > >> > > > > > > > year+cyclone #, but the tc_pairs_wrapper.py said
to do
>> > >> > > > > month+cyclone#)
>> > >> > > > > > > >
>> > >> > > > > > > > I changed Adeck and Bdeck.
>> > >> > > > > > > > Just a couple lines of my file:
>> > >> > > > > > > >
>> > >> > > > > > > > ML, 090002, 2018090100, 03, GFSO, 000, 511S,
897W,
>> 40,
>> > >> 976,
>> > >> > > XX,
>> > >> > > > > 34,
>> > >> > > > > > > > NEQ, 0625, 0630, 0000, 0226, 978, 68, -9999,
123,
>> 155,
>> > >> > -9999,
>> > >> > > > > -9999,
>> > >> > > > > > > > -9999, -297, -394, -287, -458, 0073
>> > >> > > > > > > > ML, 090002, 2018090106, 03, GFSO, 000, 537S,
854W,
>> 44,
>> > >> 964,
>> > >> > > XX,
>> > >> > > > > 34,
>> > >> > > > > > > > NEQ, 0640, 0645, 0000, 0297, 969, 108, -9999,
119,
>> 147,
>> > >> > -9999,
>> > >> > > > > -9999,
>> > >> > > > > > > > -9999, -403, -648, -305, -394, 0055
>> > >> > > > > > > >
>> > >> > > > > > > > So, I've run tc_pairs again and my output is
below. I
>> > don't
>> > >> > > > > understand
>> > >> > > > > > > the
>> > >> > > > > > > > Model Suffix: (nul). Is the input format still
wrong?
>> > >> > > > > > > >
>> > >> > > > > > > > Roz
>> > >> > > > > > > >
>> > >> > > > > > > > Output:
>> > >> > > > > > > > /opt/MET7/bin/tc_pairs -adeck
>> > >> > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat
>> > >> > > > > > > > -bdeck
>> > >> > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat
>> > >> > > > > > > > -config
>> > >> > > > > >
>> > ~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
>> > >> > > > > > > > -out
>> > >> > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt
>> > >> > > > > > > > -log
>> > >> > > > > > >
>> > >> > >
>> >
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
>> > >> > > > > > > > -v 4
>> > >> > > > > > > > DEBUG 1: [Source 1 of 1] ADECK Source:
>> > >> > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat,
>> > >> > > > > > > > Model Suffix: (nul)
>> > >> > > > > > > > DEBUG 1: [Source 1 of 1] BDECK Source:
>> > >> > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat,
>> > >> > > > > > > > Model Suffix: (nul)
>> > >> > > > > > > > DEBUG 1: Config File Default:
>> > >> > > > > > > > /opt/MET7/share/met/config/TCPairsConfig_default
>> > >> > > > > > > > DEBUG 1: Config File User:
>> > >> > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
>> > >> > > > > > > > DEBUG 1: Distance to land file:
>> > /opt/MET7/share/met/tc_data/
>> > >> > > > > > > > dland_global_tenth_degree.nc
>> > >> > > > > > > > DEBUG 4:
>> > >> > > > > > > > DEBUG 4: Latitude/Longitude Grid Data:
>> > >> > > > > > > > DEBUG 4: lat_ll: -90
>> > >> > > > > > > > DEBUG 4: lon_ll: 180
>> > >> > > > > > > > DEBUG 4: delta_lat: 0.1
>> > >> > > > > > > > DEBUG 4: delta_lon: 0.1
>> > >> > > > > > > > DEBUG 4: Nlat: 1801
>> > >> > > > > > > > DEBUG 4: Nlon: 3601
>> > >> > > > > > > > DEBUG 4:
>> > >> > > > > > > > DEBUG 2: Processing 1 BDECK file(s).
>> > >> > > > > > > > DEBUG 4: Updating storm name from "-287" to "-
305" for
>> > >> > > > ML0900022018.
>> > >> > > > > > > > DEBUG 4: Updating storm name from "-305" to "-
255" for
>> > >> > > > ML0900022018.
>> > >> > > > > > > > DEBUG 4: Updating storm name from "-255" to "-
203" for
>> > >> > > > ML0900022018.
>> > >> > > > > > > > DEBUG 4: Updating storm name from "-203" to "-
205" for
>> > >> > > > ML0900022018.
>> > >> > > > > > > > DEBUG 4: Updating storm name from "-205" to "-
217" for
>> > >> > > > ML0900022018.
>> > >> > > > > > > > DEBUG 4: Updating storm name from "-217" to "-
211" for
>> > >> > > > ML0900022018.
>> > >> > > > > > > > DEBUG 4: Updating storm name from "-211" to "-
182" for
>> > >> > > > ML0900022018.
>> > >> > > > > > > > DEBUG 4: Updating storm name from "-182" to "-
138" for
>> > >> > > > ML0900022018.
>> > >> > > > > > > > DEBUG 4: [File 1 of 1] Used 9 of 9 lines read
from file
>> > >> > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat"
>> > >> > > > > > > > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
>> > >> > > > > > > > DEBUG 3: Identified 1 track(s).
>> > >> > > > > > > > DEBUG 4: [Track 1 of 1] TrackInfo: StormId =
>> > "ML0900022018",
>> > >> > > > IsBest =
>> > >> > > > > > > > false, IsOper = false, CheckAnly = true, IsAnly
= true,
>> > >> Basin =
>> > >> > > > "ML",
>> > >> > > > > > > > Cyclone = "090002", StormName = "-138",
>> TechniqueNumber =
>> > 3,
>> > >> > > > > Technique
>> > >> > > > > > =
>> > >> > > > > > > > "GFSO", Initials = "-9999", InitTime = NA,
>> MinValidTime =
>> > >> > > > > > > 20180901_000000,
>> > >> > > > > > > > MaxValidTime = 20180903_000000, NPoints = 9,
NAlloc =
>> 100,
>> > >> > > > > NTrackLines
>> > >> > > > > > =
>> > >> > > > > > > 0
>> > >> > > > > > > > DEBUG 2: Found 1 BDECK track(s).
>> > >> > > > > > > > DEBUG 2: Processing 1 ADECK file(s).
>> > >> > > > > > > > DEBUG 4: [File 1 of 1] Used 0 of 21 lines read
from
>> file
>> > >> > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat"
>> > >> > > > > > > > DEBUG 3: Used 0 of 21 lines read from 1 file(s).
>> > >> > > > > > > > DEBUG 3: Identified 0 track(s).
>> > >> > > > > > > > DEBUG 2: Deriving 12-hour interpolated ADECK
tracks.
>> > >> > > > > > > > DEBUG 2: Finished adding 0 and replacing 0
Interp12
>> > >> track(s).
>> > >> > > > > > > > DEBUG 2: Deriving 0 ADECK consensus model(s).
>> > >> > > > > > > > DEBUG 2: Added 0 ADECK consensus tracks(s).
>> > >> > > > > > > > DEBUG 2: Deriving 0 ADECK lag model(s).
>> > >> > > > > > > > DEBUG 2: Added 0 ADECK lag tracks(s).
>> > >> > > > > > > > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline
model(s).
>> > >> > > > > > > > DEBUG 2: Added 0 CLIPER/SHIFOR baseline
track(s).
>> > >> > > > > > > > DEBUG 2: Filtering 0 ADECK tracks based on
config file
>> > >> > settings.
>> > >> > > > > > > > DEBUG 3: Total tracks read = 0
>> > >> > > > > > > > DEBUG 3: Total tracks kept = 0
>> > >> > > > > > > > DEBUG 3: Rejected for storm name = 0
>> > >> > > > > > > > DEBUG 3: Rejected for valid time = 0
>> > >> > > > > > > > DEBUG 3: Rejected for required lead times = 0
>> > >> > > > > > > > DEBUG 3: Rejected for init mask = 0
>> > >> > > > > > > > DEBUG 3: Rejected for valid mask = 0
>> > >> > > > > > > > DEBUG 2: Matching 0 ADECK tracks to 1 BDECK
tracks.
>> > >> > > > > > > > DEBUG 1: Watch/Warning file:
>> > >> > > > /opt/MET7/share/met/tc_data/wwpts_us.txt
>> > >> > > > > > > > DEBUG 1: Output file:
>> > >> > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt.tcst
>> > >> > > > > > > >
>> > >> > > > > > > > On Wed, Nov 21, 2018 at 9:16 AM Rosalyn
MacCracken -
>> NOAA
>> > >> > > > Affiliate <
>> > >> > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
>> > >> > > > > > > >
>> > >> > > > > > > > > Hi John,
>> > >> > > > > > > > >
>> > >> > > > > > > > > Yes, that makes sense, and yes, I am using
Guang
>> Pings'
>> > >> > > modified
>> > >> > > > > > files.
>> > >> > > > > > > > > I'm not sure how to run tc_pairs_wrapper.py,
>> though. I
>> > >> > think I
>> > >> > > > > need
>> > >> > > > > > > some
>> > >> > > > > > > > > type of *.conf file to pass in the files I
need, or
>> > >> something
>> > >> > > > like
>> > >> > > > > > > that.
>> > >> > > > > > > > > I'm in the process of reading through the code
now.
>> I
>> > >> wonder
>> > >> > > if
>> > >> > > > I
>> > >> > > > > > > could
>> > >> > > > > > > > > just pull out the function process_non_atcf,
to run
>> in a
>> > >> > > > > stand-alone
>> > >> > > > > > > > mode.
>> > >> > > > > > > > > I wish there was some sort of documentation
with this
>> > >> code...
>> > >> > > > > > > > >
>> > >> > > > > > > > > And, so, once I can read these files, I can
use
>> tc_pairs
>> > >> to
>> > >> > > > create
>> > >> > > > > > the
>> > >> > > > > > > > > forecast error statistics, and then use
tc_stats to
>> > >> aggregate
>> > >> > > the
>> > >> > > > > > files
>> > >> > > > > > > > and
>> > >> > > > > > > > > get the mean, etc, for a period of time,
correct?
>> Was
>> > >> there
>> > >> > > some
>> > >> > > > > > sort
>> > >> > > > > > > of
>> > >> > > > > > > > > plotting scripts for the output files?
>> > >> > > > > > > > >
>> > >> > > > > > > > > So, with tc_pairs, I can define a region, like
I did
>> > with
>> > >> > > > > point_stat,
>> > >> > > > > > > > > correct? Can I use the same *.nc files, or do
I use
>> the
>> > >> > > original
>> > >> > > > > > > *.poly
>> > >> > > > > > > > > files?
>> > >> > > > > > > > >
>> > >> > > > > > > > > Roz
>> > >> > > > > > > > >
>> > >> > > > > > > > > On Tue, Nov 20, 2018 at 7:53 PM John Halley
Gotway
>> via
>> > RT
>> > >> <
>> > >> > > > > > > > > met_help at ucar.edu> wrote:
>> > >> > > > > > > > >
>> > >> > > > > > > > >> Hi Roz,
>> > >> > > > > > > > >>
>> > >> > > > > > > > >> Yes, I see you’re using the modified ATCF
format
>> that
>> > >> Guang
>> > >> > > Ping
>> > >> > > > > > uses.
>> > >> > > > > > > > >> You’re correct that tc_pairs doesn’t parse
this
>> > directly
>> > >> > since
>> > >> > > > it
>> > >> > > > > > > > differs
>> > >> > > > > > > > >> slightly from the ATCF format.
>> > >> > > > > > > > >>
>> > >> > > > > > > > >> When we’ve processed this data in the past,
we first
>> > >> > > > pre-processed
>> > >> > > > > > it
>> > >> > > > > > > > in a
>> > >> > > > > > > > >> shell script to:
>> > >> > > > > > > > >>
>> > >> > > > > > > > >> 1. Replace the -99 value with -9999 for
missing
>> data.
>> > >> > > > > > > > >> 2. Remove the 3rd column
>> > >> > > > > > > > >> 3. Updating the 2nd column to include the
year and
>> > >> cyclone
>> > >> > > > number.
>> > >> > > > > > > > >>
>> > >> > > > > > > > >> And that enable tc_pairs to read it.
>> > >> > > > > > > > >>
>> > >> > > > > > > > >> This is actually Done in python by one of the
use
>> cases
>> > >> in
>> > >> > > > > METplus.
>> > >> > > > > > > > >>
>> > >> > > > > > > > >> Take a look at this file:
>> > >> > > > > > > > >>
>> > >> > > > >
>> > >>
https://github.com/NCAR/METplus/blob/master/ush/tc_pairs_wrapper.py
>> > >> > > > > > > > >>
>> > >> > > > > > > > >> At the function named:
>> > >> > > > > > > > >> process_non_atcf
>> > >> > > > > > > > >>
>> > >> > > > > > > > >> I haven’t run this using METplus scripts
myself, but
>> > all
>> > >> the
>> > >> > > > > pieces
>> > >> > > > > > of
>> > >> > > > > > > > >> logic are there, and it should be possible.
>> > >> > > > > > > > >>
>> > >> > > > > > > > >> Make sense?
>> > >> > > > > > > > >>
>> > >> > > > > > > > >> Thanks
>> > >> > > > > > > > >> John
>> > >> > > > > > > > >>
>> > >> > > > > > > > >> On Tue, Nov 20, 2018 at 1:13 PM Rosalyn
MacCracken -
>> > NOAA
>> > >> > > > > Affiliate
>> > >> > > > > > > via
>> > >> > > > > > > > >> RT <
>> > >> > > > > > > > >> met_help at ucar.edu> wrote:
>> > >> > > > > > > > >>
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> > Tue Nov 20 13:13:05 2018: Request 87885 was
acted
>> > upon.
>> > >> > > > > > > > >> > Transaction: Ticket created by
>> > >> > rosalyn.maccracken at noaa.gov
>> > >> > > > > > > > >> > Queue: met_help
>> > >> > > > > > > > >> > Subject: Help with using TC_pairs
>> > >> > > > > > > > >> > Owner: Nobody
>> > >> > > > > > > > >> > Requestors: rosalyn.maccracken at noaa.gov
>> > >> > > > > > > > >> > Status: new
>> > >> > > > > > > > >> > Ticket <URL:
>> > >> > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
>> > >> > > > > > > > >
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> > Hi,
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> > I'm trying to use TC_pairs for extra-
tropical
>> > cyclones
>> > >> > > > > (basin=ML).
>> > >> > > > > > > I
>> > >> > > > > > > > >> think
>> > >> > > > > > > > >> > the format of the ATCF data is the issue,
but, I'm
>> > not
>> > >> > sure.
>> > >> > > > I
>> > >> > > > > > have
>> > >> > > > > > > > an
>> > >> > > > > > > > >> > Adeck and a Bdeck file, where a sample of
one of
>> the
>> > >> lines
>> > >> > > is:
>> > >> > > > > > > > >> > ML, 0003, 2018090100_F000_687N_0159W_FOF,
>> 2018090100,
>> > >> 03,
>> > >> > > > GFSO,
>> > >> > > > > > 000,
>> > >> > > > > > > > >> 687N,
>> > >> > > > > > > > >> > 159W, 32, 987, XX, 34, NEQ, 0301, 0338,
0000,
>> > 0000,
>> > >> > 988,
>> > >> > > > > 52,
>> > >> > > > > > > > -99,
>> > >> > > > > > > > >> > 330, 86, -99, -9999, -9999, 175, 202,
137,
>> 135,
>> > >> 0081
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> > This is slightly different from the NRL
ATCF
>> format
>> > on
>> > >> > their
>> > >> > > > > > > website,
>> > >> > > > > > > > >> such
>> > >> > > > > > > > >> > as the second column. Anyway, I configured
the
>> > config
>> > >> > file
>> > >> > > > very
>> > >> > > > > > > > simply,
>> > >> > > > > > > > >> > (model=GFS, basin=ML), and ran tc_pairs,
and got
>> the
>> > >> > output
>> > >> > > to
>> > >> > > > > the
>> > >> > > > > > > > >> screen
>> > >> > > > > > > > >> > below.
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> > Is there a way to work around the diferent
ATCF
>> > format?
>> > >> > > I've
>> > >> > > > > > > > attached a
>> > >> > > > > > > > >> > sample of my Adeck and Bdeck files.
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> > Thanks,
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> > Roz
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> > Output to the screen:
>> > >> > > > > > > > >> > /opt/MET7/bin/tc_pairs -adeck
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >>
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
~/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003
>> > >> > > > > > > > >> > -bdeck
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >>
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
~/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003
>> > >> > > > > > > > >> > -config
>> > >> > > > > > > > >> >
>> > >> > > > > > > >
>> > >> > > > >
>> > >> >
>> >
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
>> > >> > > > > > > > >> > -out
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >>
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt
>> > >> > > > > > > > >> > -log
>> > >> > > > > > > > >>
>> > >> > > > > >
>> > >> >
>>
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
>> > >> > > > > > > > >> > -v 3
>> > >> > > > > > > > >> > DEBUG 1: [Source 1 of 1] ADECK Source:
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >>
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003,
>> > >> > > > > > > > >> > Model Suffix: (nul)
>> > >> > > > > > > > >> > DEBUG 1: [Source 1 of 1] BDECK Source:
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >>
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003,
>> > >> > > > > > > > >> > Model Suffix: (nul)
>> > >> > > > > > > > >> > DEBUG 1: Config File Default:
>> > >> > > > > > > > >> >
/opt/MET7/share/met/config/TCPairsConfig_default
>> > >> > > > > > > > >> > DEBUG 1: Config File User:
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >>
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
>> > >> > > > > > > > >> > DEBUG 1: Distance to land file:
>> > >> > /opt/MET7/share/met/tc_data/
>> > >> > > > > > > > >> > dland_global_tenth_degree.nc
>> > >> > > > > > > > >> > DEBUG 2: Processing 1 BDECK file(s).
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
>> latitude
>> > >> ...
>> > >> > > "000"
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
>> longitude
>> > >> ...
>> > >> > > > "687N"
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
>> latitude
>> > >> ...
>> > >> > > "000"
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
>> longitude
>> > >> ...
>> > >> > > > "701N"
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
>> latitude
>> > >> ...
>> > >> > > "000"
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
>> longitude
>> > >> ...
>> > >> > > > "711N"
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
>> latitude
>> > >> ...
>> > >> > > "000"
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
>> longitude
>> > >> ...
>> > >> > > > "720N"
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
>> latitude
>> > >> ...
>> > >> > > "000"
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
>> longitude
>> > >> ...
>> > >> > > > "730N"
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
>> latitude
>> > >> ...
>> > >> > > "000"
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
>> longitude
>> > >> ...
>> > >> > > > "741N"
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
>> latitude
>> > >> ...
>> > >> > > "000"
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
>> longitude
>> > >> ...
>> > >> > > > "751N"
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
>> latitude
>> > >> ...
>> > >> > > "000"
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
>> longitude
>> > >> ...
>> > >> > > > "760N"
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING: int parse_lat(const char *) -> bad
>> latitude
>> > >> ...
>> > >> > > "000"
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > WARNING: int parse_lon(const char *) -> bad
>> longitude
>> > >> ...
>> > >> > > > "764N"
>> > >> > > > > > > > >> > WARNING:
>> > >> > > > > > > > >> > DEBUG 3: Used 9 of 9 lines read from 1
file(s).
>> > >> > > > > > > > >> > DEBUG 3: Identified 9 track(s).
>> > >> > > > > > > > >> > DEBUG 2: Found 9 BDECK track(s).
>> > >> > > > > > > > >> > DEBUG 2: Processing 1 ADECK file(s).
>> > >> > > > > > > > >> > DEBUG 3: Used 0 of 22 lines read from 1
file(s).
>> > >> > > > > > > > >> > DEBUG 3: Identified 0 track(s).
>> > >> > > > > > > > >> > DEBUG 2: Deriving 12-hour interpolated
ADECK
>> tracks.
>> > >> > > > > > > > >> > DEBUG 2: Finished adding 0 and replacing 0
>> Interp12
>> > >> > > track(s).
>> > >> > > > > > > > >> > DEBUG 2: Deriving 0 ADECK consensus
model(s).
>> > >> > > > > > > > >> > DEBUG 2: Added 0 ADECK consensus tracks(s).
>> > >> > > > > > > > >> > DEBUG 2: Deriving 0 ADECK lag model(s).
>> > >> > > > > > > > >> > DEBUG 2: Added 0 ADECK lag tracks(s).
>> > >> > > > > > > > >> > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline
>> model(s).
>> > >> > > > > > > > >> > DEBUG 2: Added 0 CLIPER/SHIFOR baseline
track(s).
>> > >> > > > > > > > >> > DEBUG 2: Filtering 0 ADECK tracks based on
config
>> > file
>> > >> > > > settings.
>> > >> > > > > > > > >> > DEBUG 3: Total tracks read =
0
>> > >> > > > > > > > >> > DEBUG 3: Total tracks kept =
0
>> > >> > > > > > > > >> > DEBUG 3: Rejected for storm name =
0
>> > >> > > > > > > > >> > DEBUG 3: Rejected for valid time =
0
>> > >> > > > > > > > >> > DEBUG 3: Rejected for required lead times =
0
>> > >> > > > > > > > >> > DEBUG 3: Rejected for init mask =
0
>> > >> > > > > > > > >> > DEBUG 3: Rejected for valid mask =
0
>> > >> > > > > > > > >> > DEBUG 2: Matching 0 ADECK tracks to 9 BDECK
>> tracks.
>> > >> > > > > > > > >> > DEBUG 1: Watch/Warning file:
>> > >> > > > > > > /opt/MET7/share/met/tc_data/wwpts_us.txt
>> > >> > > > > > > > >> > DEBUG 1: Output file:
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >>
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt.tcst
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> > --
>> > >> > > > > > > > >> > Rosalyn MacCracken
>> > >> > > > > > > > >> > Support Scientist
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> > Ocean Applications Branch
>> > >> > > > > > > > >> > NOAA/NWS Ocean Prediction Center
>> > >> > > > > > > > >> > NCWCP
>> > >> > > > > > > > >> > 5830 University Research Ct
>> > >> > > > > > > > >> > College Park, MD 20740-3818
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> > (p) 301-683-1551
>> > >> > > > > > > > >> > rosalyn.maccracken at noaa.gov
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >> >
>> > >> > > > > > > > >>
>> > >> > > > > > > > >>
>> > >> > > > > > > > >
>> > >> > > > > > > > > --
>> > >> > > > > > > > > Rosalyn MacCracken
>> > >> > > > > > > > > Support Scientist
>> > >> > > > > > > > >
>> > >> > > > > > > > > Ocean Applications Branch
>> > >> > > > > > > > > NOAA/NWS Ocean Prediction Center
>> > >> > > > > > > > > NCWCP
>> > >> > > > > > > > > 5830 University Research Ct
>> > >> > > > > > > > > College Park, MD 20740-3818
>> > >> > > > > > > > >
>> > >> > > > > > > > > (p) 301-683-1551
>> > >> > > > > > > > > rosalyn.maccracken at noaa.gov
>> > >> > > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > > > --
>> > >> > > > > > > > Rosalyn MacCracken
>> > >> > > > > > > > Support Scientist
>> > >> > > > > > > >
>> > >> > > > > > > > Ocean Applications Branch
>> > >> > > > > > > > NOAA/NWS Ocean Prediction Center
>> > >> > > > > > > > NCWCP
>> > >> > > > > > > > 5830 University Research Ct
>> > >> > > > > > > > College Park, MD 20740-3818
>> > >> > > > > > > >
>> > >> > > > > > > > (p) 301-683-1551
>> > >> > > > > > > > rosalyn.maccracken at noaa.gov
>> > >> > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > > > --
>> > >> > > > > > Rosalyn MacCracken
>> > >> > > > > > Support Scientist
>> > >> > > > > >
>> > >> > > > > > Ocean Applications Branch
>> > >> > > > > > NOAA/NWS Ocean Prediction Center
>> > >> > > > > > NCWCP
>> > >> > > > > > 5830 University Research Ct
>> > >> > > > > > College Park, MD 20740-3818
>> > >> > > > > >
>> > >> > > > > > (p) 301-683-1551
>> > >> > > > > > rosalyn.maccracken at noaa.gov
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > >
>> > >> > > > --
>> > >> > > > Rosalyn MacCracken
>> > >> > > > Support Scientist
>> > >> > > >
>> > >> > > > Ocean Applications Branch
>> > >> > > > NOAA/NWS Ocean Prediction Center
>> > >> > > > NCWCP
>> > >> > > > 5830 University Research Ct
>> > >> > > > College Park, MD 20740-3818
>> > >> > > >
>> > >> > > > (p) 301-683-1551
>> > >> > > > rosalyn.maccracken at noaa.gov
>> > >> > > >
>> > >> > > >
>> > >> > >
>> > >> > >
>> > >> >
>> > >> > --
>> > >> > Rosalyn MacCracken
>> > >> > Support Scientist
>> > >> >
>> > >> > Ocean Applications Branch
>> > >> > NOAA/NWS Ocean Prediction Center
>> > >> > NCWCP
>> > >> > 5830 University Research Ct
>> > >> > College Park, MD 20740-3818
>> > >> >
>> > >> > (p) 301-683-1551
>> > >> > rosalyn.maccracken at noaa.gov
>> > >> >
>> > >> >
>> > >>
>> > >>
>> > >
>> > > --
>> > > Rosalyn MacCracken
>> > > Support Scientist
>> > >
>> > > Ocean Applications Branch
>> > > NOAA/NWS Ocean Prediction Center
>> > > NCWCP
>> > > 5830 University Research Ct
>> > > College Park, MD 20740-3818
>> > >
>> > > (p) 301-683-1551
>> > > rosalyn.maccracken at noaa.gov
>> > >
>> >
>> >
>> > --
>> > Rosalyn MacCracken
>> > Support Scientist
>> >
>> > Ocean Applications Branch
>> > NOAA/NWS Ocean Prediction Center
>> > NCWCP
>> > 5830 University Research Ct
>> > College Park, MD 20740-3818
>> >
>> > (p) 301-683-1551
>> > rosalyn.maccracken at noaa.gov
>> >
>> >
>>
>>
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
--
Rosalyn MacCracken
Support Scientist
Ocean Applications Branch
NOAA/NWS Ocean Prediction Center
NCWCP
5830 University Research Ct
College Park, MD 20740-3818
(p) 301-683-1551
rosalyn.maccracken at noaa.gov
------------------------------------------------
Subject: Help with using TC_pairs
From: John Halley Gotway
Time: Thu Dec 06 10:20:46 2018
You can cat them all together but it'll include the header line which
shows
up at the top of each of those files. I *think* tc_stat is smart
enough to
ignore those when they show up in the middle of a file. However, it
may
trip you up in your python code.
Of course, there's always multiple ways of accomplishing each thing.
So
it's totally up to you.
Thanks,
John
On Thu, Dec 6, 2018 at 10:07 AM Rosalyn MacCracken - NOAA Affiliate
via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
>
> Hi John,
>
> Oh, thanks! Is this better than just doing a "cat * > merged_file"
in
> linux, or does it essentially do the same thing?
>
> Roz
>
> On Thu, Dec 6, 2018 at 12:05 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Roz,
> >
> > Sure, you can do that with tc_stat. Just use tc_stat to run a "-
filter"
> > job without actually applying any filtering criteria. And then
use the
> > "-dump_row" option to write the filtered output to a file.
> >
> > That will effectively concatenate multiple files into a single
output
> file.
> >
> > John
> >
> > On Thu, Dec 6, 2018 at 9:46 AM Rosalyn MacCracken - NOAA Affiliate
via
> RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
> > >
> > > Hi John,
> > >
> > > I got tc_stat to work the way I wanted to, and was able to
produce
> > summary
> > > statistics. Now that I have matched pairs, I was thinking about
doing
> > some
> > > other things, unrelated to MET (like data mining, clustering or
> > > classification of errors). To do this data mining, I bet it
would be
> > > easier if I had all the data in one big file of all the storm
ids, from
> > > 201711 - 201809. I could accomplish this with python, opening
the
> files
> > > and appending to dataframes, etc. But, what I thought I should
do is
> ask
> > > you if there was an easy way to do that in MET. Is there a way
to
> concat
> > > all the files with the *tcst ending into one big file?
> > >
> > > Thanks!
> > >
> > > Roz
> > >
> > > On Tue, Nov 27, 2018 at 2:31 PM Rosalyn MacCracken - NOAA
Affiliate <
> > > rosalyn.maccracken at noaa.gov> wrote:
> > >
> > > > Hi John,
> > > >
> > > > That worked! Thank you so much for seeing my little mistake.
I
> think,
> > > > originally, I had something in those brackets, and took it
out, but,
> > left
> > > > the quotes inside the brackets. I was hoping it was something
easy
> > like
> > > > that.
> > > >
> > > > So, I have a few more questions.
> > > > 1) I want to use a domain file, like I did with point_stat.
For
> > that, I
> > > > created a *.poly file, then, converted it to a *.nc. Which do
I use
> > for
> > > > tc_pairs?
> > > > 2) Is there a way to change the wind speed warning
categories, from
> > > > 34kts, 50kts and 64kts, just in case OPC might want some other
> > category?
> > > >
> > > > Thinking about what I'm going to do with this using tc_stat, I
will
> > need
> > > > to calculate track errors(in nm) and intensity errors by
pressure,
> and
> > > > also, intensity errors by lead time. I imagine I will be able
to do
> > that
> > > > using -column_thresh, or something else, correct? I saw that
on the
> > > Online
> > > > Tutorial page. But, I guess my question about those things
would be,
> > is
> > > > there anything I need to make sure that is in my adeck/bdeck
files
> that
> > > is
> > > > not included now, that I should include to make those tasks
easier?
> I
> > > just
> > > > want to make sure that I've included everything I need.
> > > >
> > > > Roz
> > > >
> > > > On Tue, Nov 27, 2018 at 11:43 AM John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >> Roz,
> > > >>
> > > >> Thanks for posting your sample data. I found the problem in
your
> > > >> TCPairsConfig file. Change...
> > > >> FROM: storm_id = [ "" ];
> > > >> TO: storm_id = [ ];
> > > >>
> > > >> The storm_id entry is an array of storm ID's to match. An
empty
> array
> > > is
> > > >> just square brackets with nothing inside ([ ]). Since you
had
> > > >> double-quotes inside, that's an array of length 1 containing
an
> empty
> > > >> string.
> > > >>
> > > >> Thanks,
> > > >> John
> > > >>
> > > >>
> > > >> On Tue, Nov 27, 2018 at 4:50 AM Rosalyn MacCracken - NOAA
Affiliate
> > via
> > > >> RT <
> > > >> met_help at ucar.edu> wrote:
> > > >>
> > > >> >
> > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885 >
> > > >> >
> > > >> > Hi John,
> > > >> >
> > > >> > I uploaded the original files (Adeck = aml*, Bdeck=bml*,
without
> the
> > > >> > *modified.dat ending), the modified files (with the
*modified.dat
> > > >> ending)
> > > >> > and the config file into my ftp directory (*cd
> > > >> > incoming/irap/met_help/maccracken). *Let me know what you
find.
> > > >> >
> > > >> > Thanks for your help!
> > > >> >
> > > >> > Roz
> > > >> >
> > > >> > On Mon, Nov 26, 2018 at 5:22 PM John Halley Gotway via RT <
> > > >> > met_help at ucar.edu>
> > > >> > wrote:
> > > >> >
> > > >> > > Roz,
> > > >> > >
> > > >> > > Yes, the first 8 columns are required. The additional
ones are
> > > >> optional.
> > > >> > >
> > > >> > > Looking at your log output, I see that tc_pairs is
skipping over
> > all
> > > >> 642
> > > >> > > input lines. And the question is why? If you send me
your
> latest
> > > >> ADeck,
> > > >> > > BDeck, and TC-Pairs config files, I could run it here and
try to
> > > >> figure
> > > >> > out
> > > >> > > why it isn't defining any ADECK tracks.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > John
> > > >> > >
> > > >> > > On Mon, Nov 26, 2018 at 12:23 PM Rosalyn MacCracken -
NOAA
> > Affiliate
> > > >> via
> > > >> > RT
> > > >> > > <met_help at ucar.edu> wrote:
> > > >> > >
> > > >> > > >
> > > >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
> >
> > > >> > > >
> > > >> > > > So, can I just have 8 columns for the input file, or,
do I
> need
> > to
> > > >> make
> > > >> > > > columns of -9999 or NaN for all the rest of the
columns?
> > > >> > > >
> > > >> > > > I experimented, and just wrote out 10 columns for the
input
> > file:
> > > >> > > > agfso_trackdf_mod =
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
agfso_trackdf[['basin_name','storm_id','model_start_date','const','model','fcst_lead_time','vort_lat_cent','vort_lon_cent','vort_10m_max',
> > > >> > > > 'vort_min_mslp','type','thresh_wspd']]
> > > >> > > >
> > > >> > > > because I wanted stats for these variables, and I got
the
> output
> > > >> below.
> > > >> > > > So, maybe it's working, but, the config file needs to
be
> > > different?
> > > >> > Any
> > > >> > > > other ideas of what I should try doing/looking at?
> > > >> > > >
> > > >> > > > Roz
> > > >> > > >
> > > >> > > > Output:
> > > >> > > > DEBUG 1: [Source 1 of 1] ADECK Source:
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018083118.gfso.0053_modified.dat,
> > > >> > > > Model Suffix: _EXP
> > > >> > > > DEBUG 1: [Source 1 of 1] BDECK Source:
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018083118.gfso.0053_modified.dat,
> > > >> > > > Model Suffix: (nul)
> > > >> > > > DEBUG 1: Config File Default:
> > > >> > > > /opt/MET7/share/met/config/TCPairsConfig_default
> > > >> > > > DEBUG 1: Config File User:
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > > >> > > > DEBUG 1: Distance to land file:
/opt/MET7/share/met/tc_data/
> > > >> > > > dland_global_tenth_degree.nc
> > > >> > > > DEBUG 4:
> > > >> > > > DEBUG 4: Latitude/Longitude Grid Data:
> > > >> > > > DEBUG 4: lat_ll: -90
> > > >> > > > DEBUG 4: lon_ll: 180
> > > >> > > > DEBUG 4: delta_lat: 0.1
> > > >> > > > DEBUG 4: delta_lon: 0.1
> > > >> > > > DEBUG 4: Nlat: 1801
> > > >> > > > DEBUG 4: Nlon: 3601
> > > >> > > > DEBUG 4:
> > > >> > > > DEBUG 2: Processing 1 BDECK file(s).
> > > >> > > > DEBUG 4: [File 1 of 1] Used 30 of 30 lines read from
file
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018083118.gfso.0053_modified.dat"
> > > >> > > > DEBUG 3: Used 30 of 30 lines read from 1 file(s).
> > > >> > > > DEBUG 3: Identified 1 track(s).
> > > >> > > > DEBUG 4: [Track 1 of 1] TrackInfo: StormId =
"ML0800532018",
> > > IsBest
> > > >> =
> > > >> > > > false, IsOper = false, CheckAnly = true, IsAnly = true,
Basin
> =
> > > >> "ML",
> > > >> > > > Cyclone = "080053", StormName = "", TechniqueNumber =
3,
> > > Technique =
> > > >> > > > "GFSO", Initials = "", InitTime = NA, MinValidTime =
> > > >> 20180805_060000,
> > > >> > > > MaxValidTime = 20180812_120000, NPoints = 30, NAlloc =
100,
> > > >> NTrackLines
> > > >> > > = 0
> > > >> > > > DEBUG 2: Found 1 BDECK track(s).
> > > >> > > > DEBUG 2: Processing 1 ADECK file(s).
> > > >> > > > DEBUG 4: [File 1 of 1] Used 0 of 642 lines read from
file
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018083118.gfso.0053_modified.dat"
> > > >> > > > DEBUG 3: Used 0 of 642 lines read from 1 file(s).
> > > >> > > > DEBUG 3: Identified 0 track(s).
> > > >> > > > DEBUG 2: Deriving 12-hour interpolated ADECK tracks.
> > > >> > > > DEBUG 2: Finished adding 0 and replacing 0 Interp12
track(s).
> > > >> > > > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > > >> > > > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > > >> > > > DEBUG 2: Deriving 0 ADECK lag model(s).
> > > >> > > > DEBUG 2: Added 0 ADECK lag tracks(s).
> > > >> > > > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> > > >> > > > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> > > >> > > > DEBUG 2: Filtering 0 ADECK tracks based on config file
> settings.
> > > >> > > > DEBUG 3: Total tracks read = 0
> > > >> > > > DEBUG 3: Total tracks kept = 0
> > > >> > > > DEBUG 3: Rejected for storm name = 0
> > > >> > > > DEBUG 3: Rejected for valid time = 0
> > > >> > > > DEBUG 3: Rejected for required lead times = 0
> > > >> > > > DEBUG 3: Rejected for init mask = 0
> > > >> > > > DEBUG 3: Rejected for valid mask = 0
> > > >> > > > DEBUG 2: Matching 0 ADECK tracks to 1 BDECK tracks.
> > > >> > > > DEBUG 1: Watch/Warning file:
> > > >> /opt/MET7/share/met/tc_data/wwpts_us.txt
> > > >> > > > DEBUG 1: Output file:
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018083118.gfso.0053_test_modified_output.txt.tcst
> > > >> > > >
> > > >> > > >
> > > >> > > > On Mon, Nov 26, 2018 at 2:06 PM John Halley Gotway via
RT <
> > > >> > > > met_help at ucar.edu>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Roz,
> > > >> > > > >
> > > >> > > > > Page 334 of the user's guide (
> > > >> > > > >
> > > >> > >
> > > >>
> > >
>
https://dtcenter.org/met/users/docs/users_guide/MET_Users_Guide_v8.0.pdf
> > > >> > > > )
> > > >> > > > > mentions that only the first 8 ATCF columns are
required.
> If
> > > >> > > additional
> > > >> > > > > columns are present in the input ATCF files, then
tc_pairs
> > will
> > > >> > > interpret
> > > >> > > > > them as specified by the ATCF format.
> > > >> > > > >
> > > >> > > > > So if the storm name column is present, MET in
interpret
> those
> > > >> values
> > > >> > > as
> > > >> > > > > the storm name.
> > > >> > > > >
> > > >> > > > > For Guang Ping's data, probably best to have that
column
> > empty.
> > > >> > > > >
> > > >> > > > > John
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > On Mon, Nov 26, 2018 at 11:51 AM Rosalyn MacCracken -
NOAA
> > > >> Affiliate
> > > >> > > via
> > > >> > > > RT
> > > >> > > > > <met_help at ucar.edu> wrote:
> > > >> > > > >
> > > >> > > > > >
> > > >> > > > > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
> > > >
> > > >> > > > > >
> > > >> > > > > > Hi John,
> > > >> > > > > >
> > > >> > > > > > Ok, I will ignore the suffix error. I do have
numeric
> > values
> > > in
> > > >> > > > > > STORMNAME. I wonder if in the modified files I
have, this
> > > >> column
> > > >> > is
> > > >> > > > not
> > > >> > > > > > actually STORMNAME, but, some other variable. Now,
I
> don't
> > > have
> > > >> > the
> > > >> > > > > number
> > > >> > > > > > of columns that the NRL website has. Will that be
a
> > problem?
> > > >> > > Should I
> > > >> > > > > > create dummy columns?
> > > >> > > > > >
> > > >> > > > > > Roz
> > > >> > > > > >
> > > >> > > > > > On Mon, Nov 26, 2018 at 1:14 PM John Halley Gotway
via RT
> <
> > > >> > > > > > met_help at ucar.edu>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Roz,
> > > >> > > > > > >
> > > >> > > > > > > I see that you're wondering about this log
message:
> > > >> > > > > > > *Model Suffix: (nul)*
> > > >> > > > > > >
> > > >> > > > > > > Everything is fine, you do not need to be
concerned
> about
> > > >> this.
> > > >> > > > > Here's a
> > > >> > > > > > > line taken from the usage statement for the
tc_pairs
> tool:
> > > >> > > > > > > *Note: The "-adeck", "-edeck", and "-bdeck"
options may
> > > >> include
> > > >> > > > > > > "suffix=string" to modify the model names from
that
> > source.*
> > > >> > > > > > >
> > > >> > > > > > > Since you didn't specify "suffix=string" on the
command
> > > line,
> > > >> the
> > > >> > > > > suffix
> > > >> > > > > > > remains the default value of an empty string,
i.e.
> > "(nul)".
> > > >> So
> > > >> > why
> > > >> > > > > would
> > > >> > > > > > > you ever want to use a suffix? Sometimes data is
messy.
> > We
> > > >> were
> > > >> > > > > running
> > > >> > > > > > > tc_pairs to evaluate track data from multiple
sources
> and
> > we
> > > >> > found
> > > >> > > > that
> > > >> > > > > > > some model names we reused in both sources.
Let's say
> the
> > > >> OFCL
> > > >> > > > > forecast
> > > >> > > > > > > showed up in both stream 1.0 and stream 1.5 data.
> > > Specifying
> > > >> a
> > > >> > > > suffix
> > > >> > > > > > such
> > > >> > > > > > > as "suffix=_STREAM1.5" changes the model name
from
> "OFCL"
> > to
> > > >> > > > > > > "OFCL_STREAM1.5". And that enables us to
distinguish
> > > between
> > > >> > those
> > > >> > > > > > tracks
> > > >> > > > > > > in the output.
> > > >> > > > > > >
> > > >> > > > > > > But uou probably don't have any need for this
option in
> > your
> > > >> > > current
> > > >> > > > > > work.
> > > >> > > > > > >
> > > >> > > > > > > However, I am concerned about these log messages:
> > > >> > > > > > > DEBUG 4: Updating storm name from "-287" to "-
305" for
> > > >> > > ML0900022018.
> > > >> > > > > > > DEBUG 4: Updating storm name from "-305" to "-
255" for
> > > >> > > ML0900022018.
> > > >> > > > > > > DEBUG 4: Updating storm name from "-255" to "-
203" for
> > > >> > > ML0900022018.
> > > >> > > > > > > DEBUG 4: Updating storm name from "-203" to "-
205" for
> > > >> > > ML0900022018.
> > > >> > > > > > > DEBUG 4: Updating storm name from "-205" to "-
217" for
> > > >> > > ML0900022018.
> > > >> > > > > > > DEBUG 4: Updating storm name from "-217" to "-
211" for
> > > >> > > ML0900022018.
> > > >> > > > > > > DEBUG 4: Updating storm name from "-211" to "-
182" for
> > > >> > > ML0900022018.
> > > >> > > > > > > DEBUG 4: Updating storm name from "-182" to "-
138" for
> > > >> > > ML0900022018.
> > > >> > > > > > >
> > > >> > > > > > > You likely have numeric values showing up in the
> > "STORMNAME"
> > > >> ATCF
> > > >> > > > > column:
> > > >> > > > > > >
> > > >> > >
> https://www.nrlmry.navy.mil/atcf_web/docs/database/new/abdeck.txt
> > > >> > > > > > >
> > > >> > > > > > > I'd recommend looking into that.
> > > >> > > > > > >
> > > >> > > > > > > Thanks,
> > > >> > > > > > > John
> > > >> > > > > > >
> > > >> > > > > > > On Mon, Nov 26, 2018 at 8:44 AM Rosalyn
MacCracken -
> NOAA
> > > >> > Affiliate
> > > >> > > > via
> > > >> > > > > > RT
> > > >> > > > > > > <
> > > >> > > > > > > met_help at ucar.edu> wrote:
> > > >> > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > <URL:
> > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
> > > >> > >
> > > >> > > > > > > >
> > > >> > > > > > > > Hi John,
> > > >> > > > > > > >
> > > >> > > > > > > > I figured out an easy way to read in and write
out
> Guang
> > > >> Ping's
> > > >> > > > > > modified
> > > >> > > > > > > > ATCF file, so that it:
> > > >> > > > > > > > 1) Removes the 3rd column
> > > >> > > > > > > > 2) Replaces -99 with -9999 in column 21 and 24
(after
> > the
> > > >> 3rd
> > > >> > > > column
> > > >> > > > > > is
> > > >> > > > > > > > removed)
> > > >> > > > > > > > 3) Updates column 2 to include month+cyclone
number.
> > > (You
> > > >> had
> > > >> > > > said
> > > >> > > > > to
> > > >> > > > > > > do
> > > >> > > > > > > > year+cyclone #, but the tc_pairs_wrapper.py
said to do
> > > >> > > > > month+cyclone#)
> > > >> > > > > > > >
> > > >> > > > > > > > I changed Adeck and Bdeck.
> > > >> > > > > > > > Just a couple lines of my file:
> > > >> > > > > > > >
> > > >> > > > > > > > ML, 090002, 2018090100, 03, GFSO, 000, 511S,
897W,
> 40,
> > > >> 976,
> > > >> > > XX,
> > > >> > > > > 34,
> > > >> > > > > > > > NEQ, 0625, 0630, 0000, 0226, 978, 68, -9999,
123,
> > 155,
> > > >> > -9999,
> > > >> > > > > -9999,
> > > >> > > > > > > > -9999, -297, -394, -287, -458, 0073
> > > >> > > > > > > > ML, 090002, 2018090106, 03, GFSO, 000, 537S,
854W,
> 44,
> > > >> 964,
> > > >> > > XX,
> > > >> > > > > 34,
> > > >> > > > > > > > NEQ, 0640, 0645, 0000, 0297, 969, 108, -9999,
119,
> > 147,
> > > >> > -9999,
> > > >> > > > > -9999,
> > > >> > > > > > > > -9999, -403, -648, -305, -394, 0055
> > > >> > > > > > > >
> > > >> > > > > > > > So, I've run tc_pairs again and my output is
below. I
> > > don't
> > > >> > > > > understand
> > > >> > > > > > > the
> > > >> > > > > > > > Model Suffix: (nul). Is the input format still
wrong?
> > > >> > > > > > > >
> > > >> > > > > > > > Roz
> > > >> > > > > > > >
> > > >> > > > > > > > Output:
> > > >> > > > > > > > /opt/MET7/bin/tc_pairs -adeck
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat
> > > >> > > > > > > > -bdeck
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat
> > > >> > > > > > > > -config
> > > >> > > > > >
> > > ~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > > >> > > > > > > > -out
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt
> > > >> > > > > > > > -log
> > > >> > > > > > >
> > > >> > >
> > >
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> > > >> > > > > > > > -v 4
> > > >> > > > > > > > DEBUG 1: [Source 1 of 1] ADECK Source:
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat,
> > > >> > > > > > > > Model Suffix: (nul)
> > > >> > > > > > > > DEBUG 1: [Source 1 of 1] BDECK Source:
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat,
> > > >> > > > > > > > Model Suffix: (nul)
> > > >> > > > > > > > DEBUG 1: Config File Default:
> > > >> > > > > > > >
/opt/MET7/share/met/config/TCPairsConfig_default
> > > >> > > > > > > > DEBUG 1: Config File User:
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_ML
> > > >> > > > > > > > DEBUG 1: Distance to land file:
> > > /opt/MET7/share/met/tc_data/
> > > >> > > > > > > > dland_global_tenth_degree.nc
> > > >> > > > > > > > DEBUG 4:
> > > >> > > > > > > > DEBUG 4: Latitude/Longitude Grid Data:
> > > >> > > > > > > > DEBUG 4: lat_ll: -90
> > > >> > > > > > > > DEBUG 4: lon_ll: 180
> > > >> > > > > > > > DEBUG 4: delta_lat: 0.1
> > > >> > > > > > > > DEBUG 4: delta_lon: 0.1
> > > >> > > > > > > > DEBUG 4: Nlat: 1801
> > > >> > > > > > > > DEBUG 4: Nlon: 3601
> > > >> > > > > > > > DEBUG 4:
> > > >> > > > > > > > DEBUG 2: Processing 1 BDECK file(s).
> > > >> > > > > > > > DEBUG 4: Updating storm name from "-287" to "-
305" for
> > > >> > > > ML0900022018.
> > > >> > > > > > > > DEBUG 4: Updating storm name from "-305" to "-
255" for
> > > >> > > > ML0900022018.
> > > >> > > > > > > > DEBUG 4: Updating storm name from "-255" to "-
203" for
> > > >> > > > ML0900022018.
> > > >> > > > > > > > DEBUG 4: Updating storm name from "-203" to "-
205" for
> > > >> > > > ML0900022018.
> > > >> > > > > > > > DEBUG 4: Updating storm name from "-205" to "-
217" for
> > > >> > > > ML0900022018.
> > > >> > > > > > > > DEBUG 4: Updating storm name from "-217" to "-
211" for
> > > >> > > > ML0900022018.
> > > >> > > > > > > > DEBUG 4: Updating storm name from "-211" to "-
182" for
> > > >> > > > ML0900022018.
> > > >> > > > > > > > DEBUG 4: Updating storm name from "-182" to "-
138" for
> > > >> > > > ML0900022018.
> > > >> > > > > > > > DEBUG 4: [File 1 of 1] Used 9 of 9 lines read
from
> file
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/bmlq2018093018.gfso.0002_modified.dat"
> > > >> > > > > > > > DEBUG 3: Used 9 of 9 lines read from 1 file(s).
> > > >> > > > > > > > DEBUG 3: Identified 1 track(s).
> > > >> > > > > > > > DEBUG 4: [Track 1 of 1] TrackInfo: StormId =
> > > "ML0900022018",
> > > >> > > > IsBest =
> > > >> > > > > > > > false, IsOper = false, CheckAnly = true, IsAnly
=
> true,
> > > >> Basin =
> > > >> > > > "ML",
> > > >> > > > > > > > Cyclone = "090002", StormName = "-138",
> TechniqueNumber
> > =
> > > 3,
> > > >> > > > > Technique
> > > >> > > > > > =
> > > >> > > > > > > > "GFSO", Initials = "-9999", InitTime = NA,
> MinValidTime
> > =
> > > >> > > > > > > 20180901_000000,
> > > >> > > > > > > > MaxValidTime = 20180903_000000, NPoints = 9,
NAlloc =
> > 100,
> > > >> > > > > NTrackLines
> > > >> > > > > > =
> > > >> > > > > > > 0
> > > >> > > > > > > > DEBUG 2: Found 1 BDECK track(s).
> > > >> > > > > > > > DEBUG 2: Processing 1 ADECK file(s).
> > > >> > > > > > > > DEBUG 4: [File 1 of 1] Used 0 of 21 lines read
from
> file
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
"/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/modified_files_tmp/amlq2018093018.gfso.0002_modified.dat"
> > > >> > > > > > > > DEBUG 3: Used 0 of 21 lines read from 1
file(s).
> > > >> > > > > > > > DEBUG 3: Identified 0 track(s).
> > > >> > > > > > > > DEBUG 2: Deriving 12-hour interpolated ADECK
tracks.
> > > >> > > > > > > > DEBUG 2: Finished adding 0 and replacing 0
Interp12
> > > >> track(s).
> > > >> > > > > > > > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > > >> > > > > > > > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > > >> > > > > > > > DEBUG 2: Deriving 0 ADECK lag model(s).
> > > >> > > > > > > > DEBUG 2: Added 0 ADECK lag tracks(s).
> > > >> > > > > > > > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline
model(s).
> > > >> > > > > > > > DEBUG 2: Added 0 CLIPER/SHIFOR baseline
track(s).
> > > >> > > > > > > > DEBUG 2: Filtering 0 ADECK tracks based on
config file
> > > >> > settings.
> > > >> > > > > > > > DEBUG 3: Total tracks read = 0
> > > >> > > > > > > > DEBUG 3: Total tracks kept = 0
> > > >> > > > > > > > DEBUG 3: Rejected for storm name = 0
> > > >> > > > > > > > DEBUG 3: Rejected for valid time = 0
> > > >> > > > > > > > DEBUG 3: Rejected for required lead times = 0
> > > >> > > > > > > > DEBUG 3: Rejected for init mask = 0
> > > >> > > > > > > > DEBUG 3: Rejected for valid mask = 0
> > > >> > > > > > > > DEBUG 2: Matching 0 ADECK tracks to 1 BDECK
tracks.
> > > >> > > > > > > > DEBUG 1: Watch/Warning file:
> > > >> > > > /opt/MET7/share/met/tc_data/wwpts_us.txt
> > > >> > > > > > > > DEBUG 1: Output file:
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0002_test_modified_output.txt.tcst
> > > >> > > > > > > >
> > > >> > > > > > > > On Wed, Nov 21, 2018 at 9:16 AM Rosalyn
MacCracken -
> > NOAA
> > > >> > > > Affiliate <
> > > >> > > > > > > > rosalyn.maccracken at noaa.gov> wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > Hi John,
> > > >> > > > > > > > >
> > > >> > > > > > > > > Yes, that makes sense, and yes, I am using
Guang
> > Pings'
> > > >> > > modified
> > > >> > > > > > files.
> > > >> > > > > > > > > I'm not sure how to run tc_pairs_wrapper.py,
though.
> > I
> > > >> > think I
> > > >> > > > > need
> > > >> > > > > > > some
> > > >> > > > > > > > > type of *.conf file to pass in the files I
need, or
> > > >> something
> > > >> > > > like
> > > >> > > > > > > that.
> > > >> > > > > > > > > I'm in the process of reading through the
code
> now. I
> > > >> wonder
> > > >> > > if
> > > >> > > > I
> > > >> > > > > > > could
> > > >> > > > > > > > > just pull out the function process_non_atcf,
to run
> > in a
> > > >> > > > > stand-alone
> > > >> > > > > > > > mode.
> > > >> > > > > > > > > I wish there was some sort of documentation
with
> this
> > > >> code...
> > > >> > > > > > > > >
> > > >> > > > > > > > > And, so, once I can read these files, I can
use
> > tc_pairs
> > > >> to
> > > >> > > > create
> > > >> > > > > > the
> > > >> > > > > > > > > forecast error statistics, and then use
tc_stats to
> > > >> aggregate
> > > >> > > the
> > > >> > > > > > files
> > > >> > > > > > > > and
> > > >> > > > > > > > > get the mean, etc, for a period of time,
correct?
> Was
> > > >> there
> > > >> > > some
> > > >> > > > > > sort
> > > >> > > > > > > of
> > > >> > > > > > > > > plotting scripts for the output files?
> > > >> > > > > > > > >
> > > >> > > > > > > > > So, with tc_pairs, I can define a region,
like I did
> > > with
> > > >> > > > > point_stat,
> > > >> > > > > > > > > correct? Can I use the same *.nc files, or
do I use
> > the
> > > >> > > original
> > > >> > > > > > > *.poly
> > > >> > > > > > > > > files?
> > > >> > > > > > > > >
> > > >> > > > > > > > > Roz
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Tue, Nov 20, 2018 at 7:53 PM John Halley
Gotway
> via
> > > RT
> > > >> <
> > > >> > > > > > > > > met_help at ucar.edu> wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > >> Hi Roz,
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> Yes, I see you’re using the modified ATCF
format
> that
> > > >> Guang
> > > >> > > Ping
> > > >> > > > > > uses.
> > > >> > > > > > > > >> You’re correct that tc_pairs doesn’t parse
this
> > > directly
> > > >> > since
> > > >> > > > it
> > > >> > > > > > > > differs
> > > >> > > > > > > > >> slightly from the ATCF format.
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> When we’ve processed this data in the past,
we
> first
> > > >> > > > pre-processed
> > > >> > > > > > it
> > > >> > > > > > > > in a
> > > >> > > > > > > > >> shell script to:
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> 1. Replace the -99 value with -9999 for
missing
> data.
> > > >> > > > > > > > >> 2. Remove the 3rd column
> > > >> > > > > > > > >> 3. Updating the 2nd column to include the
year and
> > > >> cyclone
> > > >> > > > number.
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> And that enable tc_pairs to read it.
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> This is actually Done in python by one of
the use
> > cases
> > > >> in
> > > >> > > > > METplus.
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> Take a look at this file:
> > > >> > > > > > > > >>
> > > >> > > > >
> > > >>
https://github.com/NCAR/METplus/blob/master/ush/tc_pairs_wrapper.py
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> At the function named:
> > > >> > > > > > > > >> process_non_atcf
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> I haven’t run this using METplus scripts
myself,
> but
> > > all
> > > >> the
> > > >> > > > > pieces
> > > >> > > > > > of
> > > >> > > > > > > > >> logic are there, and it should be possible.
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> Make sense?
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> Thanks
> > > >> > > > > > > > >> John
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> On Tue, Nov 20, 2018 at 1:13 PM Rosalyn
MacCracken
> -
> > > NOAA
> > > >> > > > > Affiliate
> > > >> > > > > > > via
> > > >> > > > > > > > >> RT <
> > > >> > > > > > > > >> met_help at ucar.edu> wrote:
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > Tue Nov 20 13:13:05 2018: Request 87885
was acted
> > > upon.
> > > >> > > > > > > > >> > Transaction: Ticket created by
> > > >> > rosalyn.maccracken at noaa.gov
> > > >> > > > > > > > >> > Queue: met_help
> > > >> > > > > > > > >> > Subject: Help with using TC_pairs
> > > >> > > > > > > > >> > Owner: Nobody
> > > >> > > > > > > > >> > Requestors: rosalyn.maccracken at noaa.gov
> > > >> > > > > > > > >> > Status: new
> > > >> > > > > > > > >> > Ticket <URL:
> > > >> > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=87885
> > > >> > > > > > > > >
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > Hi,
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > I'm trying to use TC_pairs for extra-
tropical
> > > cyclones
> > > >> > > > > (basin=ML).
> > > >> > > > > > > I
> > > >> > > > > > > > >> think
> > > >> > > > > > > > >> > the format of the ATCF data is the issue,
but,
> I'm
> > > not
> > > >> > sure.
> > > >> > > > I
> > > >> > > > > > have
> > > >> > > > > > > > an
> > > >> > > > > > > > >> > Adeck and a Bdeck file, where a sample of
one of
> > the
> > > >> lines
> > > >> > > is:
> > > >> > > > > > > > >> > ML, 0003, 2018090100_F000_687N_0159W_FOF,
> > 2018090100,
> > > >> 03,
> > > >> > > > GFSO,
> > > >> > > > > > 000,
> > > >> > > > > > > > >> 687N,
> > > >> > > > > > > > >> > 159W, 32, 987, XX, 34, NEQ, 0301, 0338,
0000,
> > > 0000,
> > > >> > 988,
> > > >> > > > > 52,
> > > >> > > > > > > > -99,
> > > >> > > > > > > > >> > 330, 86, -99, -9999, -9999, 175, 202,
137,
> > 135,
> > > >> 0081
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > This is slightly different from the NRL
ATCF
> format
> > > on
> > > >> > their
> > > >> > > > > > > website,
> > > >> > > > > > > > >> such
> > > >> > > > > > > > >> > as the second column. Anyway, I
configured the
> > > config
> > > >> > file
> > > >> > > > very
> > > >> > > > > > > > simply,
> > > >> > > > > > > > >> > (model=GFS, basin=ML), and ran tc_pairs,
and got
> > the
> > > >> > output
> > > >> > > to
> > > >> > > > > the
> > > >> > > > > > > > >> screen
> > > >> > > > > > > > >> > below.
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > Is there a way to work around the diferent
ATCF
> > > format?
> > > >> > > I've
> > > >> > > > > > > > attached a
> > > >> > > > > > > > >> > sample of my Adeck and Bdeck files.
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > Thanks,
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > Roz
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > Output to the screen:
> > > >> > > > > > > > >> > /opt/MET7/bin/tc_pairs -adeck
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >>
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003
> > > >> > > > > > > > >> > -bdeck
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >>
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003
> > > >> > > > > > > > >> > -config
> > > >> > > > > > > > >> >
> > > >> > > > > > > >
> > > >> > > > >
> > > >> >
> > >
~/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> > > >> > > > > > > > >> > -out
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >>
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
~/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt
> > > >> > > > > > > > >> > -log
> > > >> > > > > > > > >>
> > > >> > > > > >
> > > >> >
> >
~/data/met_verif/track_forecast_errors/met_proc/output/log_output.txt
> > > >> > > > > > > > >> > -v 3
> > > >> > > > > > > > >> > DEBUG 1: [Source 1 of 1] ADECK Source:
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >>
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/amlq2018093018.gfso.0003,
> > > >> > > > > > > > >> > Model Suffix: (nul)
> > > >> > > > > > > > >> > DEBUG 1: [Source 1 of 1] BDECK Source:
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >>
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/cj/bmlq2018093018.gfso.0003,
> > > >> > > > > > > > >> > Model Suffix: (nul)
> > > >> > > > > > > > >> > DEBUG 1: Config File Default:
> > > >> > > > > > > > >> >
/opt/MET7/share/met/config/TCPairsConfig_default
> > > >> > > > > > > > >> > DEBUG 1: Config File User:
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >>
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/TCPairsConfig_opc_basin
> > > >> > > > > > > > >> > DEBUG 1: Distance to land file:
> > > >> > /opt/MET7/share/met/tc_data/
> > > >> > > > > > > > >> > dland_global_tenth_degree.nc
> > > >> > > > > > > > >> > DEBUG 2: Processing 1 BDECK file(s).
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING: int parse_lat(const char *) ->
bad
> > latitude
> > > >> ...
> > > >> > > "000"
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING: int parse_lon(const char *) ->
bad
> > longitude
> > > >> ...
> > > >> > > > "687N"
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING: int parse_lat(const char *) ->
bad
> > latitude
> > > >> ...
> > > >> > > "000"
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING: int parse_lon(const char *) ->
bad
> > longitude
> > > >> ...
> > > >> > > > "701N"
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING: int parse_lat(const char *) ->
bad
> > latitude
> > > >> ...
> > > >> > > "000"
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING: int parse_lon(const char *) ->
bad
> > longitude
> > > >> ...
> > > >> > > > "711N"
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING: int parse_lat(const char *) ->
bad
> > latitude
> > > >> ...
> > > >> > > "000"
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING: int parse_lon(const char *) ->
bad
> > longitude
> > > >> ...
> > > >> > > > "720N"
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING: int parse_lat(const char *) ->
bad
> > latitude
> > > >> ...
> > > >> > > "000"
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING: int parse_lon(const char *) ->
bad
> > longitude
> > > >> ...
> > > >> > > > "730N"
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING: int parse_lat(const char *) ->
bad
> > latitude
> > > >> ...
> > > >> > > "000"
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING: int parse_lon(const char *) ->
bad
> > longitude
> > > >> ...
> > > >> > > > "741N"
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING: int parse_lat(const char *) ->
bad
> > latitude
> > > >> ...
> > > >> > > "000"
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING: int parse_lon(const char *) ->
bad
> > longitude
> > > >> ...
> > > >> > > > "751N"
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING: int parse_lat(const char *) ->
bad
> > latitude
> > > >> ...
> > > >> > > "000"
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING: int parse_lon(const char *) ->
bad
> > longitude
> > > >> ...
> > > >> > > > "760N"
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING: int parse_lat(const char *) ->
bad
> > latitude
> > > >> ...
> > > >> > > "000"
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > WARNING: int parse_lon(const char *) ->
bad
> > longitude
> > > >> ...
> > > >> > > > "764N"
> > > >> > > > > > > > >> > WARNING:
> > > >> > > > > > > > >> > DEBUG 3: Used 9 of 9 lines read from 1
file(s).
> > > >> > > > > > > > >> > DEBUG 3: Identified 9 track(s).
> > > >> > > > > > > > >> > DEBUG 2: Found 9 BDECK track(s).
> > > >> > > > > > > > >> > DEBUG 2: Processing 1 ADECK file(s).
> > > >> > > > > > > > >> > DEBUG 3: Used 0 of 22 lines read from 1
file(s).
> > > >> > > > > > > > >> > DEBUG 3: Identified 0 track(s).
> > > >> > > > > > > > >> > DEBUG 2: Deriving 12-hour interpolated
ADECK
> > tracks.
> > > >> > > > > > > > >> > DEBUG 2: Finished adding 0 and replacing 0
> Interp12
> > > >> > > track(s).
> > > >> > > > > > > > >> > DEBUG 2: Deriving 0 ADECK consensus
model(s).
> > > >> > > > > > > > >> > DEBUG 2: Added 0 ADECK consensus
tracks(s).
> > > >> > > > > > > > >> > DEBUG 2: Deriving 0 ADECK lag model(s).
> > > >> > > > > > > > >> > DEBUG 2: Added 0 ADECK lag tracks(s).
> > > >> > > > > > > > >> > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline
> > model(s).
> > > >> > > > > > > > >> > DEBUG 2: Added 0 CLIPER/SHIFOR baseline
track(s).
> > > >> > > > > > > > >> > DEBUG 2: Filtering 0 ADECK tracks based on
config
> > > file
> > > >> > > > settings.
> > > >> > > > > > > > >> > DEBUG 3: Total tracks read
= 0
> > > >> > > > > > > > >> > DEBUG 3: Total tracks kept
= 0
> > > >> > > > > > > > >> > DEBUG 3: Rejected for storm name
= 0
> > > >> > > > > > > > >> > DEBUG 3: Rejected for valid time
= 0
> > > >> > > > > > > > >> > DEBUG 3: Rejected for required lead times
= 0
> > > >> > > > > > > > >> > DEBUG 3: Rejected for init mask
= 0
> > > >> > > > > > > > >> > DEBUG 3: Rejected for valid mask
= 0
> > > >> > > > > > > > >> > DEBUG 2: Matching 0 ADECK tracks to 9
BDECK
> tracks.
> > > >> > > > > > > > >> > DEBUG 1: Watch/Warning file:
> > > >> > > > > > > /opt/MET7/share/met/tc_data/wwpts_us.txt
> > > >> > > > > > > > >> > DEBUG 1: Output file:
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >>
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
/opc_test/home/opc_test/data/met_verif/track_forecast_errors/met_proc/output/2018093018.gfso.0003.output.txt.tcst
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > --
> > > >> > > > > > > > >> > Rosalyn MacCracken
> > > >> > > > > > > > >> > Support Scientist
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > Ocean Applications Branch
> > > >> > > > > > > > >> > NOAA/NWS Ocean Prediction Center
> > > >> > > > > > > > >> > NCWCP
> > > >> > > > > > > > >> > 5830 University Research Ct
> > > >> > > > > > > > >> > College Park, MD 20740-3818
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > (p) 301-683-1551
> > > >> > > > > > > > >> > rosalyn.maccracken at noaa.gov
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >>
> > > >> > > > > > > > >>
> > > >> > > > > > > > >
> > > >> > > > > > > > > --
> > > >> > > > > > > > > Rosalyn MacCracken
> > > >> > > > > > > > > Support Scientist
> > > >> > > > > > > > >
> > > >> > > > > > > > > Ocean Applications Branch
> > > >> > > > > > > > > NOAA/NWS Ocean Prediction Center
> > > >> > > > > > > > > NCWCP
> > > >> > > > > > > > > 5830 University Research Ct
> > > >> > > > > > > > > College Park, MD 20740-3818
> > > >> > > > > > > > >
> > > >> > > > > > > > > (p) 301-683-1551
> > > >> > > > > > > > > rosalyn.maccracken at noaa.gov
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > --
> > > >> > > > > > > > Rosalyn MacCracken
> > > >> > > > > > > > Support Scientist
> > > >> > > > > > > >
> > > >> > > > > > > > Ocean Applications Branch
> > > >> > > > > > > > NOAA/NWS Ocean Prediction Center
> > > >> > > > > > > > NCWCP
> > > >> > > > > > > > 5830 University Research Ct
> > > >> > > > > > > > College Park, MD 20740-3818
> > > >> > > > > > > >
> > > >> > > > > > > > (p) 301-683-1551
> > > >> > > > > > > > rosalyn.maccracken at noaa.gov
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > > > --
> > > >> > > > > > Rosalyn MacCracken
> > > >> > > > > > Support Scientist
> > > >> > > > > >
> > > >> > > > > > Ocean Applications Branch
> > > >> > > > > > NOAA/NWS Ocean Prediction Center
> > > >> > > > > > NCWCP
> > > >> > > > > > 5830 University Research Ct
> > > >> > > > > > College Park, MD 20740-3818
> > > >> > > > > >
> > > >> > > > > > (p) 301-683-1551
> > > >> > > > > > rosalyn.maccracken at noaa.gov
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > > > --
> > > >> > > > Rosalyn MacCracken
> > > >> > > > Support Scientist
> > > >> > > >
> > > >> > > > Ocean Applications Branch
> > > >> > > > NOAA/NWS Ocean Prediction Center
> > > >> > > > NCWCP
> > > >> > > > 5830 University Research Ct
> > > >> > > > College Park, MD 20740-3818
> > > >> > > >
> > > >> > > > (p) 301-683-1551
> > > >> > > > rosalyn.maccracken at noaa.gov
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> > --
> > > >> > Rosalyn MacCracken
> > > >> > Support Scientist
> > > >> >
> > > >> > Ocean Applications Branch
> > > >> > NOAA/NWS Ocean Prediction Center
> > > >> > NCWCP
> > > >> > 5830 University Research Ct
> > > >> > College Park, MD 20740-3818
> > > >> >
> > > >> > (p) 301-683-1551
> > > >> > rosalyn.maccracken at noaa.gov
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >
> > > > --
> > > > Rosalyn MacCracken
> > > > Support Scientist
> > > >
> > > > Ocean Applications Branch
> > > > NOAA/NWS Ocean Prediction Center
> > > > NCWCP
> > > > 5830 University Research Ct
> > > > College Park, MD 20740-3818
> > > >
> > > > (p) 301-683-1551
> > > > rosalyn.maccracken at noaa.gov
> > > >
> > >
> > >
> > > --
> > > Rosalyn MacCracken
> > > Support Scientist
> > >
> > > Ocean Applications Branch
> > > NOAA/NWS Ocean Prediction Center
> > > NCWCP
> > > 5830 University Research Ct
> > > College Park, MD 20740-3818
> > >
> > > (p) 301-683-1551
> > > rosalyn.maccracken at noaa.gov
> > >
> > >
> >
> >
>
> --
> Rosalyn MacCracken
> Support Scientist
>
> Ocean Applications Branch
> NOAA/NWS Ocean Prediction Center
> NCWCP
> 5830 University Research Ct
> College Park, MD 20740-3818
>
> (p) 301-683-1551
> rosalyn.maccracken at noaa.gov
>
>
------------------------------------------------
More information about the Met_help
mailing list