[Met_help] [rt.rap.ucar.edu #68793] History for Issues with init_beg and init_end namelist with tc_pairs in MetTC
John Halley Gotway via RT
met_help at ucar.edu
Fri Aug 29 09:28:58 MDT 2014
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello,
Using tc_pairs, I am trying to parse out three months (ASO) of data from a
merged a-deck and b-deck files for the year 2012 (to match our period of
study). If I leave init_beg and init_end blank, I get expected behavior
(matches all storms, even those outside my range (ex: MJJ)).
If I add
init_beg = "20120801";
init_end = "20121101";
to the config file to attempt to filter only ASO, running the tc_pairs
command provides 0 matching tracks (shell output below). It should have
matching tracks, since I have verified that running with empty strings for
both lines produces a file that contains matches between these dates
(amongst the others for those calendar years).
I have used:
20121101
20121101_00
20121101_000000
with the same results.
Also note the user guide is somewhat ambiguous here (showing an example of
YYYYMMDD but stating the field takes a YYYYMMDD_HH input).
Code was built on Yellowstone with ifort compiler. I have attached the
config file (as well as below results when running interactively in bash.
Please let me know if this is a simple miss on my part or if anything else
is needed. I am doing this analysis on Yellowstone/Geyser, so all the
scripts/files are readily available.
Relatedly, I need to do this analysis for multiple years. It seems that
init_beg and init_end are single strings (not arrays) so I anticipate that
I will have to run tc_pairs for each year in the data set (filtering by
ASO) and then catting the resultant pairs files together before doing
quantitative statistics. Is this the easiest way to accomplish this or is
there some ability to select multiple time subsets for filtering in
tc_pairs?
Cheers,
-Colin
----------- Shell output -----------------
-bash-4.1$ ~/METv4.1/bin/tc_pairs -adeck
~/work/TC_deckfiles/adeck/aal*fullwCAM.dat -bdeck
~/work/TC_deckfiles/bdeck/bal*full.dat -config config
DEBUG 1: [Source 1 of 2] ADECK Source:
/glade/u/home/zarzycki/work/TC_deckfiles/adeck/aal2012fullwCAM.dat, Model
Suffix: (nul)
DEBUG 1: [Source 2 of 2] ADECK Source:
/glade/u/home/zarzycki/work/TC_deckfiles/adeck/aal2013fullwCAM.dat, Model
Suffix: (nul)
DEBUG 1: [Source 1 of 2] BDECK Source:
/glade/u/home/zarzycki/work/TC_deckfiles/bdeck/bal2012full.dat, Model
Suffix: (nul)
DEBUG 1: [Source 2 of 2] BDECK Source:
/glade/u/home/zarzycki/work/TC_deckfiles/bdeck/bal2013full.dat, Model
Suffix: (nul)
DEBUG 1: Config File Default:
/glade/u/home/zarzycki/METv4.1/data/config/TCPairsConfig_default
DEBUG 1: Config File User: config
DEBUG 1: Distance to land file: /glade/u/home/zarzycki/METv4.1/data/tc_data/
dland_nw_hem_tenth_degree.nc
DEBUG 2: Processing 2 ADECK file(s).
DEBUG 2: Found 0 ADECK track(s).
DEBUG 2: Filtering 0 ADECK tracks based on config file settings.
DEBUG 2: Processing 2 BDECK file(s).
DEBUG 2: Found 62 BDECK track(s).
DEBUG 2: Deriving 12-hour interpolated tracks.
DEBUG 2: Finished deriving 0 12-hour interpolated tracks.
DEBUG 2: Deriving 0 ADECK consensus model(s).
DEBUG 2: Added 0 ADECK consensus tracks(s).
DEBUG 2: Deriving 0 ADECK lag model(s).
DEBUG 2: Added 0 ADECK lag tracks(s).
DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
DEBUG 2: Matching 0 ADECK tracks to 62 BDECK tracks.
DEBUG 1: Watch/Warning file:
/glade/u/home/zarzycki/METv4.1/data/tc_data/wwpts_us.txt
DEBUG 1: Output file: ./out_tcmpr.tcst
--
Colin M. Zarzycki, Postdoctoral fellow
University of Michigan, Dept. of Atmos., Ocean., and Space Sciences
2133 Space Research Building, 2455 Hayward St., Ann Arbor, MI 48109
http://www.colinzarzycki.com
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Issues with init_beg and init_end namelist with tc_pairs in MetTC
From: John Halley Gotway
Time: Thu Aug 28 14:59:44 2014
Colin,
Very good point. I ran a sample case through the debugger and
unfortunately we have the logic reversed on those init_beg and
init_end
settings! If you were to set init_beg to the end time and init_end to
the
being time, it would work as expected.
I've corrected this bug in the development version and it will be
included
in the next release - which is due out in a week or so.
I could easily post a METv4.1 bugfix for this issue if you'd like.
Just
let me if that'd be helpful.
Thanks for the heads up about the documentation - we've corrected that
for
the next release. Those date/times are specified in
YYYYMMDD[_HH[MMSS]]
where the brackets indicate that it's optional. So all of the
following
formats should work: 20140215, 20140215_00, 20140215_000000
And you're correct that TC-Pairs can only handle a single time window.
Yes,
you will need to run it once per year, but there's no need to cat
together
the output files. If you're using the TC-Stat tool to run analysis
jobs,
you can just pass it a "-lookin" directory and it'll parse all files
ending
in .tcst in that directory.
I suppose we could enhance the TC-Pairs logic to handle a set of time
windows. But practically speaking, that's probably not how you'd want
to
run the tool. It uses a lot of memory, and if you give it too much
data to
process. It'll slow down to a crawl. I suspect that running it with
multiple years would slow it down a lot - but it all depends on the
number
of tracks.
In the DTC, our ATCF files are organized by storm. We have one ATCF
file
containing all model tracks for a single storm. We run TC-Pairs once
per
storm and then use TC-Stat to filter down the data and perform
analysis
jobs.
I realize right now that you're only interest in MJO. But if you want
to
study other seasons as well, you could just run TC-Pairs on all the
data
and then do the filtering in TC-Stat. It all just depends on what
you're
studying and how you'd like to organize the data.
Hope that helps, please let me know about the patch, and please send
any
additional comments or questions about MET-TC that may come up.
Thanks,
John Halley Gotway
met_help at ucar.edu
On Thu, Aug 28, 2014 at 11:36 AM, Colin Zarzycki via RT
<met_help at ucar.edu>
wrote:
>
> Thu Aug 28 11:36:47 2014: Request 68793 was acted upon.
> Transaction: Ticket created by zarzycki at umich.edu
> Queue: met_help
> Subject: Issues with init_beg and init_end namelist with
tc_pairs in
> MetTC
> Owner: Nobody
> Requestors: zarzycki at umich.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68793 >
>
>
> Hello,
>
> Using tc_pairs, I am trying to parse out three months (ASO) of data
from a
> merged a-deck and b-deck files for the year 2012 (to match our
period of
> study). If I leave init_beg and init_end blank, I get expected
behavior
> (matches all storms, even those outside my range (ex: MJJ)).
>
> If I add
>
> init_beg = "20120801";
> init_end = "20121101";
>
> to the config file to attempt to filter only ASO, running the
tc_pairs
> command provides 0 matching tracks (shell output below). It should
have
> matching tracks, since I have verified that running with empty
strings for
> both lines produces a file that contains matches between these dates
> (amongst the others for those calendar years).
>
> I have used:
>
> 20121101
> 20121101_00
> 20121101_000000
>
> with the same results.
>
> Also note the user guide is somewhat ambiguous here (showing an
example of
> YYYYMMDD but stating the field takes a YYYYMMDD_HH input).
>
> Code was built on Yellowstone with ifort compiler. I have attached
the
> config file (as well as below results when running interactively in
bash.
>
> Please let me know if this is a simple miss on my part or if
anything else
> is needed. I am doing this analysis on Yellowstone/Geyser, so all
the
> scripts/files are readily available.
>
> Relatedly, I need to do this analysis for multiple years. It seems
that
> init_beg and init_end are single strings (not arrays) so I
anticipate that
> I will have to run tc_pairs for each year in the data set (filtering
by
> ASO) and then catting the resultant pairs files together before
doing
> quantitative statistics. Is this the easiest way to accomplish this
or is
> there some ability to select multiple time subsets for filtering in
> tc_pairs?
>
> Cheers,
>
> -Colin
>
> ----------- Shell output -----------------
>
> -bash-4.1$ ~/METv4.1/bin/tc_pairs -adeck
> ~/work/TC_deckfiles/adeck/aal*fullwCAM.dat -bdeck
> ~/work/TC_deckfiles/bdeck/bal*full.dat -config config
> DEBUG 1: [Source 1 of 2] ADECK Source:
> /glade/u/home/zarzycki/work/TC_deckfiles/adeck/aal2012fullwCAM.dat,
Model
> Suffix: (nul)
> DEBUG 1: [Source 2 of 2] ADECK Source:
> /glade/u/home/zarzycki/work/TC_deckfiles/adeck/aal2013fullwCAM.dat,
Model
> Suffix: (nul)
> DEBUG 1: [Source 1 of 2] BDECK Source:
> /glade/u/home/zarzycki/work/TC_deckfiles/bdeck/bal2012full.dat,
Model
> Suffix: (nul)
> DEBUG 1: [Source 2 of 2] BDECK Source:
> /glade/u/home/zarzycki/work/TC_deckfiles/bdeck/bal2013full.dat,
Model
> Suffix: (nul)
> DEBUG 1: Config File Default:
> /glade/u/home/zarzycki/METv4.1/data/config/TCPairsConfig_default
> DEBUG 1: Config File User: config
> DEBUG 1: Distance to land file:
> /glade/u/home/zarzycki/METv4.1/data/tc_data/
> dland_nw_hem_tenth_degree.nc
> DEBUG 2: Processing 2 ADECK file(s).
> DEBUG 2: Found 0 ADECK track(s).
> DEBUG 2: Filtering 0 ADECK tracks based on config file settings.
> DEBUG 2: Processing 2 BDECK file(s).
> DEBUG 2: Found 62 BDECK track(s).
> DEBUG 2: Deriving 12-hour interpolated tracks.
> DEBUG 2: Finished deriving 0 12-hour interpolated tracks.
> DEBUG 2: Deriving 0 ADECK consensus model(s).
> DEBUG 2: Added 0 ADECK consensus tracks(s).
> DEBUG 2: Deriving 0 ADECK lag model(s).
> DEBUG 2: Added 0 ADECK lag tracks(s).
> DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> DEBUG 2: Matching 0 ADECK tracks to 62 BDECK tracks.
> DEBUG 1: Watch/Warning file:
> /glade/u/home/zarzycki/METv4.1/data/tc_data/wwpts_us.txt
> DEBUG 1: Output file: ./out_tcmpr.tcst
>
>
> --
> Colin M. Zarzycki, Postdoctoral fellow
> University of Michigan, Dept. of Atmos., Ocean., and Space Sciences
> 2133 Space Research Building, 2455 Hayward St., Ann Arbor, MI 48109
> http://www.colinzarzycki.com
>
>
------------------------------------------------
Subject: Issues with init_beg and init_end namelist with tc_pairs in MetTC
From: Colin Zarzycki
Time: Thu Aug 28 16:43:00 2014
Hi John,
You are absolutely correct -- that appears to have fixed the problem.
I had
quickly looked at the source, but was more worried about actually
reading/converting the string than looking at the filtering logic, so
I
skimmed right over it. I'll just keep them reversed in the config file
for
now -- it's not a big deal.
Thanks for the clarification re: the filtering multiple subsets. I
agree
that it is not a critical concern. I have split the analysis between
some
NCL code I have and TC_stat which is why I prefer to do all the
filtering
in TC_pairs (I only have a finite # of model runs using our model, so
there
really is no reason for me to ever compare to years/times we didn't
analyze). However, it is useful to know that TC_stat can take multiple
tcst
files. I guess I should have realized that since I've already figured
that
tc_pair can do the same with multiple ATCF datasets.
Thanks again!
-Colin
On Thu, Aug 28, 2014 at 2:59 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Colin,
>
> Very good point. I ran a sample case through the debugger and
> unfortunately we have the logic reversed on those init_beg and
init_end
> settings! If you were to set init_beg to the end time and init_end
to the
> being time, it would work as expected.
>
> I've corrected this bug in the development version and it will be
included
> in the next release - which is due out in a week or so.
>
> I could easily post a METv4.1 bugfix for this issue if you'd like.
Just
> let me if that'd be helpful.
>
> Thanks for the heads up about the documentation - we've corrected
that for
> the next release. Those date/times are specified in
YYYYMMDD[_HH[MMSS]]
> where the brackets indicate that it's optional. So all of the
following
> formats should work: 20140215, 20140215_00, 20140215_000000
>
> And you're correct that TC-Pairs can only handle a single time
window. Yes,
> you will need to run it once per year, but there's no need to cat
together
> the output files. If you're using the TC-Stat tool to run analysis
jobs,
> you can just pass it a "-lookin" directory and it'll parse all files
ending
> in .tcst in that directory.
>
> I suppose we could enhance the TC-Pairs logic to handle a set of
time
> windows. But practically speaking, that's probably not how you'd
want to
> run the tool. It uses a lot of memory, and if you give it too much
data to
> process. It'll slow down to a crawl. I suspect that running it
with
> multiple years would slow it down a lot - but it all depends on the
number
> of tracks.
>
> In the DTC, our ATCF files are organized by storm. We have one ATCF
file
> containing all model tracks for a single storm. We run TC-Pairs
once per
> storm and then use TC-Stat to filter down the data and perform
analysis
> jobs.
>
> I realize right now that you're only interest in MJO. But if you
want to
> study other seasons as well, you could just run TC-Pairs on all the
data
> and then do the filtering in TC-Stat. It all just depends on what
you're
> studying and how you'd like to organize the data.
>
> Hope that helps, please let me know about the patch, and please send
any
> additional comments or questions about MET-TC that may come up.
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu
>
>
>
>
> On Thu, Aug 28, 2014 at 11:36 AM, Colin Zarzycki via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > Thu Aug 28 11:36:47 2014: Request 68793 was acted upon.
> > Transaction: Ticket created by zarzycki at umich.edu
> > Queue: met_help
> > Subject: Issues with init_beg and init_end namelist with
tc_pairs in
> > MetTC
> > Owner: Nobody
> > Requestors: zarzycki at umich.edu
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=68793 >
> >
> >
> > Hello,
> >
> > Using tc_pairs, I am trying to parse out three months (ASO) of
data from
> a
> > merged a-deck and b-deck files for the year 2012 (to match our
period of
> > study). If I leave init_beg and init_end blank, I get expected
behavior
> > (matches all storms, even those outside my range (ex: MJJ)).
> >
> > If I add
> >
> > init_beg = "20120801";
> > init_end = "20121101";
> >
> > to the config file to attempt to filter only ASO, running the
tc_pairs
> > command provides 0 matching tracks (shell output below). It should
have
> > matching tracks, since I have verified that running with empty
strings
> for
> > both lines produces a file that contains matches between these
dates
> > (amongst the others for those calendar years).
> >
> > I have used:
> >
> > 20121101
> > 20121101_00
> > 20121101_000000
> >
> > with the same results.
> >
> > Also note the user guide is somewhat ambiguous here (showing an
example
> of
> > YYYYMMDD but stating the field takes a YYYYMMDD_HH input).
> >
> > Code was built on Yellowstone with ifort compiler. I have attached
the
> > config file (as well as below results when running interactively
in bash.
> >
> > Please let me know if this is a simple miss on my part or if
anything
> else
> > is needed. I am doing this analysis on Yellowstone/Geyser, so all
the
> > scripts/files are readily available.
> >
> > Relatedly, I need to do this analysis for multiple years. It seems
that
> > init_beg and init_end are single strings (not arrays) so I
anticipate
> that
> > I will have to run tc_pairs for each year in the data set
(filtering by
> > ASO) and then catting the resultant pairs files together before
doing
> > quantitative statistics. Is this the easiest way to accomplish
this or is
> > there some ability to select multiple time subsets for filtering
in
> > tc_pairs?
> >
> > Cheers,
> >
> > -Colin
> >
> > ----------- Shell output -----------------
> >
> > -bash-4.1$ ~/METv4.1/bin/tc_pairs -adeck
> > ~/work/TC_deckfiles/adeck/aal*fullwCAM.dat -bdeck
> > ~/work/TC_deckfiles/bdeck/bal*full.dat -config config
> > DEBUG 1: [Source 1 of 2] ADECK Source:
> >
/glade/u/home/zarzycki/work/TC_deckfiles/adeck/aal2012fullwCAM.dat,
Model
> > Suffix: (nul)
> > DEBUG 1: [Source 2 of 2] ADECK Source:
> >
/glade/u/home/zarzycki/work/TC_deckfiles/adeck/aal2013fullwCAM.dat,
Model
> > Suffix: (nul)
> > DEBUG 1: [Source 1 of 2] BDECK Source:
> > /glade/u/home/zarzycki/work/TC_deckfiles/bdeck/bal2012full.dat,
Model
> > Suffix: (nul)
> > DEBUG 1: [Source 2 of 2] BDECK Source:
> > /glade/u/home/zarzycki/work/TC_deckfiles/bdeck/bal2013full.dat,
Model
> > Suffix: (nul)
> > DEBUG 1: Config File Default:
> > /glade/u/home/zarzycki/METv4.1/data/config/TCPairsConfig_default
> > DEBUG 1: Config File User: config
> > DEBUG 1: Distance to land file:
> > /glade/u/home/zarzycki/METv4.1/data/tc_data/
> > dland_nw_hem_tenth_degree.nc
> > DEBUG 2: Processing 2 ADECK file(s).
> > DEBUG 2: Found 0 ADECK track(s).
> > DEBUG 2: Filtering 0 ADECK tracks based on config file settings.
> > DEBUG 2: Processing 2 BDECK file(s).
> > DEBUG 2: Found 62 BDECK track(s).
> > DEBUG 2: Deriving 12-hour interpolated tracks.
> > DEBUG 2: Finished deriving 0 12-hour interpolated tracks.
> > DEBUG 2: Deriving 0 ADECK consensus model(s).
> > DEBUG 2: Added 0 ADECK consensus tracks(s).
> > DEBUG 2: Deriving 0 ADECK lag model(s).
> > DEBUG 2: Added 0 ADECK lag tracks(s).
> > DEBUG 2: Deriving 0 CLIPER/SHIFOR baseline model(s).
> > DEBUG 2: Added 0 CLIPER/SHIFOR baseline track(s).
> > DEBUG 2: Matching 0 ADECK tracks to 62 BDECK tracks.
> > DEBUG 1: Watch/Warning file:
> > /glade/u/home/zarzycki/METv4.1/data/tc_data/wwpts_us.txt
> > DEBUG 1: Output file: ./out_tcmpr.tcst
> >
> >
> > --
> > Colin M. Zarzycki, Postdoctoral fellow
> > University of Michigan, Dept. of Atmos., Ocean., and Space
Sciences
> > 2133 Space Research Building, 2455 Hayward St., Ann Arbor, MI
48109
> > http://www.colinzarzycki.com
> >
> >
>
>
--
Colin M. Zarzycki, Postdoctoral fellow
University of Michigan, Dept. of Atmos., Ocean., and Space Sciences
2133 Space Research Building, 2455 Hayward St., Ann Arbor, MI 48109
http://www.colinzarzycki.com
------------------------------------------------
More information about the Met_help
mailing list