[Met_help] [rt.rap.ucar.edu #100602] History for tcpairs
George McCabe via RT
met_help at ucar.edu
Wed Sep 22 16:30:06 MDT 2021
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi George,
When I tried to plot out the error stats using "TCMPRPlotter" and
"TCPairs", the time range is always at the first month period. For example,
if I set
INIT_BEG = 20200101
INIT_END = 20200131
and
VALID_BEG = 20200101
VALID_END = 20200131
Both "TCMPRPlotter" and "TCPairs" will generate correct pairs and error
stats for the period.
But when I change the time range to
INIT_BEG = 20200101
INIT_END = 20200331
and
VALID_BEG = 20200101
VALID_END = 20200331
It still generates the same results even though I expanded the time range.
I saw on the DTC github discussion board, Molly asked about similar issue
(#985):
https://github.com/dtcenter/METplus/discussions/985
In your reply, you seemed to have resolved this issue by adding a few lines
in the configuration file and changing looping. Do you happen to have it
installed on Hera already?
Currently, I am using your temporary build:
/home/George.Mccabe/scratch1_George.Mccabe/METplus.feature_926/ush/run_metplus.py
Hera is the directory:
/scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test_george2
Here is the command line:
/home/George.Mccabe/scratch1_George.Mccabe/METplus.feature_926/ush/run_metplus.py
-c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c Plotter_8.conf
Do you have an update to this build?
Thanks,
Guang Ping
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: tcpairs
From: John Halley Gotway
Time: Mon Jul 19 15:01:01 2021
Hi Guang Ping,
The command that you ran here:
/home/George.Mccabe/scratch1_George.Mccabe/METplus.feature_926/ush/run_metplus.py
-c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c Plotter_8.conf
only runs TCMPRPlotter. This is because PROCESS_LIST is defined in
both
Plotter_fcstGFS_obsGFS_RPlotting_8.conf and Plotter_8.conf. Since
Plotter_8.conf comes last, then the value for PROCESS_LIST from that
file
is used, which only has TCMPRPlotter listed. The plotter is reading
all of
the available files generated from TCPairs and since that process is
not
running again with the new time range, the data that is read only
covers
the first month.
We are planning on releasing METplus 4.1.0-beta1 this week. However,
if I
understand what you are trying to do here, it shouldn't require the
changes
described in Molly's issue. It looks like you want to run TCPairs once
and
filter the data based on the init and valid times. While this
configuration
should achieve what you are trying to do, the following configuration
is
more explicit to what you are actually running:
# run once using 20200101 to substitute the init time in your string
substitution, i.e. TC_PAIRS_ADECK_TEMPLATE =
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
INIT_BEG = 20200101
INIT_END = 20200101
# filter the init times of the lines read from the input files in the
TCPairs configuration file
TC_PAIRS_INIT_BEG = 20200101
TC_PAIRS_INIT_END = 20200331
# filter the valid times of the lines read from the input files in the
TCPairs configuration file
TC_PAIRS_VALID_BEG = 20200101
TC_PAIRS_VALID_END = 20200331
If any of TC_PAIRS_[INIT/VALID]_[BEG/END] are not set, then the
corresponding [INIT/VALID]_[BEG/END] values are used. This is done to
preserve backwards compatibility for old METplus configurations. The
above
settings are more explicit as to what is actually be configured.
You may want to consider either clearing your output directory or
creating
a new output directory for a new run (especially if the time range you
are
using has changed) to ensure that the files you are generated are from
that
run and not from a previous run. Doing this would have alerted you
that
TCPairs was not run in this case.
Let me know if you have any other issues getting this running.
FYI:
We switched from providing support through this email address to
providing
support through GitHub Discussions. In the future, you will need to
create
a free GitHub account if you don’t have one already and post your
questions
to the METplus Components Discussion Forum:
https://github.com/dtcenter/METplus/discussions
To post a new question, click on the green button in the upper right
corner
of the page labeled "New discussion" and select "Incoming" as the
category.
Thanks,
George
------------------------------------------------
Subject: tcpairs
From: Guang Lou - NOAA Affiliate
Time: Tue Jul 20 13:20:25 2021
Hi John,
Thanks for forwarding George's email answer to me. I created a test
case
that does the tcpairs only on Hera:
/scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test3
Command line:
run_metplus.py -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c
usecase_8.conf
where PROCESS_LIST = TCPairs
It generated the month of 202002, that is because I changed
INIT_BEG = 20200101
INIT_END = 20200331
to
INIT_BEG = 20200201
INIT_END = 20200331
If I use "INIT_BEG = 20200101,INIT_END = 20200331" it will generate
20200101 only.
The wild card does not work either for the model's place:
TC_PAIRS_ADECK_TEMPLATE =
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
I populated adeck and bdeck on
/scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/METplus/METplus-
4.0_sample_data/model_applications/medium_range/tracks_all
(each subdirectory contains all three months' decks.)
I was wondering if you could copy my directory,
/scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test3
and test:
1. It can generate multiple months instead of just one month and one
model.
2. The wild card for models in the DECK_TEMPATE for multiple models.
George, I'm downloading NetCDF model data to Hera:
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071300_nc.tar
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071306_nc.tar
Please let me know after you downloaded the files and I will have to
remove
them due to tight disk space.
Thanks,
Guang Ping
On Mon, Jul 19, 2021 at 5:01 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Hi Guang Ping,
>
> The command that you ran here:
>
>
>
/home/George.Mccabe/scratch1_George.Mccabe/METplus.feature_926/ush/run_metplus.py
> -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c Plotter_8.conf
>
> only runs TCMPRPlotter. This is because PROCESS_LIST is defined in
both
> Plotter_fcstGFS_obsGFS_RPlotting_8.conf and Plotter_8.conf. Since
> Plotter_8.conf comes last, then the value for PROCESS_LIST from that
file
> is used, which only has TCMPRPlotter listed. The plotter is reading
all of
> the available files generated from TCPairs and since that process is
not
> running again with the new time range, the data that is read only
covers
> the first month.
>
> We are planning on releasing METplus 4.1.0-beta1 this week. However,
if I
> understand what you are trying to do here, it shouldn't require the
changes
> described in Molly's issue. It looks like you want to run TCPairs
once and
> filter the data based on the init and valid times. While this
configuration
> should achieve what you are trying to do, the following
configuration is
> more explicit to what you are actually running:
>
> # run once using 20200101 to substitute the init time in your string
> substitution, i.e. TC_PAIRS_ADECK_TEMPLATE =
> {date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> INIT_BEG = 20200101
> INIT_END = 20200101
>
> # filter the init times of the lines read from the input files in
the
> TCPairs configuration file
> TC_PAIRS_INIT_BEG = 20200101
> TC_PAIRS_INIT_END = 20200331
>
> # filter the valid times of the lines read from the input files in
the
> TCPairs configuration file
> TC_PAIRS_VALID_BEG = 20200101
> TC_PAIRS_VALID_END = 20200331
>
> If any of TC_PAIRS_[INIT/VALID]_[BEG/END] are not set, then the
> corresponding [INIT/VALID]_[BEG/END] values are used. This is done
to
> preserve backwards compatibility for old METplus configurations. The
above
> settings are more explicit as to what is actually be configured.
>
> You may want to consider either clearing your output directory or
creating
> a new output directory for a new run (especially if the time range
you are
> using has changed) to ensure that the files you are generated are
from that
> run and not from a previous run. Doing this would have alerted you
that
> TCPairs was not run in this case.
>
> Let me know if you have any other issues getting this running.
>
> FYI:
> We switched from providing support through this email address to
providing
> support through GitHub Discussions. In the future, you will need to
create
> a free GitHub account if you don’t have one already and post your
questions
> to the METplus Components Discussion Forum:
> https://github.com/dtcenter/METplus/discussions
> To post a new question, click on the green button in the upper right
corner
> of the page labeled "New discussion" and select "Incoming" as the
category.
>
> Thanks,
> George
>
>
------------------------------------------------
Subject: tcpairs
From: George McCabe
Time: Tue Jul 20 15:21:42 2021
Hi Guang Ping,
Thank you for providing the NetCDF files. I am currently downloading
one of
the files but it is going very slowly.
In regards to your use case, the logic for TCPairs wrapper is a little
strange. It behaves differently based on the value of LOOP_BY. If
LOOP_BY =
processes, then it will run once using the INIT_BEG time as the run
time to
find input files. If LOOP_BY = times, then it will loop over each run
time
specified between INIT_BEG and INIT_END by INIT_INCREMENT and find
files to
process for each run time.
If you change your configuration file to include LOOP_BY = times, then
it
should look over each run time and process for each. It looks like
your
INIT_INCREMENT is set to 21600 or 6 hours, so it will run every 6
hours
from 20200201 to 2020331. If you want to run once per month, you will
need
to set INIT_INCREMENT = 1m so that it will increment by 1 month for
each
iteration.
The caveat is that TCMPRPlotter will only run if LOOP_BY = processes
and
will produce an error if LOOP_BY = times. This is because we typically
want
to process all of the data that is available for all run times, then
run
TCMPRPlotter so it can read all of the data. For now, to handle this
use
case you will have to call run_metplus.py 2 separate times, once
calling
TCPairs with LOOP_BY = times and once calling TCMPRPlotter with
LOOP_BY =
processes.
I am going to add a configuration option to TCPairs wrapper so that
you can
loop over the run times when LOOP_BY = processes is set if desired. I
will
try to get this change into METplus 4.1.0-beta1 so you can test the
change
and make sure it works as expected. I can let you know when that is
available.
Thanks,
George
On Tue, Jul 20, 2021 at 1:27 PM Guang Lou - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
> Hi John,
>
> Thanks for forwarding George's email answer to me. I created a test
case
> that does the tcpairs only on Hera:
> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test3
> Command line:
> run_metplus.py -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c
usecase_8.conf
> where PROCESS_LIST = TCPairs
>
> It generated the month of 202002, that is because I changed
> INIT_BEG = 20200101
> INIT_END = 20200331
> to
> INIT_BEG = 20200201
> INIT_END = 20200331
>
> If I use "INIT_BEG = 20200101,INIT_END = 20200331" it will generate
> 20200101 only.
>
> The wild card does not work either for the model's place:
> TC_PAIRS_ADECK_TEMPLATE =
> {date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> I populated adeck and bdeck on
>
> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/METplus/METplus-
4.0_sample_data/model_applications/medium_range/tracks_all
> (each subdirectory contains all three months' decks.)
>
> I was wondering if you could copy my directory,
> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test3
> and test:
> 1. It can generate multiple months instead of just one month and one
model.
> 2. The wild card for models in the DECK_TEMPATE for multiple models.
>
> George, I'm downloading NetCDF model data to Hera:
> /scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071300_nc.tar
> /scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071306_nc.tar
>
> Please let me know after you downloaded the files and I will have to
remove
> them due to tight disk space.
>
> Thanks,
> Guang Ping
>
> On Mon, Jul 19, 2021 at 5:01 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Hi Guang Ping,
> >
> > The command that you ran here:
> >
> >
> >
>
/home/George.Mccabe/scratch1_George.Mccabe/METplus.feature_926/ush/run_metplus.py
> > -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c Plotter_8.conf
> >
> > only runs TCMPRPlotter. This is because PROCESS_LIST is defined in
both
> > Plotter_fcstGFS_obsGFS_RPlotting_8.conf and Plotter_8.conf. Since
> > Plotter_8.conf comes last, then the value for PROCESS_LIST from
that file
> > is used, which only has TCMPRPlotter listed. The plotter is
reading all
> of
> > the available files generated from TCPairs and since that process
is not
> > running again with the new time range, the data that is read only
covers
> > the first month.
> >
> > We are planning on releasing METplus 4.1.0-beta1 this week.
However, if I
> > understand what you are trying to do here, it shouldn't require
the
> changes
> > described in Molly's issue. It looks like you want to run TCPairs
once
> and
> > filter the data based on the init and valid times. While this
> configuration
> > should achieve what you are trying to do, the following
configuration is
> > more explicit to what you are actually running:
> >
> > # run once using 20200101 to substitute the init time in your
string
> > substitution, i.e. TC_PAIRS_ADECK_TEMPLATE =
> >
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> > INIT_BEG = 20200101
> > INIT_END = 20200101
> >
> > # filter the init times of the lines read from the input files in
the
> > TCPairs configuration file
> > TC_PAIRS_INIT_BEG = 20200101
> > TC_PAIRS_INIT_END = 20200331
> >
> > # filter the valid times of the lines read from the input files in
the
> > TCPairs configuration file
> > TC_PAIRS_VALID_BEG = 20200101
> > TC_PAIRS_VALID_END = 20200331
> >
> > If any of TC_PAIRS_[INIT/VALID]_[BEG/END] are not set, then the
> > corresponding [INIT/VALID]_[BEG/END] values are used. This is done
to
> > preserve backwards compatibility for old METplus configurations.
The
> above
> > settings are more explicit as to what is actually be configured.
> >
> > You may want to consider either clearing your output directory or
> creating
> > a new output directory for a new run (especially if the time range
you
> are
> > using has changed) to ensure that the files you are generated are
from
> that
> > run and not from a previous run. Doing this would have alerted you
that
> > TCPairs was not run in this case.
> >
> > Let me know if you have any other issues getting this running.
> >
> > FYI:
> > We switched from providing support through this email address to
> providing
> > support through GitHub Discussions. In the future, you will need
to
> create
> > a free GitHub account if you don’t have one already and post your
> questions
> > to the METplus Components Discussion Forum:
> > https://github.com/dtcenter/METplus/discussions
> > To post a new question, click on the green button in the upper
right
> corner
> > of the page labeled "New discussion" and select "Incoming" as the
> category.
> >
> > Thanks,
> > George
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: tcpairs
From: George McCabe
Time: Tue Jul 20 15:36:10 2021
Hi Guang Ping,
I mistyped in my previous email. The variable to set to "times" is
LOOP_ORDER not LOOP_BY.
LOOP_ORDER = times
INIT_INCREMENT = 1m
Sorry for the confusion.
- George
On Tue, Jul 20, 2021 at 3:21 PM George McCabe <mccabe at ucar.edu> wrote:
> Hi Guang Ping,
>
> Thank you for providing the NetCDF files. I am currently downloading
one
> of the files but it is going very slowly.
>
> In regards to your use case, the logic for TCPairs wrapper is a
little
> strange. It behaves differently based on the value of LOOP_BY. If
LOOP_BY =
> processes, then it will run once using the INIT_BEG time as the run
time to
> find input files. If LOOP_BY = times, then it will loop over each
run time
> specified between INIT_BEG and INIT_END by INIT_INCREMENT and find
files to
> process for each run time.
>
> If you change your configuration file to include LOOP_BY = times,
then it
> should look over each run time and process for each. It looks like
your
> INIT_INCREMENT is set to 21600 or 6 hours, so it will run every 6
hours
> from 20200201 to 2020331. If you want to run once per month, you
will need
> to set INIT_INCREMENT = 1m so that it will increment by 1 month for
each
> iteration.
>
> The caveat is that TCMPRPlotter will only run if LOOP_BY = processes
and
> will produce an error if LOOP_BY = times. This is because we
typically want
> to process all of the data that is available for all run times, then
run
> TCMPRPlotter so it can read all of the data. For now, to handle this
use
> case you will have to call run_metplus.py 2 separate times, once
calling
> TCPairs with LOOP_BY = times and once calling TCMPRPlotter with
LOOP_BY =
> processes.
>
> I am going to add a configuration option to TCPairs wrapper so that
you
> can loop over the run times when LOOP_BY = processes is set if
desired. I
> will try to get this change into METplus 4.1.0-beta1 so you can test
the
> change and make sure it works as expected. I can let you know when
that is
> available.
>
> Thanks,
> George
>
> On Tue, Jul 20, 2021 at 1:27 PM Guang Lou - NOAA Affiliate via RT <
> met_help at ucar.edu> wrote:
>
>> Hi John,
>>
>> Thanks for forwarding George's email answer to me. I created a test
case
>> that does the tcpairs only on Hera:
>> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test3
>> Command line:
>> run_metplus.py -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c
>> usecase_8.conf
>> where PROCESS_LIST = TCPairs
>>
>> It generated the month of 202002, that is because I changed
>> INIT_BEG = 20200101
>> INIT_END = 20200331
>> to
>> INIT_BEG = 20200201
>> INIT_END = 20200331
>>
>> If I use "INIT_BEG = 20200101,INIT_END = 20200331" it will
generate
>> 20200101 only.
>>
>> The wild card does not work either for the model's place:
>> TC_PAIRS_ADECK_TEMPLATE =
>> {date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
>> I populated adeck and bdeck on
>>
>> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/METplus/METplus-
4.0_sample_data/model_applications/medium_range/tracks_all
>> (each subdirectory contains all three months' decks.)
>>
>> I was wondering if you could copy my directory,
>> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test3
>> and test:
>> 1. It can generate multiple months instead of just one month and
one
>> model.
>> 2. The wild card for models in the DECK_TEMPATE for multiple
models.
>>
>> George, I'm downloading NetCDF model data to Hera:
>>
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071300_nc.tar
>>
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071306_nc.tar
>>
>> Please let me know after you downloaded the files and I will have
to
>> remove
>> them due to tight disk space.
>>
>> Thanks,
>> Guang Ping
>>
>> On Mon, Jul 19, 2021 at 5:01 PM John Halley Gotway via RT <
>> met_help at ucar.edu>
>> wrote:
>>
>> > Hi Guang Ping,
>> >
>> > The command that you ran here:
>> >
>> >
>> >
>>
/home/George.Mccabe/scratch1_George.Mccabe/METplus.feature_926/ush/run_metplus.py
>> > -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c Plotter_8.conf
>> >
>> > only runs TCMPRPlotter. This is because PROCESS_LIST is defined
in both
>> > Plotter_fcstGFS_obsGFS_RPlotting_8.conf and Plotter_8.conf. Since
>> > Plotter_8.conf comes last, then the value for PROCESS_LIST from
that
>> file
>> > is used, which only has TCMPRPlotter listed. The plotter is
reading all
>> of
>> > the available files generated from TCPairs and since that process
is not
>> > running again with the new time range, the data that is read only
covers
>> > the first month.
>> >
>> > We are planning on releasing METplus 4.1.0-beta1 this week.
However, if
>> I
>> > understand what you are trying to do here, it shouldn't require
the
>> changes
>> > described in Molly's issue. It looks like you want to run TCPairs
once
>> and
>> > filter the data based on the init and valid times. While this
>> configuration
>> > should achieve what you are trying to do, the following
configuration is
>> > more explicit to what you are actually running:
>> >
>> > # run once using 20200101 to substitute the init time in your
string
>> > substitution, i.e. TC_PAIRS_ADECK_TEMPLATE =
>> >
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
>> > INIT_BEG = 20200101
>> > INIT_END = 20200101
>> >
>> > # filter the init times of the lines read from the input files in
the
>> > TCPairs configuration file
>> > TC_PAIRS_INIT_BEG = 20200101
>> > TC_PAIRS_INIT_END = 20200331
>> >
>> > # filter the valid times of the lines read from the input files
in the
>> > TCPairs configuration file
>> > TC_PAIRS_VALID_BEG = 20200101
>> > TC_PAIRS_VALID_END = 20200331
>> >
>> > If any of TC_PAIRS_[INIT/VALID]_[BEG/END] are not set, then the
>> > corresponding [INIT/VALID]_[BEG/END] values are used. This is
done to
>> > preserve backwards compatibility for old METplus configurations.
The
>> above
>> > settings are more explicit as to what is actually be configured.
>> >
>> > You may want to consider either clearing your output directory or
>> creating
>> > a new output directory for a new run (especially if the time
range you
>> are
>> > using has changed) to ensure that the files you are generated are
from
>> that
>> > run and not from a previous run. Doing this would have alerted
you that
>> > TCPairs was not run in this case.
>> >
>> > Let me know if you have any other issues getting this running.
>> >
>> > FYI:
>> > We switched from providing support through this email address to
>> providing
>> > support through GitHub Discussions. In the future, you will need
to
>> create
>> > a free GitHub account if you don’t have one already and post your
>> questions
>> > to the METplus Components Discussion Forum:
>> > https://github.com/dtcenter/METplus/discussions
>> > To post a new question, click on the green button in the upper
right
>> corner
>> > of the page labeled "New discussion" and select "Incoming" as the
>> category.
>> >
>> > Thanks,
>> > George
>> >
>> >
>>
>>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: tcpairs
From: George McCabe
Time: Wed Jul 21 12:41:26 2021
Hi Guang Ping,
I have finished downloading the GFS NetCDF files from Hera so you can
remove them. Thanks again for providing that data.
- George
On Tue, Jul 20, 2021 at 3:35 PM George McCabe <mccabe at ucar.edu> wrote:
> Hi Guang Ping,
>
> I mistyped in my previous email. The variable to set to "times" is
> LOOP_ORDER not LOOP_BY.
>
> LOOP_ORDER = times
> INIT_INCREMENT = 1m
>
> Sorry for the confusion.
>
> - George
>
> On Tue, Jul 20, 2021 at 3:21 PM George McCabe <mccabe at ucar.edu>
wrote:
>
>> Hi Guang Ping,
>>
>> Thank you for providing the NetCDF files. I am currently
downloading one
>> of the files but it is going very slowly.
>>
>> In regards to your use case, the logic for TCPairs wrapper is a
little
>> strange. It behaves differently based on the value of LOOP_BY. If
LOOP_BY =
>> processes, then it will run once using the INIT_BEG time as the run
time to
>> find input files. If LOOP_BY = times, then it will loop over each
run time
>> specified between INIT_BEG and INIT_END by INIT_INCREMENT and find
files to
>> process for each run time.
>>
>> If you change your configuration file to include LOOP_BY = times,
then it
>> should look over each run time and process for each. It looks like
your
>> INIT_INCREMENT is set to 21600 or 6 hours, so it will run every 6
hours
>> from 20200201 to 2020331. If you want to run once per month, you
will need
>> to set INIT_INCREMENT = 1m so that it will increment by 1 month for
each
>> iteration.
>>
>> The caveat is that TCMPRPlotter will only run if LOOP_BY =
processes and
>> will produce an error if LOOP_BY = times. This is because we
typically want
>> to process all of the data that is available for all run times,
then run
>> TCMPRPlotter so it can read all of the data. For now, to handle
this use
>> case you will have to call run_metplus.py 2 separate times, once
calling
>> TCPairs with LOOP_BY = times and once calling TCMPRPlotter with
LOOP_BY =
>> processes.
>>
>> I am going to add a configuration option to TCPairs wrapper so that
you
>> can loop over the run times when LOOP_BY = processes is set if
desired. I
>> will try to get this change into METplus 4.1.0-beta1 so you can
test the
>> change and make sure it works as expected. I can let you know when
that is
>> available.
>>
>> Thanks,
>> George
>>
>> On Tue, Jul 20, 2021 at 1:27 PM Guang Lou - NOAA Affiliate via RT <
>> met_help at ucar.edu> wrote:
>>
>>> Hi John,
>>>
>>> Thanks for forwarding George's email answer to me. I created a
test case
>>> that does the tcpairs only on Hera:
>>> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test3
>>> Command line:
>>> run_metplus.py -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c
>>> usecase_8.conf
>>> where PROCESS_LIST = TCPairs
>>>
>>> It generated the month of 202002, that is because I changed
>>> INIT_BEG = 20200101
>>> INIT_END = 20200331
>>> to
>>> INIT_BEG = 20200201
>>> INIT_END = 20200331
>>>
>>> If I use "INIT_BEG = 20200101,INIT_END = 20200331" it will
generate
>>> 20200101 only.
>>>
>>> The wild card does not work either for the model's place:
>>> TC_PAIRS_ADECK_TEMPLATE =
>>>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
>>> I populated adeck and bdeck on
>>>
>>> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/METplus/METplus-
4.0_sample_data/model_applications/medium_range/tracks_all
>>> (each subdirectory contains all three months' decks.)
>>>
>>> I was wondering if you could copy my directory,
>>> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test3
>>> and test:
>>> 1. It can generate multiple months instead of just one month and
one
>>> model.
>>> 2. The wild card for models in the DECK_TEMPATE for multiple
models.
>>>
>>> George, I'm downloading NetCDF model data to Hera:
>>>
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071300_nc.tar
>>>
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071306_nc.tar
>>>
>>> Please let me know after you downloaded the files and I will have
to
>>> remove
>>> them due to tight disk space.
>>>
>>> Thanks,
>>> Guang Ping
>>>
>>> On Mon, Jul 19, 2021 at 5:01 PM John Halley Gotway via RT <
>>> met_help at ucar.edu>
>>> wrote:
>>>
>>> > Hi Guang Ping,
>>> >
>>> > The command that you ran here:
>>> >
>>> >
>>> >
>>>
/home/George.Mccabe/scratch1_George.Mccabe/METplus.feature_926/ush/run_metplus.py
>>> > -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c Plotter_8.conf
>>> >
>>> > only runs TCMPRPlotter. This is because PROCESS_LIST is defined
in both
>>> > Plotter_fcstGFS_obsGFS_RPlotting_8.conf and Plotter_8.conf.
Since
>>> > Plotter_8.conf comes last, then the value for PROCESS_LIST from
that
>>> file
>>> > is used, which only has TCMPRPlotter listed. The plotter is
reading
>>> all of
>>> > the available files generated from TCPairs and since that
process is
>>> not
>>> > running again with the new time range, the data that is read
only
>>> covers
>>> > the first month.
>>> >
>>> > We are planning on releasing METplus 4.1.0-beta1 this week.
However,
>>> if I
>>> > understand what you are trying to do here, it shouldn't require
the
>>> changes
>>> > described in Molly's issue. It looks like you want to run
TCPairs once
>>> and
>>> > filter the data based on the init and valid times. While this
>>> configuration
>>> > should achieve what you are trying to do, the following
configuration
>>> is
>>> > more explicit to what you are actually running:
>>> >
>>> > # run once using 20200101 to substitute the init time in your
string
>>> > substitution, i.e. TC_PAIRS_ADECK_TEMPLATE =
>>> >
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
>>> > INIT_BEG = 20200101
>>> > INIT_END = 20200101
>>> >
>>> > # filter the init times of the lines read from the input files
in the
>>> > TCPairs configuration file
>>> > TC_PAIRS_INIT_BEG = 20200101
>>> > TC_PAIRS_INIT_END = 20200331
>>> >
>>> > # filter the valid times of the lines read from the input files
in the
>>> > TCPairs configuration file
>>> > TC_PAIRS_VALID_BEG = 20200101
>>> > TC_PAIRS_VALID_END = 20200331
>>> >
>>> > If any of TC_PAIRS_[INIT/VALID]_[BEG/END] are not set, then the
>>> > corresponding [INIT/VALID]_[BEG/END] values are used. This is
done to
>>> > preserve backwards compatibility for old METplus configurations.
The
>>> above
>>> > settings are more explicit as to what is actually be configured.
>>> >
>>> > You may want to consider either clearing your output directory
or
>>> creating
>>> > a new output directory for a new run (especially if the time
range you
>>> are
>>> > using has changed) to ensure that the files you are generated
are from
>>> that
>>> > run and not from a previous run. Doing this would have alerted
you that
>>> > TCPairs was not run in this case.
>>> >
>>> > Let me know if you have any other issues getting this running.
>>> >
>>> > FYI:
>>> > We switched from providing support through this email address to
>>> providing
>>> > support through GitHub Discussions. In the future, you will need
to
>>> create
>>> > a free GitHub account if you don’t have one already and post
your
>>> questions
>>> > to the METplus Components Discussion Forum:
>>> > https://github.com/dtcenter/METplus/discussions
>>> > To post a new question, click on the green button in the upper
right
>>> corner
>>> > of the page labeled "New discussion" and select "Incoming" as
the
>>> category.
>>> >
>>> > Thanks,
>>> > George
>>> >
>>> >
>>>
>>>
>>
>> --
>> George McCabe - Software Engineer III
>> National Center for Atmospheric Research
>> Research Applications Laboratory
>> 303-497-2768
>> ---
>> My working day may not be your working day. Please do not feel
obliged to
>> reply to this email outside of your normal working hours.
>>
>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: tcpairs
From: Guang Lou - NOAA Affiliate
Time: Fri Jul 23 11:45:47 2021
Hi George,
Thanks for letting me know you have downloaded the files. I've removed
the
files to save disk space.
Per your suggestion, I was able to generate consecutive months of
paired
tracks by changing a few lines
and parameters in the configure file. I've also loaded the module
"metplus/4.1.0-beta1".
This was done by:
/scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_pairs
run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c
usecase_pairing_8.conf
However, when I tried to generate plots from the just generated pairs,
it
ran into error:
in /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_pairs
run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c usecase_8.conf
where I changed the following to:
LOOP_ORDER = processes
PROCESS_LIST = TCMPRPlotter
error messages are:
07/23 17:38:51.477 metplus INFO: Check the logfile for more
information on
why it failed:
/scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
07/23 17:38:51.478 metplus INFO: Plotting complete
07/23 17:38:51.481 metplus.TCMPRPlotter (met_util.py:171) ERROR:
TCMPRPlotter had 9 errors.
07/23 17:38:51.481 metplus INFO:
/scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/metplus_final.conf:
write metplus.conf here
07/23 17:38:51.486 metplus (met_util.py:208) ERROR: METplus has
finished
running but had 9 errors.
07/23 17:38:51.486 metplus INFO: Check the log file for more
information:
/scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
-----------------------------------------------------------
The wild card in the place of
TC_PAIRS_ADECK_TEMPLATE =
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.nam.{cyclone?fmt=%s}
to
TC_PAIRS_ADECK_TEMPLATE =
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
still does not work.
Thanks,
Guang Ping
On Wed, Jul 21, 2021 at 2:41 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Guang Ping,
>
> I have finished downloading the GFS NetCDF files from Hera so you
can
> remove them. Thanks again for providing that data.
>
> - George
>
> On Tue, Jul 20, 2021 at 3:35 PM George McCabe <mccabe at ucar.edu>
wrote:
>
> > Hi Guang Ping,
> >
> > I mistyped in my previous email. The variable to set to "times" is
> > LOOP_ORDER not LOOP_BY.
> >
> > LOOP_ORDER = times
> > INIT_INCREMENT = 1m
> >
> > Sorry for the confusion.
> >
> > - George
> >
> > On Tue, Jul 20, 2021 at 3:21 PM George McCabe <mccabe at ucar.edu>
wrote:
> >
> >> Hi Guang Ping,
> >>
> >> Thank you for providing the NetCDF files. I am currently
downloading one
> >> of the files but it is going very slowly.
> >>
> >> In regards to your use case, the logic for TCPairs wrapper is a
little
> >> strange. It behaves differently based on the value of LOOP_BY. If
> LOOP_BY =
> >> processes, then it will run once using the INIT_BEG time as the
run
> time to
> >> find input files. If LOOP_BY = times, then it will loop over each
run
> time
> >> specified between INIT_BEG and INIT_END by INIT_INCREMENT and
find
> files to
> >> process for each run time.
> >>
> >> If you change your configuration file to include LOOP_BY = times,
then
> it
> >> should look over each run time and process for each. It looks
like your
> >> INIT_INCREMENT is set to 21600 or 6 hours, so it will run every 6
hours
> >> from 20200201 to 2020331. If you want to run once per month, you
will
> need
> >> to set INIT_INCREMENT = 1m so that it will increment by 1 month
for each
> >> iteration.
> >>
> >> The caveat is that TCMPRPlotter will only run if LOOP_BY =
processes and
> >> will produce an error if LOOP_BY = times. This is because we
typically
> want
> >> to process all of the data that is available for all run times,
then run
> >> TCMPRPlotter so it can read all of the data. For now, to handle
this use
> >> case you will have to call run_metplus.py 2 separate times, once
calling
> >> TCPairs with LOOP_BY = times and once calling TCMPRPlotter with
LOOP_BY
> =
> >> processes.
> >>
> >> I am going to add a configuration option to TCPairs wrapper so
that you
> >> can loop over the run times when LOOP_BY = processes is set if
desired.
> I
> >> will try to get this change into METplus 4.1.0-beta1 so you can
test the
> >> change and make sure it works as expected. I can let you know
when that
> is
> >> available.
> >>
> >> Thanks,
> >> George
> >>
> >> On Tue, Jul 20, 2021 at 1:27 PM Guang Lou - NOAA Affiliate via RT
<
> >> met_help at ucar.edu> wrote:
> >>
> >>> Hi John,
> >>>
> >>> Thanks for forwarding George's email answer to me. I created a
test
> case
> >>> that does the tcpairs only on Hera:
> >>> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test3
> >>> Command line:
> >>> run_metplus.py -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c
> >>> usecase_8.conf
> >>> where PROCESS_LIST = TCPairs
> >>>
> >>> It generated the month of 202002, that is because I changed
> >>> INIT_BEG = 20200101
> >>> INIT_END = 20200331
> >>> to
> >>> INIT_BEG = 20200201
> >>> INIT_END = 20200331
> >>>
> >>> If I use "INIT_BEG = 20200101,INIT_END = 20200331" it will
generate
> >>> 20200101 only.
> >>>
> >>> The wild card does not work either for the model's place:
> >>> TC_PAIRS_ADECK_TEMPLATE =
> >>>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> >>> I populated adeck and bdeck on
> >>>
> >>>
> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/METplus/METplus-
4.0_sample_data/model_applications/medium_range/tracks_all
> >>> (each subdirectory contains all three months' decks.)
> >>>
> >>> I was wondering if you could copy my directory,
> >>> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test3
> >>> and test:
> >>> 1. It can generate multiple months instead of just one month and
one
> >>> model.
> >>> 2. The wild card for models in the DECK_TEMPATE for multiple
models.
> >>>
> >>> George, I'm downloading NetCDF model data to Hera:
> >>>
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071300_nc.tar
> >>>
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071306_nc.tar
> >>>
> >>> Please let me know after you downloaded the files and I will
have to
> >>> remove
> >>> them due to tight disk space.
> >>>
> >>> Thanks,
> >>> Guang Ping
> >>>
> >>> On Mon, Jul 19, 2021 at 5:01 PM John Halley Gotway via RT <
> >>> met_help at ucar.edu>
> >>> wrote:
> >>>
> >>> > Hi Guang Ping,
> >>> >
> >>> > The command that you ran here:
> >>> >
> >>> >
> >>> >
> >>>
>
/home/George.Mccabe/scratch1_George.Mccabe/METplus.feature_926/ush/run_metplus.py
> >>> > -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c Plotter_8.conf
> >>> >
> >>> > only runs TCMPRPlotter. This is because PROCESS_LIST is
defined in
> both
> >>> > Plotter_fcstGFS_obsGFS_RPlotting_8.conf and Plotter_8.conf.
Since
> >>> > Plotter_8.conf comes last, then the value for PROCESS_LIST
from that
> >>> file
> >>> > is used, which only has TCMPRPlotter listed. The plotter is
reading
> >>> all of
> >>> > the available files generated from TCPairs and since that
process is
> >>> not
> >>> > running again with the new time range, the data that is read
only
> >>> covers
> >>> > the first month.
> >>> >
> >>> > We are planning on releasing METplus 4.1.0-beta1 this week.
However,
> >>> if I
> >>> > understand what you are trying to do here, it shouldn't
require the
> >>> changes
> >>> > described in Molly's issue. It looks like you want to run
TCPairs
> once
> >>> and
> >>> > filter the data based on the init and valid times. While this
> >>> configuration
> >>> > should achieve what you are trying to do, the following
configuration
> >>> is
> >>> > more explicit to what you are actually running:
> >>> >
> >>> > # run once using 20200101 to substitute the init time in your
string
> >>> > substitution, i.e. TC_PAIRS_ADECK_TEMPLATE =
> >>> >
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> >>> > INIT_BEG = 20200101
> >>> > INIT_END = 20200101
> >>> >
> >>> > # filter the init times of the lines read from the input files
in the
> >>> > TCPairs configuration file
> >>> > TC_PAIRS_INIT_BEG = 20200101
> >>> > TC_PAIRS_INIT_END = 20200331
> >>> >
> >>> > # filter the valid times of the lines read from the input
files in
> the
> >>> > TCPairs configuration file
> >>> > TC_PAIRS_VALID_BEG = 20200101
> >>> > TC_PAIRS_VALID_END = 20200331
> >>> >
> >>> > If any of TC_PAIRS_[INIT/VALID]_[BEG/END] are not set, then
the
> >>> > corresponding [INIT/VALID]_[BEG/END] values are used. This is
done to
> >>> > preserve backwards compatibility for old METplus
configurations. The
> >>> above
> >>> > settings are more explicit as to what is actually be
configured.
> >>> >
> >>> > You may want to consider either clearing your output directory
or
> >>> creating
> >>> > a new output directory for a new run (especially if the time
range
> you
> >>> are
> >>> > using has changed) to ensure that the files you are generated
are
> from
> >>> that
> >>> > run and not from a previous run. Doing this would have alerted
you
> that
> >>> > TCPairs was not run in this case.
> >>> >
> >>> > Let me know if you have any other issues getting this running.
> >>> >
> >>> > FYI:
> >>> > We switched from providing support through this email address
to
> >>> providing
> >>> > support through GitHub Discussions. In the future, you will
need to
> >>> create
> >>> > a free GitHub account if you don’t have one already and post
your
> >>> questions
> >>> > to the METplus Components Discussion Forum:
> >>> > https://github.com/dtcenter/METplus/discussions
> >>> > To post a new question, click on the green button in the upper
right
> >>> corner
> >>> > of the page labeled "New discussion" and select "Incoming" as
the
> >>> category.
> >>> >
> >>> > Thanks,
> >>> > George
> >>> >
> >>> >
> >>>
> >>>
> >>
> >> --
> >> George McCabe - Software Engineer III
> >> National Center for Atmospheric Research
> >> Research Applications Laboratory
> >> 303-497-2768
> >> ---
> >> My working day may not be your working day. Please do not feel
obliged
> to
> >> reply to this email outside of your normal working hours.
> >>
> >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: tcpairs
From: George McCabe
Time: Mon Jul 26 17:57:37 2021
Hi Guang Ping,
I copied your config file and made some changes. The new file can be
found
on hera here:
/home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8.conf
Using METplus 4.1.0-beta1, you can set TC_PAIRS_RUN_ONCE = True to
loop
over the init times even with LOOP_ORDER = processes.
I noticed that you have TC_PAIRS_REFORMAT_DECK = yes, which isn't
necessary
because tc_pairs can read the input files without reformatting. That
can be
unset or set to False/no.
TC_PAIRS_REFORMAT_DECK = no
I misread your email about trying to use the wildcard for the model.
This
won't work as you expect because the output template will not know how
to
name the output file with the correct model. Each model will overwrite
the
output files. To get around this you can use
TC_PAIRS_CUSTOM_LOOP_LIST.
This will run the TCPairs wrapper once for each item in the custom
loop
list. Here you can specify the list of models (lowercase) and
reference the
{custom} template tag to find the files.
TC_PAIRS_CUSTOM_LOOP_LIST = best, cmc, emx, gfso, nam, ngx, ukx
TC_PAIRS_ADECK_TEMPLATE =
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
TC_PAIRS_BDECK_TEMPLATE =
{date?fmt=%Y%m}/b{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
TC_PAIRS_OUTPUT_TEMPLATE =
{date?fmt=%Y%m}/{basin?fmt=%s}q{date?fmt=%Y%m%d%H}.{custom}.{cyclone?fmt=%s}
I removed the MODEL values (you had it twice in your config file)
because
we don't need to filter by the model if we are reading all of them in
individually.
Using this config file I was able to run TCPairs for each model
processing
each for all 3 months and creating output files from tc_pairs here:
/scratch1/BMC/dtc/George.Mccabe/tcmpr-test2/tc_pairs
The error in your log file from TCMPRPlotter mentions that you do not
have
-bmodel set in the filter options. When I ran it, I got an error with
a
command that appears to list the files which leads me to think that
there
are too many files in the tc_pairs directory to process at once. I
need to
stop working for today, but I will resume debugging this tomorrow and
see
what I can find. It may be the case that the plotter cannot handle all
7
models over a 3 day period all at once. If this is the case, what
would be
useful to display as a subset of this data?
Thanks,
George
On Fri, Jul 23, 2021 at 11:46 AM Guang Lou - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100602 >
>
> Hi George,
>
> Thanks for letting me know you have downloaded the files. I've
removed the
> files to save disk space.
> Per your suggestion, I was able to generate consecutive months of
paired
> tracks by changing a few lines
> and parameters in the configure file. I've also loaded the module
> "metplus/4.1.0-beta1".
>
> This was done by:
> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_pairs
> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c
usecase_pairing_8.conf
>
> However, when I tried to generate plots from the just generated
pairs, it
> ran into error:
> in /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_pairs
> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c usecase_8.conf
>
> where I changed the following to:
> LOOP_ORDER = processes
> PROCESS_LIST = TCMPRPlotter
> error messages are:
> 07/23 17:38:51.477 metplus INFO: Check the logfile for more
information on
> why it failed:
>
> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
> 07/23 17:38:51.478 metplus INFO: Plotting complete
> 07/23 17:38:51.481 metplus.TCMPRPlotter (met_util.py:171) ERROR:
> TCMPRPlotter had 9 errors.
> 07/23 17:38:51.481 metplus INFO:
>
> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/metplus_final.conf:
> write metplus.conf here
> 07/23 17:38:51.486 metplus (met_util.py:208) ERROR: METplus has
finished
> running but had 9 errors.
> 07/23 17:38:51.486 metplus INFO: Check the log file for more
information:
>
> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
> -----------------------------------------------------------
> The wild card in the place of
> TC_PAIRS_ADECK_TEMPLATE =
>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.nam.{cyclone?fmt=%s}
> to
> TC_PAIRS_ADECK_TEMPLATE =
> {date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> still does not work.
>
> Thanks,
> Guang Ping
>
> On Wed, Jul 21, 2021 at 2:41 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Guang Ping,
> >
> > I have finished downloading the GFS NetCDF files from Hera so you
can
> > remove them. Thanks again for providing that data.
> >
> > - George
> >
> > On Tue, Jul 20, 2021 at 3:35 PM George McCabe <mccabe at ucar.edu>
wrote:
> >
> > > Hi Guang Ping,
> > >
> > > I mistyped in my previous email. The variable to set to "times"
is
> > > LOOP_ORDER not LOOP_BY.
> > >
> > > LOOP_ORDER = times
> > > INIT_INCREMENT = 1m
> > >
> > > Sorry for the confusion.
> > >
> > > - George
> > >
> > > On Tue, Jul 20, 2021 at 3:21 PM George McCabe <mccabe at ucar.edu>
wrote:
> > >
> > >> Hi Guang Ping,
> > >>
> > >> Thank you for providing the NetCDF files. I am currently
downloading
> one
> > >> of the files but it is going very slowly.
> > >>
> > >> In regards to your use case, the logic for TCPairs wrapper is a
little
> > >> strange. It behaves differently based on the value of LOOP_BY.
If
> > LOOP_BY =
> > >> processes, then it will run once using the INIT_BEG time as the
run
> > time to
> > >> find input files. If LOOP_BY = times, then it will loop over
each run
> > time
> > >> specified between INIT_BEG and INIT_END by INIT_INCREMENT and
find
> > files to
> > >> process for each run time.
> > >>
> > >> If you change your configuration file to include LOOP_BY =
times, then
> > it
> > >> should look over each run time and process for each. It looks
like
> your
> > >> INIT_INCREMENT is set to 21600 or 6 hours, so it will run every
6
> hours
> > >> from 20200201 to 2020331. If you want to run once per month,
you will
> > need
> > >> to set INIT_INCREMENT = 1m so that it will increment by 1 month
for
> each
> > >> iteration.
> > >>
> > >> The caveat is that TCMPRPlotter will only run if LOOP_BY =
processes
> and
> > >> will produce an error if LOOP_BY = times. This is because we
typically
> > want
> > >> to process all of the data that is available for all run times,
then
> run
> > >> TCMPRPlotter so it can read all of the data. For now, to handle
this
> use
> > >> case you will have to call run_metplus.py 2 separate times,
once
> calling
> > >> TCPairs with LOOP_BY = times and once calling TCMPRPlotter with
> LOOP_BY
> > =
> > >> processes.
> > >>
> > >> I am going to add a configuration option to TCPairs wrapper so
that
> you
> > >> can loop over the run times when LOOP_BY = processes is set if
> desired.
> > I
> > >> will try to get this change into METplus 4.1.0-beta1 so you can
test
> the
> > >> change and make sure it works as expected. I can let you know
when
> that
> > is
> > >> available.
> > >>
> > >> Thanks,
> > >> George
> > >>
> > >> On Tue, Jul 20, 2021 at 1:27 PM Guang Lou - NOAA Affiliate via
RT <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >>> Hi John,
> > >>>
> > >>> Thanks for forwarding George's email answer to me. I created a
test
> > case
> > >>> that does the tcpairs only on Hera:
> > >>> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test3
> > >>> Command line:
> > >>> run_metplus.py -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c
> > >>> usecase_8.conf
> > >>> where PROCESS_LIST = TCPairs
> > >>>
> > >>> It generated the month of 202002, that is because I changed
> > >>> INIT_BEG = 20200101
> > >>> INIT_END = 20200331
> > >>> to
> > >>> INIT_BEG = 20200201
> > >>> INIT_END = 20200331
> > >>>
> > >>> If I use "INIT_BEG = 20200101,INIT_END = 20200331" it will
generate
> > >>> 20200101 only.
> > >>>
> > >>> The wild card does not work either for the model's place:
> > >>> TC_PAIRS_ADECK_TEMPLATE =
> > >>>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> > >>> I populated adeck and bdeck on
> > >>>
> > >>>
> >
> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/METplus/METplus-
4.0_sample_data/model_applications/medium_range/tracks_all
> > >>> (each subdirectory contains all three months' decks.)
> > >>>
> > >>> I was wondering if you could copy my directory,
> > >>> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test3
> > >>> and test:
> > >>> 1. It can generate multiple months instead of just one month
and one
> > >>> model.
> > >>> 2. The wild card for models in the DECK_TEMPATE for multiple
models.
> > >>>
> > >>> George, I'm downloading NetCDF model data to Hera:
> > >>>
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071300_nc.tar
> > >>>
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071306_nc.tar
> > >>>
> > >>> Please let me know after you downloaded the files and I will
have to
> > >>> remove
> > >>> them due to tight disk space.
> > >>>
> > >>> Thanks,
> > >>> Guang Ping
> > >>>
> > >>> On Mon, Jul 19, 2021 at 5:01 PM John Halley Gotway via RT <
> > >>> met_help at ucar.edu>
> > >>> wrote:
> > >>>
> > >>> > Hi Guang Ping,
> > >>> >
> > >>> > The command that you ran here:
> > >>> >
> > >>> >
> > >>> >
> > >>>
> >
>
/home/George.Mccabe/scratch1_George.Mccabe/METplus.feature_926/ush/run_metplus.py
> > >>> > -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c Plotter_8.conf
> > >>> >
> > >>> > only runs TCMPRPlotter. This is because PROCESS_LIST is
defined in
> > both
> > >>> > Plotter_fcstGFS_obsGFS_RPlotting_8.conf and Plotter_8.conf.
Since
> > >>> > Plotter_8.conf comes last, then the value for PROCESS_LIST
from
> that
> > >>> file
> > >>> > is used, which only has TCMPRPlotter listed. The plotter is
reading
> > >>> all of
> > >>> > the available files generated from TCPairs and since that
process
> is
> > >>> not
> > >>> > running again with the new time range, the data that is read
only
> > >>> covers
> > >>> > the first month.
> > >>> >
> > >>> > We are planning on releasing METplus 4.1.0-beta1 this week.
> However,
> > >>> if I
> > >>> > understand what you are trying to do here, it shouldn't
require the
> > >>> changes
> > >>> > described in Molly's issue. It looks like you want to run
TCPairs
> > once
> > >>> and
> > >>> > filter the data based on the init and valid times. While
this
> > >>> configuration
> > >>> > should achieve what you are trying to do, the following
> configuration
> > >>> is
> > >>> > more explicit to what you are actually running:
> > >>> >
> > >>> > # run once using 20200101 to substitute the init time in
your
> string
> > >>> > substitution, i.e. TC_PAIRS_ADECK_TEMPLATE =
> > >>> >
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> > >>> > INIT_BEG = 20200101
> > >>> > INIT_END = 20200101
> > >>> >
> > >>> > # filter the init times of the lines read from the input
files in
> the
> > >>> > TCPairs configuration file
> > >>> > TC_PAIRS_INIT_BEG = 20200101
> > >>> > TC_PAIRS_INIT_END = 20200331
> > >>> >
> > >>> > # filter the valid times of the lines read from the input
files in
> > the
> > >>> > TCPairs configuration file
> > >>> > TC_PAIRS_VALID_BEG = 20200101
> > >>> > TC_PAIRS_VALID_END = 20200331
> > >>> >
> > >>> > If any of TC_PAIRS_[INIT/VALID]_[BEG/END] are not set, then
the
> > >>> > corresponding [INIT/VALID]_[BEG/END] values are used. This
is done
> to
> > >>> > preserve backwards compatibility for old METplus
configurations.
> The
> > >>> above
> > >>> > settings are more explicit as to what is actually be
configured.
> > >>> >
> > >>> > You may want to consider either clearing your output
directory or
> > >>> creating
> > >>> > a new output directory for a new run (especially if the time
range
> > you
> > >>> are
> > >>> > using has changed) to ensure that the files you are
generated are
> > from
> > >>> that
> > >>> > run and not from a previous run. Doing this would have
alerted you
> > that
> > >>> > TCPairs was not run in this case.
> > >>> >
> > >>> > Let me know if you have any other issues getting this
running.
> > >>> >
> > >>> > FYI:
> > >>> > We switched from providing support through this email
address to
> > >>> providing
> > >>> > support through GitHub Discussions. In the future, you will
need to
> > >>> create
> > >>> > a free GitHub account if you don’t have one already and post
your
> > >>> questions
> > >>> > to the METplus Components Discussion Forum:
> > >>> > https://github.com/dtcenter/METplus/discussions
> > >>> > To post a new question, click on the green button in the
upper
> right
> > >>> corner
> > >>> > of the page labeled "New discussion" and select "Incoming"
as the
> > >>> category.
> > >>> >
> > >>> > Thanks,
> > >>> > George
> > >>> >
> > >>> >
> > >>>
> > >>>
> > >>
> > >> --
> > >> George McCabe - Software Engineer III
> > >> National Center for Atmospheric Research
> > >> Research Applications Laboratory
> > >> 303-497-2768
> > >> ---
> > >> My working day may not be your working day. Please do not feel
obliged
> > to
> > >> reply to this email outside of your normal working hours.
> > >>
> > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: tcpairs
From: George McCabe
Time: Tue Jul 27 12:33:22 2021
Hi Guang Ping,
I looked at this use case more today. I modified the use case to read
a
single month into TCMPRPlotter because it cannot handle all of the
files at
once. It looks like you need to set the -bmodel in
TCMPR_PLOTTER_FILTER to
specify the Bdeck to compare the models to. However, each of the
output
files from tc_pairs use the same model for adeck and bdeck, so the
AMODEL
and BMODEL values in those files are the same, i.e. both GFSO. If you
want
to compare all of the models to a single bdeck, you will have to
configure
TCPairs wrapper to use the same bdeck for each model with
TC_PAIRS_BDECK_TEMPLATE. I tried running using best for the bdeck and
it
produced no pairs. I am rerunning now using GFSO.
- George
On Mon, Jul 26, 2021 at 5:57 PM George McCabe <mccabe at ucar.edu> wrote:
> Hi Guang Ping,
>
> I copied your config file and made some changes. The new file can be
found
> on hera here:
>
> /home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8.conf
>
> Using METplus 4.1.0-beta1, you can set TC_PAIRS_RUN_ONCE = True to
loop
> over the init times even with LOOP_ORDER = processes.
>
> I noticed that you have TC_PAIRS_REFORMAT_DECK = yes, which isn't
> necessary because tc_pairs can read the input files without
reformatting.
> That can be unset or set to False/no.
>
> TC_PAIRS_REFORMAT_DECK = no
>
> I misread your email about trying to use the wildcard for the model.
This
> won't work as you expect because the output template will not know
how to
> name the output file with the correct model. Each model will
overwrite the
> output files. To get around this you can use
TC_PAIRS_CUSTOM_LOOP_LIST.
> This will run the TCPairs wrapper once for each item in the custom
loop
> list. Here you can specify the list of models (lowercase) and
reference the
> {custom} template tag to find the files.
>
> TC_PAIRS_CUSTOM_LOOP_LIST = best, cmc, emx, gfso, nam, ngx, ukx
>
> TC_PAIRS_ADECK_TEMPLATE =
>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
> TC_PAIRS_BDECK_TEMPLATE =
>
{date?fmt=%Y%m}/b{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
> TC_PAIRS_OUTPUT_TEMPLATE =
>
{date?fmt=%Y%m}/{basin?fmt=%s}q{date?fmt=%Y%m%d%H}.{custom}.{cyclone?fmt=%s}
>
> I removed the MODEL values (you had it twice in your config file)
because
> we don't need to filter by the model if we are reading all of them
in
> individually.
>
> Using this config file I was able to run TCPairs for each model
processing
> each for all 3 months and creating output files from tc_pairs here:
>
> /scratch1/BMC/dtc/George.Mccabe/tcmpr-test2/tc_pairs
>
> The error in your log file from TCMPRPlotter mentions that you do
not have
> -bmodel set in the filter options. When I ran it, I got an error
with a
> command that appears to list the files which leads me to think that
there
> are too many files in the tc_pairs directory to process at once. I
need to
> stop working for today, but I will resume debugging this tomorrow
and see
> what I can find. It may be the case that the plotter cannot handle
all 7
> models over a 3 day period all at once. If this is the case, what
would be
> useful to display as a subset of this data?
>
> Thanks,
> George
>
>
>
> On Fri, Jul 23, 2021 at 11:46 AM Guang Lou - NOAA Affiliate via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100602 >
>>
>> Hi George,
>>
>> Thanks for letting me know you have downloaded the files. I've
removed the
>> files to save disk space.
>> Per your suggestion, I was able to generate consecutive months of
paired
>> tracks by changing a few lines
>> and parameters in the configure file. I've also loaded the module
>> "metplus/4.1.0-beta1".
>>
>> This was done by:
>> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_pairs
>> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c
>> usecase_pairing_8.conf
>>
>> However, when I tried to generate plots from the just generated
pairs, it
>> ran into error:
>> in /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_pairs
>> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c usecase_8.conf
>>
>> where I changed the following to:
>> LOOP_ORDER = processes
>> PROCESS_LIST = TCMPRPlotter
>> error messages are:
>> 07/23 17:38:51.477 metplus INFO: Check the logfile for more
information on
>> why it failed:
>>
>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
>> 07/23 17:38:51.478 metplus INFO: Plotting complete
>> 07/23 17:38:51.481 metplus.TCMPRPlotter (met_util.py:171) ERROR:
>> TCMPRPlotter had 9 errors.
>> 07/23 17:38:51.481 metplus INFO:
>>
>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/metplus_final.conf:
>> write metplus.conf here
>> 07/23 17:38:51.486 metplus (met_util.py:208) ERROR: METplus has
finished
>> running but had 9 errors.
>> 07/23 17:38:51.486 metplus INFO: Check the log file for more
information:
>>
>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
>> -----------------------------------------------------------
>> The wild card in the place of
>> TC_PAIRS_ADECK_TEMPLATE =
>>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.nam.{cyclone?fmt=%s}
>> to
>> TC_PAIRS_ADECK_TEMPLATE =
>> {date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
>> still does not work.
>>
>> Thanks,
>> Guang Ping
>>
>> On Wed, Jul 21, 2021 at 2:41 PM George McCabe via RT
<met_help at ucar.edu>
>> wrote:
>>
>> > Hi Guang Ping,
>> >
>> > I have finished downloading the GFS NetCDF files from Hera so you
can
>> > remove them. Thanks again for providing that data.
>> >
>> > - George
>> >
>> > On Tue, Jul 20, 2021 at 3:35 PM George McCabe <mccabe at ucar.edu>
wrote:
>> >
>> > > Hi Guang Ping,
>> > >
>> > > I mistyped in my previous email. The variable to set to "times"
is
>> > > LOOP_ORDER not LOOP_BY.
>> > >
>> > > LOOP_ORDER = times
>> > > INIT_INCREMENT = 1m
>> > >
>> > > Sorry for the confusion.
>> > >
>> > > - George
>> > >
>> > > On Tue, Jul 20, 2021 at 3:21 PM George McCabe <mccabe at ucar.edu>
>> wrote:
>> > >
>> > >> Hi Guang Ping,
>> > >>
>> > >> Thank you for providing the NetCDF files. I am currently
downloading
>> one
>> > >> of the files but it is going very slowly.
>> > >>
>> > >> In regards to your use case, the logic for TCPairs wrapper is
a
>> little
>> > >> strange. It behaves differently based on the value of LOOP_BY.
If
>> > LOOP_BY =
>> > >> processes, then it will run once using the INIT_BEG time as
the run
>> > time to
>> > >> find input files. If LOOP_BY = times, then it will loop over
each run
>> > time
>> > >> specified between INIT_BEG and INIT_END by INIT_INCREMENT and
find
>> > files to
>> > >> process for each run time.
>> > >>
>> > >> If you change your configuration file to include LOOP_BY =
times,
>> then
>> > it
>> > >> should look over each run time and process for each. It looks
like
>> your
>> > >> INIT_INCREMENT is set to 21600 or 6 hours, so it will run
every 6
>> hours
>> > >> from 20200201 to 2020331. If you want to run once per month,
you will
>> > need
>> > >> to set INIT_INCREMENT = 1m so that it will increment by 1
month for
>> each
>> > >> iteration.
>> > >>
>> > >> The caveat is that TCMPRPlotter will only run if LOOP_BY =
processes
>> and
>> > >> will produce an error if LOOP_BY = times. This is because we
>> typically
>> > want
>> > >> to process all of the data that is available for all run
times, then
>> run
>> > >> TCMPRPlotter so it can read all of the data. For now, to
handle this
>> use
>> > >> case you will have to call run_metplus.py 2 separate times,
once
>> calling
>> > >> TCPairs with LOOP_BY = times and once calling TCMPRPlotter
with
>> LOOP_BY
>> > =
>> > >> processes.
>> > >>
>> > >> I am going to add a configuration option to TCPairs wrapper so
that
>> you
>> > >> can loop over the run times when LOOP_BY = processes is set if
>> desired.
>> > I
>> > >> will try to get this change into METplus 4.1.0-beta1 so you
can test
>> the
>> > >> change and make sure it works as expected. I can let you know
when
>> that
>> > is
>> > >> available.
>> > >>
>> > >> Thanks,
>> > >> George
>> > >>
>> > >> On Tue, Jul 20, 2021 at 1:27 PM Guang Lou - NOAA Affiliate via
RT <
>> > >> met_help at ucar.edu> wrote:
>> > >>
>> > >>> Hi John,
>> > >>>
>> > >>> Thanks for forwarding George's email answer to me. I created
a test
>> > case
>> > >>> that does the tcpairs only on Hera:
>> > >>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_test3
>> > >>> Command line:
>> > >>> run_metplus.py -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c
>> > >>> usecase_8.conf
>> > >>> where PROCESS_LIST = TCPairs
>> > >>>
>> > >>> It generated the month of 202002, that is because I changed
>> > >>> INIT_BEG = 20200101
>> > >>> INIT_END = 20200331
>> > >>> to
>> > >>> INIT_BEG = 20200201
>> > >>> INIT_END = 20200331
>> > >>>
>> > >>> If I use "INIT_BEG = 20200101,INIT_END = 20200331" it will
generate
>> > >>> 20200101 only.
>> > >>>
>> > >>> The wild card does not work either for the model's place:
>> > >>> TC_PAIRS_ADECK_TEMPLATE =
>> > >>>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
>> > >>> I populated adeck and bdeck on
>> > >>>
>> > >>>
>> >
>> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/METplus/METplus-
4.0_sample_data/model_applications/medium_range/tracks_all
>> > >>> (each subdirectory contains all three months' decks.)
>> > >>>
>> > >>> I was wondering if you could copy my directory,
>> > >>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_test3
>> > >>> and test:
>> > >>> 1. It can generate multiple months instead of just one month
and one
>> > >>> model.
>> > >>> 2. The wild card for models in the DECK_TEMPATE for multiple
models.
>> > >>>
>> > >>> George, I'm downloading NetCDF model data to Hera:
>> > >>>
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071300_nc.tar
>> > >>>
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071306_nc.tar
>> > >>>
>> > >>> Please let me know after you downloaded the files and I will
have to
>> > >>> remove
>> > >>> them due to tight disk space.
>> > >>>
>> > >>> Thanks,
>> > >>> Guang Ping
>> > >>>
>> > >>> On Mon, Jul 19, 2021 at 5:01 PM John Halley Gotway via RT <
>> > >>> met_help at ucar.edu>
>> > >>> wrote:
>> > >>>
>> > >>> > Hi Guang Ping,
>> > >>> >
>> > >>> > The command that you ran here:
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>>
>> >
>>
/home/George.Mccabe/scratch1_George.Mccabe/METplus.feature_926/ush/run_metplus.py
>> > >>> > -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c
Plotter_8.conf
>> > >>> >
>> > >>> > only runs TCMPRPlotter. This is because PROCESS_LIST is
defined in
>> > both
>> > >>> > Plotter_fcstGFS_obsGFS_RPlotting_8.conf and Plotter_8.conf.
Since
>> > >>> > Plotter_8.conf comes last, then the value for PROCESS_LIST
from
>> that
>> > >>> file
>> > >>> > is used, which only has TCMPRPlotter listed. The plotter is
>> reading
>> > >>> all of
>> > >>> > the available files generated from TCPairs and since that
process
>> is
>> > >>> not
>> > >>> > running again with the new time range, the data that is
read only
>> > >>> covers
>> > >>> > the first month.
>> > >>> >
>> > >>> > We are planning on releasing METplus 4.1.0-beta1 this week.
>> However,
>> > >>> if I
>> > >>> > understand what you are trying to do here, it shouldn't
require
>> the
>> > >>> changes
>> > >>> > described in Molly's issue. It looks like you want to run
TCPairs
>> > once
>> > >>> and
>> > >>> > filter the data based on the init and valid times. While
this
>> > >>> configuration
>> > >>> > should achieve what you are trying to do, the following
>> configuration
>> > >>> is
>> > >>> > more explicit to what you are actually running:
>> > >>> >
>> > >>> > # run once using 20200101 to substitute the init time in
your
>> string
>> > >>> > substitution, i.e. TC_PAIRS_ADECK_TEMPLATE =
>> > >>> >
>> {date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
>> > >>> > INIT_BEG = 20200101
>> > >>> > INIT_END = 20200101
>> > >>> >
>> > >>> > # filter the init times of the lines read from the input
files in
>> the
>> > >>> > TCPairs configuration file
>> > >>> > TC_PAIRS_INIT_BEG = 20200101
>> > >>> > TC_PAIRS_INIT_END = 20200331
>> > >>> >
>> > >>> > # filter the valid times of the lines read from the input
files in
>> > the
>> > >>> > TCPairs configuration file
>> > >>> > TC_PAIRS_VALID_BEG = 20200101
>> > >>> > TC_PAIRS_VALID_END = 20200331
>> > >>> >
>> > >>> > If any of TC_PAIRS_[INIT/VALID]_[BEG/END] are not set, then
the
>> > >>> > corresponding [INIT/VALID]_[BEG/END] values are used. This
is
>> done to
>> > >>> > preserve backwards compatibility for old METplus
configurations.
>> The
>> > >>> above
>> > >>> > settings are more explicit as to what is actually be
configured.
>> > >>> >
>> > >>> > You may want to consider either clearing your output
directory or
>> > >>> creating
>> > >>> > a new output directory for a new run (especially if the
time range
>> > you
>> > >>> are
>> > >>> > using has changed) to ensure that the files you are
generated are
>> > from
>> > >>> that
>> > >>> > run and not from a previous run. Doing this would have
alerted you
>> > that
>> > >>> > TCPairs was not run in this case.
>> > >>> >
>> > >>> > Let me know if you have any other issues getting this
running.
>> > >>> >
>> > >>> > FYI:
>> > >>> > We switched from providing support through this email
address to
>> > >>> providing
>> > >>> > support through GitHub Discussions. In the future, you will
need
>> to
>> > >>> create
>> > >>> > a free GitHub account if you don’t have one already and
post your
>> > >>> questions
>> > >>> > to the METplus Components Discussion Forum:
>> > >>> > https://github.com/dtcenter/METplus/discussions
>> > >>> > To post a new question, click on the green button in the
upper
>> right
>> > >>> corner
>> > >>> > of the page labeled "New discussion" and select "Incoming"
as the
>> > >>> category.
>> > >>> >
>> > >>> > Thanks,
>> > >>> > George
>> > >>> >
>> > >>> >
>> > >>>
>> > >>>
>> > >>
>> > >> --
>> > >> George McCabe - Software Engineer III
>> > >> National Center for Atmospheric Research
>> > >> Research Applications Laboratory
>> > >> 303-497-2768
>> > >> ---
>> > >> My working day may not be your working day. Please do not feel
>> obliged
>> > to
>> > >> reply to this email outside of your normal working hours.
>> > >>
>> > >
>> > >
>> > > --
>> > > George McCabe - Software Engineer III
>> > > National Center for Atmospheric Research
>> > > Research Applications Laboratory
>> > > 303-497-2768
>> > > ---
>> > > My working day may not be your working day. Please do not feel
>> obliged to
>> > > reply to this email outside of your normal working hours.
>> > >
>> >
>> >
>> > --
>> > George McCabe - Software Engineer III
>> > National Center for Atmospheric Research
>> > Research Applications Laboratory
>> > 303-497-2768
>> > ---
>> > My working day may not be your working day. Please do not feel
obliged
>> to
>> > reply to this email outside of your normal working hours.
>> >
>> >
>>
>>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: tcpairs
From: Guang Lou - NOAA Affiliate
Time: Wed Jul 28 09:46:19 2021
Hi George,
The verification should be against self analysis. That is GFS adeck
against
GFS bdeck, CMC adeck against CMC bdeck, and so on and so forth.
Thanks,
Guang Ping
On Tue, Jul 27, 2021 at 2:33 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Guang Ping,
>
> I looked at this use case more today. I modified the use case to
read a
> single month into TCMPRPlotter because it cannot handle all of the
files at
> once. It looks like you need to set the -bmodel in
TCMPR_PLOTTER_FILTER to
> specify the Bdeck to compare the models to. However, each of the
output
> files from tc_pairs use the same model for adeck and bdeck, so the
AMODEL
> and BMODEL values in those files are the same, i.e. both GFSO. If
you want
> to compare all of the models to a single bdeck, you will have to
configure
> TCPairs wrapper to use the same bdeck for each model with
> TC_PAIRS_BDECK_TEMPLATE. I tried running using best for the bdeck
and it
> produced no pairs. I am rerunning now using GFSO.
>
> - George
>
> On Mon, Jul 26, 2021 at 5:57 PM George McCabe <mccabe at ucar.edu>
wrote:
>
> > Hi Guang Ping,
> >
> > I copied your config file and made some changes. The new file can
be
> found
> > on hera here:
> >
> > /home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8.conf
> >
> > Using METplus 4.1.0-beta1, you can set TC_PAIRS_RUN_ONCE = True to
loop
> > over the init times even with LOOP_ORDER = processes.
> >
> > I noticed that you have TC_PAIRS_REFORMAT_DECK = yes, which isn't
> > necessary because tc_pairs can read the input files without
reformatting.
> > That can be unset or set to False/no.
> >
> > TC_PAIRS_REFORMAT_DECK = no
> >
> > I misread your email about trying to use the wildcard for the
model. This
> > won't work as you expect because the output template will not know
how to
> > name the output file with the correct model. Each model will
overwrite
> the
> > output files. To get around this you can use
TC_PAIRS_CUSTOM_LOOP_LIST.
> > This will run the TCPairs wrapper once for each item in the custom
loop
> > list. Here you can specify the list of models (lowercase) and
reference
> the
> > {custom} template tag to find the files.
> >
> > TC_PAIRS_CUSTOM_LOOP_LIST = best, cmc, emx, gfso, nam, ngx, ukx
> >
> > TC_PAIRS_ADECK_TEMPLATE =
> >
>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
> > TC_PAIRS_BDECK_TEMPLATE =
> >
>
{date?fmt=%Y%m}/b{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
> > TC_PAIRS_OUTPUT_TEMPLATE =
> >
>
{date?fmt=%Y%m}/{basin?fmt=%s}q{date?fmt=%Y%m%d%H}.{custom}.{cyclone?fmt=%s}
> >
> > I removed the MODEL values (you had it twice in your config file)
because
> > we don't need to filter by the model if we are reading all of them
in
> > individually.
> >
> > Using this config file I was able to run TCPairs for each model
> processing
> > each for all 3 months and creating output files from tc_pairs
here:
> >
> > /scratch1/BMC/dtc/George.Mccabe/tcmpr-test2/tc_pairs
> >
> > The error in your log file from TCMPRPlotter mentions that you do
not
> have
> > -bmodel set in the filter options. When I ran it, I got an error
with a
> > command that appears to list the files which leads me to think
that there
> > are too many files in the tc_pairs directory to process at once. I
need
> to
> > stop working for today, but I will resume debugging this tomorrow
and see
> > what I can find. It may be the case that the plotter cannot handle
all 7
> > models over a 3 day period all at once. If this is the case, what
would
> be
> > useful to display as a subset of this data?
> >
> > Thanks,
> > George
> >
> >
> >
> > On Fri, Jul 23, 2021 at 11:46 AM Guang Lou - NOAA Affiliate via RT
<
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100602 >
> >>
> >> Hi George,
> >>
> >> Thanks for letting me know you have downloaded the files. I've
removed
> the
> >> files to save disk space.
> >> Per your suggestion, I was able to generate consecutive months of
paired
> >> tracks by changing a few lines
> >> and parameters in the configure file. I've also loaded the module
> >> "metplus/4.1.0-beta1".
> >>
> >> This was done by:
> >> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_pairs
> >> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c
> >> usecase_pairing_8.conf
> >>
> >> However, when I tried to generate plots from the just generated
pairs,
> it
> >> ran into error:
> >> in /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_pairs
> >> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c
usecase_8.conf
> >>
> >> where I changed the following to:
> >> LOOP_ORDER = processes
> >> PROCESS_LIST = TCMPRPlotter
> >> error messages are:
> >> 07/23 17:38:51.477 metplus INFO: Check the logfile for more
information
> on
> >> why it failed:
> >>
> >>
> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
> >> 07/23 17:38:51.478 metplus INFO: Plotting complete
> >> 07/23 17:38:51.481 metplus.TCMPRPlotter (met_util.py:171) ERROR:
> >> TCMPRPlotter had 9 errors.
> >> 07/23 17:38:51.481 metplus INFO:
> >>
> >>
> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/metplus_final.conf:
> >> write metplus.conf here
> >> 07/23 17:38:51.486 metplus (met_util.py:208) ERROR: METplus has
finished
> >> running but had 9 errors.
> >> 07/23 17:38:51.486 metplus INFO: Check the log file for more
> information:
> >>
> >>
> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
> >> -----------------------------------------------------------
> >> The wild card in the place of
> >> TC_PAIRS_ADECK_TEMPLATE =
> >>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.nam.{cyclone?fmt=%s}
> >> to
> >> TC_PAIRS_ADECK_TEMPLATE =
> >>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> >> still does not work.
> >>
> >> Thanks,
> >> Guang Ping
> >>
> >> On Wed, Jul 21, 2021 at 2:41 PM George McCabe via RT
<met_help at ucar.edu
> >
> >> wrote:
> >>
> >> > Hi Guang Ping,
> >> >
> >> > I have finished downloading the GFS NetCDF files from Hera so
you can
> >> > remove them. Thanks again for providing that data.
> >> >
> >> > - George
> >> >
> >> > On Tue, Jul 20, 2021 at 3:35 PM George McCabe <mccabe at ucar.edu>
> wrote:
> >> >
> >> > > Hi Guang Ping,
> >> > >
> >> > > I mistyped in my previous email. The variable to set to
"times" is
> >> > > LOOP_ORDER not LOOP_BY.
> >> > >
> >> > > LOOP_ORDER = times
> >> > > INIT_INCREMENT = 1m
> >> > >
> >> > > Sorry for the confusion.
> >> > >
> >> > > - George
> >> > >
> >> > > On Tue, Jul 20, 2021 at 3:21 PM George McCabe
<mccabe at ucar.edu>
> >> wrote:
> >> > >
> >> > >> Hi Guang Ping,
> >> > >>
> >> > >> Thank you for providing the NetCDF files. I am currently
> downloading
> >> one
> >> > >> of the files but it is going very slowly.
> >> > >>
> >> > >> In regards to your use case, the logic for TCPairs wrapper
is a
> >> little
> >> > >> strange. It behaves differently based on the value of
LOOP_BY. If
> >> > LOOP_BY =
> >> > >> processes, then it will run once using the INIT_BEG time as
the run
> >> > time to
> >> > >> find input files. If LOOP_BY = times, then it will loop over
each
> run
> >> > time
> >> > >> specified between INIT_BEG and INIT_END by INIT_INCREMENT
and find
> >> > files to
> >> > >> process for each run time.
> >> > >>
> >> > >> If you change your configuration file to include LOOP_BY =
times,
> >> then
> >> > it
> >> > >> should look over each run time and process for each. It
looks like
> >> your
> >> > >> INIT_INCREMENT is set to 21600 or 6 hours, so it will run
every 6
> >> hours
> >> > >> from 20200201 to 2020331. If you want to run once per month,
you
> will
> >> > need
> >> > >> to set INIT_INCREMENT = 1m so that it will increment by 1
month for
> >> each
> >> > >> iteration.
> >> > >>
> >> > >> The caveat is that TCMPRPlotter will only run if LOOP_BY =
> processes
> >> and
> >> > >> will produce an error if LOOP_BY = times. This is because we
> >> typically
> >> > want
> >> > >> to process all of the data that is available for all run
times,
> then
> >> run
> >> > >> TCMPRPlotter so it can read all of the data. For now, to
handle
> this
> >> use
> >> > >> case you will have to call run_metplus.py 2 separate times,
once
> >> calling
> >> > >> TCPairs with LOOP_BY = times and once calling TCMPRPlotter
with
> >> LOOP_BY
> >> > =
> >> > >> processes.
> >> > >>
> >> > >> I am going to add a configuration option to TCPairs wrapper
so that
> >> you
> >> > >> can loop over the run times when LOOP_BY = processes is set
if
> >> desired.
> >> > I
> >> > >> will try to get this change into METplus 4.1.0-beta1 so you
can
> test
> >> the
> >> > >> change and make sure it works as expected. I can let you
know when
> >> that
> >> > is
> >> > >> available.
> >> > >>
> >> > >> Thanks,
> >> > >> George
> >> > >>
> >> > >> On Tue, Jul 20, 2021 at 1:27 PM Guang Lou - NOAA Affiliate
via RT <
> >> > >> met_help at ucar.edu> wrote:
> >> > >>
> >> > >>> Hi John,
> >> > >>>
> >> > >>> Thanks for forwarding George's email answer to me. I
created a
> test
> >> > case
> >> > >>> that does the tcpairs only on Hera:
> >> > >>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_test3
> >> > >>> Command line:
> >> > >>> run_metplus.py -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf
-c
> >> > >>> usecase_8.conf
> >> > >>> where PROCESS_LIST = TCPairs
> >> > >>>
> >> > >>> It generated the month of 202002, that is because I changed
> >> > >>> INIT_BEG = 20200101
> >> > >>> INIT_END = 20200331
> >> > >>> to
> >> > >>> INIT_BEG = 20200201
> >> > >>> INIT_END = 20200331
> >> > >>>
> >> > >>> If I use "INIT_BEG = 20200101,INIT_END = 20200331" it will
> generate
> >> > >>> 20200101 only.
> >> > >>>
> >> > >>> The wild card does not work either for the model's place:
> >> > >>> TC_PAIRS_ADECK_TEMPLATE =
> >> > >>>
> {date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> >> > >>> I populated adeck and bdeck on
> >> > >>>
> >> > >>>
> >> >
> >>
> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/METplus/METplus-
4.0_sample_data/model_applications/medium_range/tracks_all
> >> > >>> (each subdirectory contains all three months' decks.)
> >> > >>>
> >> > >>> I was wondering if you could copy my directory,
> >> > >>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_test3
> >> > >>> and test:
> >> > >>> 1. It can generate multiple months instead of just one
month and
> one
> >> > >>> model.
> >> > >>> 2. The wild card for models in the DECK_TEMPATE for
multiple
> models.
> >> > >>>
> >> > >>> George, I'm downloading NetCDF model data to Hera:
> >> > >>>
> /scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071300_nc.tar
> >> > >>>
> /scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071306_nc.tar
> >> > >>>
> >> > >>> Please let me know after you downloaded the files and I
will have
> to
> >> > >>> remove
> >> > >>> them due to tight disk space.
> >> > >>>
> >> > >>> Thanks,
> >> > >>> Guang Ping
> >> > >>>
> >> > >>> On Mon, Jul 19, 2021 at 5:01 PM John Halley Gotway via RT <
> >> > >>> met_help at ucar.edu>
> >> > >>> wrote:
> >> > >>>
> >> > >>> > Hi Guang Ping,
> >> > >>> >
> >> > >>> > The command that you ran here:
> >> > >>> >
> >> > >>> >
> >> > >>> >
> >> > >>>
> >> >
> >>
>
/home/George.Mccabe/scratch1_George.Mccabe/METplus.feature_926/ush/run_metplus.py
> >> > >>> > -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c
Plotter_8.conf
> >> > >>> >
> >> > >>> > only runs TCMPRPlotter. This is because PROCESS_LIST is
defined
> in
> >> > both
> >> > >>> > Plotter_fcstGFS_obsGFS_RPlotting_8.conf and
Plotter_8.conf.
> Since
> >> > >>> > Plotter_8.conf comes last, then the value for
PROCESS_LIST from
> >> that
> >> > >>> file
> >> > >>> > is used, which only has TCMPRPlotter listed. The plotter
is
> >> reading
> >> > >>> all of
> >> > >>> > the available files generated from TCPairs and since that
> process
> >> is
> >> > >>> not
> >> > >>> > running again with the new time range, the data that is
read
> only
> >> > >>> covers
> >> > >>> > the first month.
> >> > >>> >
> >> > >>> > We are planning on releasing METplus 4.1.0-beta1 this
week.
> >> However,
> >> > >>> if I
> >> > >>> > understand what you are trying to do here, it shouldn't
require
> >> the
> >> > >>> changes
> >> > >>> > described in Molly's issue. It looks like you want to run
> TCPairs
> >> > once
> >> > >>> and
> >> > >>> > filter the data based on the init and valid times. While
this
> >> > >>> configuration
> >> > >>> > should achieve what you are trying to do, the following
> >> configuration
> >> > >>> is
> >> > >>> > more explicit to what you are actually running:
> >> > >>> >
> >> > >>> > # run once using 20200101 to substitute the init time in
your
> >> string
> >> > >>> > substitution, i.e. TC_PAIRS_ADECK_TEMPLATE =
> >> > >>> >
> >>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> >> > >>> > INIT_BEG = 20200101
> >> > >>> > INIT_END = 20200101
> >> > >>> >
> >> > >>> > # filter the init times of the lines read from the input
files
> in
> >> the
> >> > >>> > TCPairs configuration file
> >> > >>> > TC_PAIRS_INIT_BEG = 20200101
> >> > >>> > TC_PAIRS_INIT_END = 20200331
> >> > >>> >
> >> > >>> > # filter the valid times of the lines read from the input
files
> in
> >> > the
> >> > >>> > TCPairs configuration file
> >> > >>> > TC_PAIRS_VALID_BEG = 20200101
> >> > >>> > TC_PAIRS_VALID_END = 20200331
> >> > >>> >
> >> > >>> > If any of TC_PAIRS_[INIT/VALID]_[BEG/END] are not set,
then the
> >> > >>> > corresponding [INIT/VALID]_[BEG/END] values are used.
This is
> >> done to
> >> > >>> > preserve backwards compatibility for old METplus
configurations.
> >> The
> >> > >>> above
> >> > >>> > settings are more explicit as to what is actually be
configured.
> >> > >>> >
> >> > >>> > You may want to consider either clearing your output
directory
> or
> >> > >>> creating
> >> > >>> > a new output directory for a new run (especially if the
time
> range
> >> > you
> >> > >>> are
> >> > >>> > using has changed) to ensure that the files you are
generated
> are
> >> > from
> >> > >>> that
> >> > >>> > run and not from a previous run. Doing this would have
alerted
> you
> >> > that
> >> > >>> > TCPairs was not run in this case.
> >> > >>> >
> >> > >>> > Let me know if you have any other issues getting this
running.
> >> > >>> >
> >> > >>> > FYI:
> >> > >>> > We switched from providing support through this email
address to
> >> > >>> providing
> >> > >>> > support through GitHub Discussions. In the future, you
will need
> >> to
> >> > >>> create
> >> > >>> > a free GitHub account if you don’t have one already and
post
> your
> >> > >>> questions
> >> > >>> > to the METplus Components Discussion Forum:
> >> > >>> > https://github.com/dtcenter/METplus/discussions
> >> > >>> > To post a new question, click on the green button in the
upper
> >> right
> >> > >>> corner
> >> > >>> > of the page labeled "New discussion" and select
"Incoming" as
> the
> >> > >>> category.
> >> > >>> >
> >> > >>> > Thanks,
> >> > >>> > George
> >> > >>> >
> >> > >>> >
> >> > >>>
> >> > >>>
> >> > >>
> >> > >> --
> >> > >> George McCabe - Software Engineer III
> >> > >> National Center for Atmospheric Research
> >> > >> Research Applications Laboratory
> >> > >> 303-497-2768
> >> > >> ---
> >> > >> My working day may not be your working day. Please do not
feel
> >> obliged
> >> > to
> >> > >> reply to this email outside of your normal working hours.
> >> > >>
> >> > >
> >> > >
> >> > > --
> >> > > George McCabe - Software Engineer III
> >> > > National Center for Atmospheric Research
> >> > > Research Applications Laboratory
> >> > > 303-497-2768
> >> > > ---
> >> > > My working day may not be your working day. Please do not
feel
> >> obliged to
> >> > > reply to this email outside of your normal working hours.
> >> > >
> >> >
> >> >
> >> > --
> >> > George McCabe - Software Engineer III
> >> > National Center for Atmospheric Research
> >> > Research Applications Laboratory
> >> > 303-497-2768
> >> > ---
> >> > My working day may not be your working day. Please do not feel
obliged
> >> to
> >> > reply to this email outside of your normal working hours.
> >> >
> >> >
> >>
> >>
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: tcpairs
From: George McCabe
Time: Wed Jul 28 10:02:59 2021
Hi Guang Ping,
OK, that is good to know. I have a few meetings this morning but I
will try
to update your configuration file to set this up.
- George
On Wed, Jul 28, 2021 at 9:46 AM Guang Lou - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
> Hi George,
>
> The verification should be against self analysis. That is GFS adeck
against
> GFS bdeck, CMC adeck against CMC bdeck, and so on and so forth.
>
> Thanks,
> Guang Ping
>
> On Tue, Jul 27, 2021 at 2:33 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Guang Ping,
> >
> > I looked at this use case more today. I modified the use case to
read a
> > single month into TCMPRPlotter because it cannot handle all of the
files
> at
> > once. It looks like you need to set the -bmodel in
TCMPR_PLOTTER_FILTER
> to
> > specify the Bdeck to compare the models to. However, each of the
output
> > files from tc_pairs use the same model for adeck and bdeck, so the
AMODEL
> > and BMODEL values in those files are the same, i.e. both GFSO. If
you
> want
> > to compare all of the models to a single bdeck, you will have to
> configure
> > TCPairs wrapper to use the same bdeck for each model with
> > TC_PAIRS_BDECK_TEMPLATE. I tried running using best for the bdeck
and it
> > produced no pairs. I am rerunning now using GFSO.
> >
> > - George
> >
> > On Mon, Jul 26, 2021 at 5:57 PM George McCabe <mccabe at ucar.edu>
wrote:
> >
> > > Hi Guang Ping,
> > >
> > > I copied your config file and made some changes. The new file
can be
> > found
> > > on hera here:
> > >
> > > /home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8.conf
> > >
> > > Using METplus 4.1.0-beta1, you can set TC_PAIRS_RUN_ONCE = True
to loop
> > > over the init times even with LOOP_ORDER = processes.
> > >
> > > I noticed that you have TC_PAIRS_REFORMAT_DECK = yes, which
isn't
> > > necessary because tc_pairs can read the input files without
> reformatting.
> > > That can be unset or set to False/no.
> > >
> > > TC_PAIRS_REFORMAT_DECK = no
> > >
> > > I misread your email about trying to use the wildcard for the
model.
> This
> > > won't work as you expect because the output template will not
know how
> to
> > > name the output file with the correct model. Each model will
overwrite
> > the
> > > output files. To get around this you can use
TC_PAIRS_CUSTOM_LOOP_LIST.
> > > This will run the TCPairs wrapper once for each item in the
custom loop
> > > list. Here you can specify the list of models (lowercase) and
reference
> > the
> > > {custom} template tag to find the files.
> > >
> > > TC_PAIRS_CUSTOM_LOOP_LIST = best, cmc, emx, gfso, nam, ngx, ukx
> > >
> > > TC_PAIRS_ADECK_TEMPLATE =
> > >
> >
>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
> > > TC_PAIRS_BDECK_TEMPLATE =
> > >
> >
>
{date?fmt=%Y%m}/b{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
> > > TC_PAIRS_OUTPUT_TEMPLATE =
> > >
> >
>
{date?fmt=%Y%m}/{basin?fmt=%s}q{date?fmt=%Y%m%d%H}.{custom}.{cyclone?fmt=%s}
> > >
> > > I removed the MODEL values (you had it twice in your config
file)
> because
> > > we don't need to filter by the model if we are reading all of
them in
> > > individually.
> > >
> > > Using this config file I was able to run TCPairs for each model
> > processing
> > > each for all 3 months and creating output files from tc_pairs
here:
> > >
> > > /scratch1/BMC/dtc/George.Mccabe/tcmpr-test2/tc_pairs
> > >
> > > The error in your log file from TCMPRPlotter mentions that you
do not
> > have
> > > -bmodel set in the filter options. When I ran it, I got an error
with a
> > > command that appears to list the files which leads me to think
that
> there
> > > are too many files in the tc_pairs directory to process at once.
I need
> > to
> > > stop working for today, but I will resume debugging this
tomorrow and
> see
> > > what I can find. It may be the case that the plotter cannot
handle all
> 7
> > > models over a 3 day period all at once. If this is the case,
what would
> > be
> > > useful to display as a subset of this data?
> > >
> > > Thanks,
> > > George
> > >
> > >
> > >
> > > On Fri, Jul 23, 2021 at 11:46 AM Guang Lou - NOAA Affiliate via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100602
>
> > >>
> > >> Hi George,
> > >>
> > >> Thanks for letting me know you have downloaded the files. I've
removed
> > the
> > >> files to save disk space.
> > >> Per your suggestion, I was able to generate consecutive months
of
> paired
> > >> tracks by changing a few lines
> > >> and parameters in the configure file. I've also loaded the
module
> > >> "metplus/4.1.0-beta1".
> > >>
> > >> This was done by:
> > >> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_pairs
> > >> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c
> > >> usecase_pairing_8.conf
> > >>
> > >> However, when I tried to generate plots from the just generated
pairs,
> > it
> > >> ran into error:
> > >> in /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs
> > >> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c
usecase_8.conf
> > >>
> > >> where I changed the following to:
> > >> LOOP_ORDER = processes
> > >> PROCESS_LIST = TCMPRPlotter
> > >> error messages are:
> > >> 07/23 17:38:51.477 metplus INFO: Check the logfile for more
> information
> > on
> > >> why it failed:
> > >>
> > >>
> >
> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
> > >> 07/23 17:38:51.478 metplus INFO: Plotting complete
> > >> 07/23 17:38:51.481 metplus.TCMPRPlotter (met_util.py:171)
ERROR:
> > >> TCMPRPlotter had 9 errors.
> > >> 07/23 17:38:51.481 metplus INFO:
> > >>
> > >>
> >
> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/metplus_final.conf:
> > >> write metplus.conf here
> > >> 07/23 17:38:51.486 metplus (met_util.py:208) ERROR: METplus has
> finished
> > >> running but had 9 errors.
> > >> 07/23 17:38:51.486 metplus INFO: Check the log file for more
> > information:
> > >>
> > >>
> >
> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
> > >> -----------------------------------------------------------
> > >> The wild card in the place of
> > >> TC_PAIRS_ADECK_TEMPLATE =
> > >>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.nam.{cyclone?fmt=%s}
> > >> to
> > >> TC_PAIRS_ADECK_TEMPLATE =
> > >>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> > >> still does not work.
> > >>
> > >> Thanks,
> > >> Guang Ping
> > >>
> > >> On Wed, Jul 21, 2021 at 2:41 PM George McCabe via RT <
> met_help at ucar.edu
> > >
> > >> wrote:
> > >>
> > >> > Hi Guang Ping,
> > >> >
> > >> > I have finished downloading the GFS NetCDF files from Hera so
you
> can
> > >> > remove them. Thanks again for providing that data.
> > >> >
> > >> > - George
> > >> >
> > >> > On Tue, Jul 20, 2021 at 3:35 PM George McCabe
<mccabe at ucar.edu>
> > wrote:
> > >> >
> > >> > > Hi Guang Ping,
> > >> > >
> > >> > > I mistyped in my previous email. The variable to set to
"times" is
> > >> > > LOOP_ORDER not LOOP_BY.
> > >> > >
> > >> > > LOOP_ORDER = times
> > >> > > INIT_INCREMENT = 1m
> > >> > >
> > >> > > Sorry for the confusion.
> > >> > >
> > >> > > - George
> > >> > >
> > >> > > On Tue, Jul 20, 2021 at 3:21 PM George McCabe
<mccabe at ucar.edu>
> > >> wrote:
> > >> > >
> > >> > >> Hi Guang Ping,
> > >> > >>
> > >> > >> Thank you for providing the NetCDF files. I am currently
> > downloading
> > >> one
> > >> > >> of the files but it is going very slowly.
> > >> > >>
> > >> > >> In regards to your use case, the logic for TCPairs wrapper
is a
> > >> little
> > >> > >> strange. It behaves differently based on the value of
LOOP_BY. If
> > >> > LOOP_BY =
> > >> > >> processes, then it will run once using the INIT_BEG time
as the
> run
> > >> > time to
> > >> > >> find input files. If LOOP_BY = times, then it will loop
over each
> > run
> > >> > time
> > >> > >> specified between INIT_BEG and INIT_END by INIT_INCREMENT
and
> find
> > >> > files to
> > >> > >> process for each run time.
> > >> > >>
> > >> > >> If you change your configuration file to include LOOP_BY =
times,
> > >> then
> > >> > it
> > >> > >> should look over each run time and process for each. It
looks
> like
> > >> your
> > >> > >> INIT_INCREMENT is set to 21600 or 6 hours, so it will run
every 6
> > >> hours
> > >> > >> from 20200201 to 2020331. If you want to run once per
month, you
> > will
> > >> > need
> > >> > >> to set INIT_INCREMENT = 1m so that it will increment by 1
month
> for
> > >> each
> > >> > >> iteration.
> > >> > >>
> > >> > >> The caveat is that TCMPRPlotter will only run if LOOP_BY =
> > processes
> > >> and
> > >> > >> will produce an error if LOOP_BY = times. This is because
we
> > >> typically
> > >> > want
> > >> > >> to process all of the data that is available for all run
times,
> > then
> > >> run
> > >> > >> TCMPRPlotter so it can read all of the data. For now, to
handle
> > this
> > >> use
> > >> > >> case you will have to call run_metplus.py 2 separate
times, once
> > >> calling
> > >> > >> TCPairs with LOOP_BY = times and once calling TCMPRPlotter
with
> > >> LOOP_BY
> > >> > =
> > >> > >> processes.
> > >> > >>
> > >> > >> I am going to add a configuration option to TCPairs
wrapper so
> that
> > >> you
> > >> > >> can loop over the run times when LOOP_BY = processes is
set if
> > >> desired.
> > >> > I
> > >> > >> will try to get this change into METplus 4.1.0-beta1 so
you can
> > test
> > >> the
> > >> > >> change and make sure it works as expected. I can let you
know
> when
> > >> that
> > >> > is
> > >> > >> available.
> > >> > >>
> > >> > >> Thanks,
> > >> > >> George
> > >> > >>
> > >> > >> On Tue, Jul 20, 2021 at 1:27 PM Guang Lou - NOAA Affiliate
via
> RT <
> > >> > >> met_help at ucar.edu> wrote:
> > >> > >>
> > >> > >>> Hi John,
> > >> > >>>
> > >> > >>> Thanks for forwarding George's email answer to me. I
created a
> > test
> > >> > case
> > >> > >>> that does the tcpairs only on Hera:
> > >> > >>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_test3
> > >> > >>> Command line:
> > >> > >>> run_metplus.py -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf
-c
> > >> > >>> usecase_8.conf
> > >> > >>> where PROCESS_LIST = TCPairs
> > >> > >>>
> > >> > >>> It generated the month of 202002, that is because I
changed
> > >> > >>> INIT_BEG = 20200101
> > >> > >>> INIT_END = 20200331
> > >> > >>> to
> > >> > >>> INIT_BEG = 20200201
> > >> > >>> INIT_END = 20200331
> > >> > >>>
> > >> > >>> If I use "INIT_BEG = 20200101,INIT_END = 20200331" it
will
> > generate
> > >> > >>> 20200101 only.
> > >> > >>>
> > >> > >>> The wild card does not work either for the model's place:
> > >> > >>> TC_PAIRS_ADECK_TEMPLATE =
> > >> > >>>
> >
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> > >> > >>> I populated adeck and bdeck on
> > >> > >>>
> > >> > >>>
> > >> >
> > >>
> >
> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/METplus/METplus-
4.0_sample_data/model_applications/medium_range/tracks_all
> > >> > >>> (each subdirectory contains all three months' decks.)
> > >> > >>>
> > >> > >>> I was wondering if you could copy my directory,
> > >> > >>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_test3
> > >> > >>> and test:
> > >> > >>> 1. It can generate multiple months instead of just one
month and
> > one
> > >> > >>> model.
> > >> > >>> 2. The wild card for models in the DECK_TEMPATE for
multiple
> > models.
> > >> > >>>
> > >> > >>> George, I'm downloading NetCDF model data to Hera:
> > >> > >>>
> >
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071300_nc.tar
> > >> > >>>
> >
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071306_nc.tar
> > >> > >>>
> > >> > >>> Please let me know after you downloaded the files and I
will
> have
> > to
> > >> > >>> remove
> > >> > >>> them due to tight disk space.
> > >> > >>>
> > >> > >>> Thanks,
> > >> > >>> Guang Ping
> > >> > >>>
> > >> > >>> On Mon, Jul 19, 2021 at 5:01 PM John Halley Gotway via RT
<
> > >> > >>> met_help at ucar.edu>
> > >> > >>> wrote:
> > >> > >>>
> > >> > >>> > Hi Guang Ping,
> > >> > >>> >
> > >> > >>> > The command that you ran here:
> > >> > >>> >
> > >> > >>> >
> > >> > >>> >
> > >> > >>>
> > >> >
> > >>
> >
>
/home/George.Mccabe/scratch1_George.Mccabe/METplus.feature_926/ush/run_metplus.py
> > >> > >>> > -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c
Plotter_8.conf
> > >> > >>> >
> > >> > >>> > only runs TCMPRPlotter. This is because PROCESS_LIST is
> defined
> > in
> > >> > both
> > >> > >>> > Plotter_fcstGFS_obsGFS_RPlotting_8.conf and
Plotter_8.conf.
> > Since
> > >> > >>> > Plotter_8.conf comes last, then the value for
PROCESS_LIST
> from
> > >> that
> > >> > >>> file
> > >> > >>> > is used, which only has TCMPRPlotter listed. The
plotter is
> > >> reading
> > >> > >>> all of
> > >> > >>> > the available files generated from TCPairs and since
that
> > process
> > >> is
> > >> > >>> not
> > >> > >>> > running again with the new time range, the data that is
read
> > only
> > >> > >>> covers
> > >> > >>> > the first month.
> > >> > >>> >
> > >> > >>> > We are planning on releasing METplus 4.1.0-beta1 this
week.
> > >> However,
> > >> > >>> if I
> > >> > >>> > understand what you are trying to do here, it shouldn't
> require
> > >> the
> > >> > >>> changes
> > >> > >>> > described in Molly's issue. It looks like you want to
run
> > TCPairs
> > >> > once
> > >> > >>> and
> > >> > >>> > filter the data based on the init and valid times.
While this
> > >> > >>> configuration
> > >> > >>> > should achieve what you are trying to do, the following
> > >> configuration
> > >> > >>> is
> > >> > >>> > more explicit to what you are actually running:
> > >> > >>> >
> > >> > >>> > # run once using 20200101 to substitute the init time
in your
> > >> string
> > >> > >>> > substitution, i.e. TC_PAIRS_ADECK_TEMPLATE =
> > >> > >>> >
> > >>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> > >> > >>> > INIT_BEG = 20200101
> > >> > >>> > INIT_END = 20200101
> > >> > >>> >
> > >> > >>> > # filter the init times of the lines read from the
input files
> > in
> > >> the
> > >> > >>> > TCPairs configuration file
> > >> > >>> > TC_PAIRS_INIT_BEG = 20200101
> > >> > >>> > TC_PAIRS_INIT_END = 20200331
> > >> > >>> >
> > >> > >>> > # filter the valid times of the lines read from the
input
> files
> > in
> > >> > the
> > >> > >>> > TCPairs configuration file
> > >> > >>> > TC_PAIRS_VALID_BEG = 20200101
> > >> > >>> > TC_PAIRS_VALID_END = 20200331
> > >> > >>> >
> > >> > >>> > If any of TC_PAIRS_[INIT/VALID]_[BEG/END] are not set,
then
> the
> > >> > >>> > corresponding [INIT/VALID]_[BEG/END] values are used.
This is
> > >> done to
> > >> > >>> > preserve backwards compatibility for old METplus
> configurations.
> > >> The
> > >> > >>> above
> > >> > >>> > settings are more explicit as to what is actually be
> configured.
> > >> > >>> >
> > >> > >>> > You may want to consider either clearing your output
directory
> > or
> > >> > >>> creating
> > >> > >>> > a new output directory for a new run (especially if the
time
> > range
> > >> > you
> > >> > >>> are
> > >> > >>> > using has changed) to ensure that the files you are
generated
> > are
> > >> > from
> > >> > >>> that
> > >> > >>> > run and not from a previous run. Doing this would have
alerted
> > you
> > >> > that
> > >> > >>> > TCPairs was not run in this case.
> > >> > >>> >
> > >> > >>> > Let me know if you have any other issues getting this
running.
> > >> > >>> >
> > >> > >>> > FYI:
> > >> > >>> > We switched from providing support through this email
address
> to
> > >> > >>> providing
> > >> > >>> > support through GitHub Discussions. In the future, you
will
> need
> > >> to
> > >> > >>> create
> > >> > >>> > a free GitHub account if you don’t have one already and
post
> > your
> > >> > >>> questions
> > >> > >>> > to the METplus Components Discussion Forum:
> > >> > >>> > https://github.com/dtcenter/METplus/discussions
> > >> > >>> > To post a new question, click on the green button in
the upper
> > >> right
> > >> > >>> corner
> > >> > >>> > of the page labeled "New discussion" and select
"Incoming" as
> > the
> > >> > >>> category.
> > >> > >>> >
> > >> > >>> > Thanks,
> > >> > >>> > George
> > >> > >>> >
> > >> > >>> >
> > >> > >>>
> > >> > >>>
> > >> > >>
> > >> > >> --
> > >> > >> George McCabe - Software Engineer III
> > >> > >> National Center for Atmospheric Research
> > >> > >> Research Applications Laboratory
> > >> > >> 303-497-2768
> > >> > >> ---
> > >> > >> My working day may not be your working day. Please do not
feel
> > >> obliged
> > >> > to
> > >> > >> reply to this email outside of your normal working hours.
> > >> > >>
> > >> > >
> > >> > >
> > >> > > --
> > >> > > George McCabe - Software Engineer III
> > >> > > National Center for Atmospheric Research
> > >> > > Research Applications Laboratory
> > >> > > 303-497-2768
> > >> > > ---
> > >> > > My working day may not be your working day. Please do not
feel
> > >> obliged to
> > >> > > reply to this email outside of your normal working hours.
> > >> > >
> > >> >
> > >> >
> > >> > --
> > >> > George McCabe - Software Engineer III
> > >> > National Center for Atmospheric Research
> > >> > Research Applications Laboratory
> > >> > 303-497-2768
> > >> > ---
> > >> > My working day may not be your working day. Please do not
feel
> obliged
> > >> to
> > >> > reply to this email outside of your normal working hours.
> > >> >
> > >> >
> > >>
> > >>
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: tcpairs
From: George McCabe
Time: Wed Jul 28 14:32:04 2021
Hi Guang Ping,
I updated the config file to process each model separately. Here is
the
config file on hera:
/home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8_test.conf
The output from my latest run can be found in here:
/scratch1/BMC/dtc/George.Mccabe/tcmpr-test4/tcmpr_plots
There is a sub-directory for each model. Please take a look at the
output
files to see if they look OK. The config file could be cleaned up a
little
bit if I add support for TCMPR_PLOTTER_CUSTOM_LOOP_LIST or support for
reading the instance name from the TCMPRPlotter instances. I will make
a
GitHub issue to implement those changes as they are relatively easy
additions and would make the wrapper consistent with the other
wrappers.
Thanks,
George
On Wed, Jul 28, 2021 at 10:02 AM George McCabe <mccabe at ucar.edu>
wrote:
> Hi Guang Ping,
>
> OK, that is good to know. I have a few meetings this morning but I
will
> try to update your configuration file to set this up.
>
> - George
>
> On Wed, Jul 28, 2021 at 9:46 AM Guang Lou - NOAA Affiliate via RT <
> met_help at ucar.edu> wrote:
>
>> Hi George,
>>
>> The verification should be against self analysis. That is GFS adeck
>> against
>> GFS bdeck, CMC adeck against CMC bdeck, and so on and so forth.
>>
>> Thanks,
>> Guang Ping
>>
>> On Tue, Jul 27, 2021 at 2:33 PM George McCabe via RT
<met_help at ucar.edu>
>> wrote:
>>
>> > Hi Guang Ping,
>> >
>> > I looked at this use case more today. I modified the use case to
read a
>> > single month into TCMPRPlotter because it cannot handle all of
the
>> files at
>> > once. It looks like you need to set the -bmodel in
TCMPR_PLOTTER_FILTER
>> to
>> > specify the Bdeck to compare the models to. However, each of the
output
>> > files from tc_pairs use the same model for adeck and bdeck, so
the
>> AMODEL
>> > and BMODEL values in those files are the same, i.e. both GFSO. If
you
>> want
>> > to compare all of the models to a single bdeck, you will have to
>> configure
>> > TCPairs wrapper to use the same bdeck for each model with
>> > TC_PAIRS_BDECK_TEMPLATE. I tried running using best for the bdeck
and it
>> > produced no pairs. I am rerunning now using GFSO.
>> >
>> > - George
>> >
>> > On Mon, Jul 26, 2021 at 5:57 PM George McCabe <mccabe at ucar.edu>
wrote:
>> >
>> > > Hi Guang Ping,
>> > >
>> > > I copied your config file and made some changes. The new file
can be
>> > found
>> > > on hera here:
>> > >
>> > > /home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8.conf
>> > >
>> > > Using METplus 4.1.0-beta1, you can set TC_PAIRS_RUN_ONCE = True
to
>> loop
>> > > over the init times even with LOOP_ORDER = processes.
>> > >
>> > > I noticed that you have TC_PAIRS_REFORMAT_DECK = yes, which
isn't
>> > > necessary because tc_pairs can read the input files without
>> reformatting.
>> > > That can be unset or set to False/no.
>> > >
>> > > TC_PAIRS_REFORMAT_DECK = no
>> > >
>> > > I misread your email about trying to use the wildcard for the
model.
>> This
>> > > won't work as you expect because the output template will not
know
>> how to
>> > > name the output file with the correct model. Each model will
overwrite
>> > the
>> > > output files. To get around this you can use
>> TC_PAIRS_CUSTOM_LOOP_LIST.
>> > > This will run the TCPairs wrapper once for each item in the
custom
>> loop
>> > > list. Here you can specify the list of models (lowercase) and
>> reference
>> > the
>> > > {custom} template tag to find the files.
>> > >
>> > > TC_PAIRS_CUSTOM_LOOP_LIST = best, cmc, emx, gfso, nam, ngx, ukx
>> > >
>> > > TC_PAIRS_ADECK_TEMPLATE =
>> > >
>> >
>>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
>> > > TC_PAIRS_BDECK_TEMPLATE =
>> > >
>> >
>>
{date?fmt=%Y%m}/b{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
>> > > TC_PAIRS_OUTPUT_TEMPLATE =
>> > >
>> >
>>
{date?fmt=%Y%m}/{basin?fmt=%s}q{date?fmt=%Y%m%d%H}.{custom}.{cyclone?fmt=%s}
>> > >
>> > > I removed the MODEL values (you had it twice in your config
file)
>> because
>> > > we don't need to filter by the model if we are reading all of
them in
>> > > individually.
>> > >
>> > > Using this config file I was able to run TCPairs for each model
>> > processing
>> > > each for all 3 months and creating output files from tc_pairs
here:
>> > >
>> > > /scratch1/BMC/dtc/George.Mccabe/tcmpr-test2/tc_pairs
>> > >
>> > > The error in your log file from TCMPRPlotter mentions that you
do not
>> > have
>> > > -bmodel set in the filter options. When I ran it, I got an
error with
>> a
>> > > command that appears to list the files which leads me to think
that
>> there
>> > > are too many files in the tc_pairs directory to process at
once. I
>> need
>> > to
>> > > stop working for today, but I will resume debugging this
tomorrow and
>> see
>> > > what I can find. It may be the case that the plotter cannot
handle
>> all 7
>> > > models over a 3 day period all at once. If this is the case,
what
>> would
>> > be
>> > > useful to display as a subset of this data?
>> > >
>> > > Thanks,
>> > > George
>> > >
>> > >
>> > >
>> > > On Fri, Jul 23, 2021 at 11:46 AM Guang Lou - NOAA Affiliate via
RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > >>
>> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100602
>
>> > >>
>> > >> Hi George,
>> > >>
>> > >> Thanks for letting me know you have downloaded the files. I've
>> removed
>> > the
>> > >> files to save disk space.
>> > >> Per your suggestion, I was able to generate consecutive months
of
>> paired
>> > >> tracks by changing a few lines
>> > >> and parameters in the configure file. I've also loaded the
module
>> > >> "metplus/4.1.0-beta1".
>> > >>
>> > >> This was done by:
>> > >> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_pairs
>> > >> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c
>> > >> usecase_pairing_8.conf
>> > >>
>> > >> However, when I tried to generate plots from the just
generated
>> pairs,
>> > it
>> > >> ran into error:
>> > >> in /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs
>> > >> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c
usecase_8.conf
>> > >>
>> > >> where I changed the following to:
>> > >> LOOP_ORDER = processes
>> > >> PROCESS_LIST = TCMPRPlotter
>> > >> error messages are:
>> > >> 07/23 17:38:51.477 metplus INFO: Check the logfile for more
>> information
>> > on
>> > >> why it failed:
>> > >>
>> > >>
>> >
>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
>> > >> 07/23 17:38:51.478 metplus INFO: Plotting complete
>> > >> 07/23 17:38:51.481 metplus.TCMPRPlotter (met_util.py:171)
ERROR:
>> > >> TCMPRPlotter had 9 errors.
>> > >> 07/23 17:38:51.481 metplus INFO:
>> > >>
>> > >>
>> >
>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/metplus_final.conf:
>> > >> write metplus.conf here
>> > >> 07/23 17:38:51.486 metplus (met_util.py:208) ERROR: METplus
has
>> finished
>> > >> running but had 9 errors.
>> > >> 07/23 17:38:51.486 metplus INFO: Check the log file for more
>> > information:
>> > >>
>> > >>
>> >
>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
>> > >> -----------------------------------------------------------
>> > >> The wild card in the place of
>> > >> TC_PAIRS_ADECK_TEMPLATE =
>> > >>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.nam.{cyclone?fmt=%s}
>> > >> to
>> > >> TC_PAIRS_ADECK_TEMPLATE =
>> > >>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
>> > >> still does not work.
>> > >>
>> > >> Thanks,
>> > >> Guang Ping
>> > >>
>> > >> On Wed, Jul 21, 2021 at 2:41 PM George McCabe via RT <
>> met_help at ucar.edu
>> > >
>> > >> wrote:
>> > >>
>> > >> > Hi Guang Ping,
>> > >> >
>> > >> > I have finished downloading the GFS NetCDF files from Hera
so you
>> can
>> > >> > remove them. Thanks again for providing that data.
>> > >> >
>> > >> > - George
>> > >> >
>> > >> > On Tue, Jul 20, 2021 at 3:35 PM George McCabe
<mccabe at ucar.edu>
>> > wrote:
>> > >> >
>> > >> > > Hi Guang Ping,
>> > >> > >
>> > >> > > I mistyped in my previous email. The variable to set to
"times"
>> is
>> > >> > > LOOP_ORDER not LOOP_BY.
>> > >> > >
>> > >> > > LOOP_ORDER = times
>> > >> > > INIT_INCREMENT = 1m
>> > >> > >
>> > >> > > Sorry for the confusion.
>> > >> > >
>> > >> > > - George
>> > >> > >
>> > >> > > On Tue, Jul 20, 2021 at 3:21 PM George McCabe
<mccabe at ucar.edu>
>> > >> wrote:
>> > >> > >
>> > >> > >> Hi Guang Ping,
>> > >> > >>
>> > >> > >> Thank you for providing the NetCDF files. I am currently
>> > downloading
>> > >> one
>> > >> > >> of the files but it is going very slowly.
>> > >> > >>
>> > >> > >> In regards to your use case, the logic for TCPairs
wrapper is a
>> > >> little
>> > >> > >> strange. It behaves differently based on the value of
LOOP_BY.
>> If
>> > >> > LOOP_BY =
>> > >> > >> processes, then it will run once using the INIT_BEG time
as the
>> run
>> > >> > time to
>> > >> > >> find input files. If LOOP_BY = times, then it will loop
over
>> each
>> > run
>> > >> > time
>> > >> > >> specified between INIT_BEG and INIT_END by INIT_INCREMENT
and
>> find
>> > >> > files to
>> > >> > >> process for each run time.
>> > >> > >>
>> > >> > >> If you change your configuration file to include LOOP_BY
=
>> times,
>> > >> then
>> > >> > it
>> > >> > >> should look over each run time and process for each. It
looks
>> like
>> > >> your
>> > >> > >> INIT_INCREMENT is set to 21600 or 6 hours, so it will run
every
>> 6
>> > >> hours
>> > >> > >> from 20200201 to 2020331. If you want to run once per
month, you
>> > will
>> > >> > need
>> > >> > >> to set INIT_INCREMENT = 1m so that it will increment by 1
month
>> for
>> > >> each
>> > >> > >> iteration.
>> > >> > >>
>> > >> > >> The caveat is that TCMPRPlotter will only run if LOOP_BY
=
>> > processes
>> > >> and
>> > >> > >> will produce an error if LOOP_BY = times. This is because
we
>> > >> typically
>> > >> > want
>> > >> > >> to process all of the data that is available for all run
times,
>> > then
>> > >> run
>> > >> > >> TCMPRPlotter so it can read all of the data. For now, to
handle
>> > this
>> > >> use
>> > >> > >> case you will have to call run_metplus.py 2 separate
times, once
>> > >> calling
>> > >> > >> TCPairs with LOOP_BY = times and once calling
TCMPRPlotter with
>> > >> LOOP_BY
>> > >> > =
>> > >> > >> processes.
>> > >> > >>
>> > >> > >> I am going to add a configuration option to TCPairs
wrapper so
>> that
>> > >> you
>> > >> > >> can loop over the run times when LOOP_BY = processes is
set if
>> > >> desired.
>> > >> > I
>> > >> > >> will try to get this change into METplus 4.1.0-beta1 so
you can
>> > test
>> > >> the
>> > >> > >> change and make sure it works as expected. I can let you
know
>> when
>> > >> that
>> > >> > is
>> > >> > >> available.
>> > >> > >>
>> > >> > >> Thanks,
>> > >> > >> George
>> > >> > >>
>> > >> > >> On Tue, Jul 20, 2021 at 1:27 PM Guang Lou - NOAA
Affiliate via
>> RT <
>> > >> > >> met_help at ucar.edu> wrote:
>> > >> > >>
>> > >> > >>> Hi John,
>> > >> > >>>
>> > >> > >>> Thanks for forwarding George's email answer to me. I
created a
>> > test
>> > >> > case
>> > >> > >>> that does the tcpairs only on Hera:
>> > >> > >>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_test3
>> > >> > >>> Command line:
>> > >> > >>> run_metplus.py -c
Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c
>> > >> > >>> usecase_8.conf
>> > >> > >>> where PROCESS_LIST = TCPairs
>> > >> > >>>
>> > >> > >>> It generated the month of 202002, that is because I
changed
>> > >> > >>> INIT_BEG = 20200101
>> > >> > >>> INIT_END = 20200331
>> > >> > >>> to
>> > >> > >>> INIT_BEG = 20200201
>> > >> > >>> INIT_END = 20200331
>> > >> > >>>
>> > >> > >>> If I use "INIT_BEG = 20200101,INIT_END = 20200331" it
will
>> > generate
>> > >> > >>> 20200101 only.
>> > >> > >>>
>> > >> > >>> The wild card does not work either for the model's
place:
>> > >> > >>> TC_PAIRS_ADECK_TEMPLATE =
>> > >> > >>>
>> >
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
>> > >> > >>> I populated adeck and bdeck on
>> > >> > >>>
>> > >> > >>>
>> > >> >
>> > >>
>> >
>> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/METplus/METplus-
4.0_sample_data/model_applications/medium_range/tracks_all
>> > >> > >>> (each subdirectory contains all three months' decks.)
>> > >> > >>>
>> > >> > >>> I was wondering if you could copy my directory,
>> > >> > >>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_test3
>> > >> > >>> and test:
>> > >> > >>> 1. It can generate multiple months instead of just one
month
>> and
>> > one
>> > >> > >>> model.
>> > >> > >>> 2. The wild card for models in the DECK_TEMPATE for
multiple
>> > models.
>> > >> > >>>
>> > >> > >>> George, I'm downloading NetCDF model data to Hera:
>> > >> > >>>
>> >
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071300_nc.tar
>> > >> > >>>
>> >
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071306_nc.tar
>> > >> > >>>
>> > >> > >>> Please let me know after you downloaded the files and I
will
>> have
>> > to
>> > >> > >>> remove
>> > >> > >>> them due to tight disk space.
>> > >> > >>>
>> > >> > >>> Thanks,
>> > >> > >>> Guang Ping
>> > >> > >>>
>> > >> > >>> On Mon, Jul 19, 2021 at 5:01 PM John Halley Gotway via
RT <
>> > >> > >>> met_help at ucar.edu>
>> > >> > >>> wrote:
>> > >> > >>>
>> > >> > >>> > Hi Guang Ping,
>> > >> > >>> >
>> > >> > >>> > The command that you ran here:
>> > >> > >>> >
>> > >> > >>> >
>> > >> > >>> >
>> > >> > >>>
>> > >> >
>> > >>
>> >
>>
/home/George.Mccabe/scratch1_George.Mccabe/METplus.feature_926/ush/run_metplus.py
>> > >> > >>> > -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c
Plotter_8.conf
>> > >> > >>> >
>> > >> > >>> > only runs TCMPRPlotter. This is because PROCESS_LIST
is
>> defined
>> > in
>> > >> > both
>> > >> > >>> > Plotter_fcstGFS_obsGFS_RPlotting_8.conf and
Plotter_8.conf.
>> > Since
>> > >> > >>> > Plotter_8.conf comes last, then the value for
PROCESS_LIST
>> from
>> > >> that
>> > >> > >>> file
>> > >> > >>> > is used, which only has TCMPRPlotter listed. The
plotter is
>> > >> reading
>> > >> > >>> all of
>> > >> > >>> > the available files generated from TCPairs and since
that
>> > process
>> > >> is
>> > >> > >>> not
>> > >> > >>> > running again with the new time range, the data that
is read
>> > only
>> > >> > >>> covers
>> > >> > >>> > the first month.
>> > >> > >>> >
>> > >> > >>> > We are planning on releasing METplus 4.1.0-beta1 this
week.
>> > >> However,
>> > >> > >>> if I
>> > >> > >>> > understand what you are trying to do here, it
shouldn't
>> require
>> > >> the
>> > >> > >>> changes
>> > >> > >>> > described in Molly's issue. It looks like you want to
run
>> > TCPairs
>> > >> > once
>> > >> > >>> and
>> > >> > >>> > filter the data based on the init and valid times.
While this
>> > >> > >>> configuration
>> > >> > >>> > should achieve what you are trying to do, the
following
>> > >> configuration
>> > >> > >>> is
>> > >> > >>> > more explicit to what you are actually running:
>> > >> > >>> >
>> > >> > >>> > # run once using 20200101 to substitute the init time
in your
>> > >> string
>> > >> > >>> > substitution, i.e. TC_PAIRS_ADECK_TEMPLATE =
>> > >> > >>> >
>> > >>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
>> > >> > >>> > INIT_BEG = 20200101
>> > >> > >>> > INIT_END = 20200101
>> > >> > >>> >
>> > >> > >>> > # filter the init times of the lines read from the
input
>> files
>> > in
>> > >> the
>> > >> > >>> > TCPairs configuration file
>> > >> > >>> > TC_PAIRS_INIT_BEG = 20200101
>> > >> > >>> > TC_PAIRS_INIT_END = 20200331
>> > >> > >>> >
>> > >> > >>> > # filter the valid times of the lines read from the
input
>> files
>> > in
>> > >> > the
>> > >> > >>> > TCPairs configuration file
>> > >> > >>> > TC_PAIRS_VALID_BEG = 20200101
>> > >> > >>> > TC_PAIRS_VALID_END = 20200331
>> > >> > >>> >
>> > >> > >>> > If any of TC_PAIRS_[INIT/VALID]_[BEG/END] are not set,
then
>> the
>> > >> > >>> > corresponding [INIT/VALID]_[BEG/END] values are used.
This is
>> > >> done to
>> > >> > >>> > preserve backwards compatibility for old METplus
>> configurations.
>> > >> The
>> > >> > >>> above
>> > >> > >>> > settings are more explicit as to what is actually be
>> configured.
>> > >> > >>> >
>> > >> > >>> > You may want to consider either clearing your output
>> directory
>> > or
>> > >> > >>> creating
>> > >> > >>> > a new output directory for a new run (especially if
the time
>> > range
>> > >> > you
>> > >> > >>> are
>> > >> > >>> > using has changed) to ensure that the files you are
generated
>> > are
>> > >> > from
>> > >> > >>> that
>> > >> > >>> > run and not from a previous run. Doing this would have
>> alerted
>> > you
>> > >> > that
>> > >> > >>> > TCPairs was not run in this case.
>> > >> > >>> >
>> > >> > >>> > Let me know if you have any other issues getting this
>> running.
>> > >> > >>> >
>> > >> > >>> > FYI:
>> > >> > >>> > We switched from providing support through this email
>> address to
>> > >> > >>> providing
>> > >> > >>> > support through GitHub Discussions. In the future, you
will
>> need
>> > >> to
>> > >> > >>> create
>> > >> > >>> > a free GitHub account if you don’t have one already
and post
>> > your
>> > >> > >>> questions
>> > >> > >>> > to the METplus Components Discussion Forum:
>> > >> > >>> > https://github.com/dtcenter/METplus/discussions
>> > >> > >>> > To post a new question, click on the green button in
the
>> upper
>> > >> right
>> > >> > >>> corner
>> > >> > >>> > of the page labeled "New discussion" and select
"Incoming" as
>> > the
>> > >> > >>> category.
>> > >> > >>> >
>> > >> > >>> > Thanks,
>> > >> > >>> > George
>> > >> > >>> >
>> > >> > >>> >
>> > >> > >>>
>> > >> > >>>
>> > >> > >>
>> > >> > >> --
>> > >> > >> George McCabe - Software Engineer III
>> > >> > >> National Center for Atmospheric Research
>> > >> > >> Research Applications Laboratory
>> > >> > >> 303-497-2768
>> > >> > >> ---
>> > >> > >> My working day may not be your working day. Please do not
feel
>> > >> obliged
>> > >> > to
>> > >> > >> reply to this email outside of your normal working hours.
>> > >> > >>
>> > >> > >
>> > >> > >
>> > >> > > --
>> > >> > > George McCabe - Software Engineer III
>> > >> > > National Center for Atmospheric Research
>> > >> > > Research Applications Laboratory
>> > >> > > 303-497-2768
>> > >> > > ---
>> > >> > > My working day may not be your working day. Please do not
feel
>> > >> obliged to
>> > >> > > reply to this email outside of your normal working hours.
>> > >> > >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > George McCabe - Software Engineer III
>> > >> > National Center for Atmospheric Research
>> > >> > Research Applications Laboratory
>> > >> > 303-497-2768
>> > >> > ---
>> > >> > My working day may not be your working day. Please do not
feel
>> obliged
>> > >> to
>> > >> > reply to this email outside of your normal working hours.
>> > >> >
>> > >> >
>> > >>
>> > >>
>> > >
>> > > --
>> > > George McCabe - Software Engineer III
>> > > National Center for Atmospheric Research
>> > > Research Applications Laboratory
>> > > 303-497-2768
>> > > ---
>> > > My working day may not be your working day. Please do not feel
>> obliged to
>> > > reply to this email outside of your normal working hours.
>> > >
>> >
>> >
>> > --
>> > George McCabe - Software Engineer III
>> > National Center for Atmospheric Research
>> > Research Applications Laboratory
>> > 303-497-2768
>> > ---
>> > My working day may not be your working day. Please do not feel
obliged
>> to
>> > reply to this email outside of your normal working hours.
>> >
>> >
>>
>>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: tcpairs
From: Guang Lou - NOAA Affiliate
Time: Thu Jul 29 07:55:32 2021
Hi George,
Thanks for the update. The purpose of this plot is to compare error
stats
between different models. So we need to plot all model errors in one
frame
instead of each model having its own frame. I did not state clearly in
my
last email.
Thanks,
Guang Ping
On Wed, Jul 28, 2021 at 4:32 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Guang Ping,
>
> I updated the config file to process each model separately. Here is
the
> config file on hera:
>
> /home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8_test.conf
>
> The output from my latest run can be found in here:
>
> /scratch1/BMC/dtc/George.Mccabe/tcmpr-test4/tcmpr_plots
>
> There is a sub-directory for each model. Please take a look at the
output
> files to see if they look OK. The config file could be cleaned up a
little
> bit if I add support for TCMPR_PLOTTER_CUSTOM_LOOP_LIST or support
for
> reading the instance name from the TCMPRPlotter instances. I will
make a
> GitHub issue to implement those changes as they are relatively easy
> additions and would make the wrapper consistent with the other
wrappers.
>
> Thanks,
> George
>
> On Wed, Jul 28, 2021 at 10:02 AM George McCabe <mccabe at ucar.edu>
wrote:
>
> > Hi Guang Ping,
> >
> > OK, that is good to know. I have a few meetings this morning but I
will
> > try to update your configuration file to set this up.
> >
> > - George
> >
> > On Wed, Jul 28, 2021 at 9:46 AM Guang Lou - NOAA Affiliate via RT
<
> > met_help at ucar.edu> wrote:
> >
> >> Hi George,
> >>
> >> The verification should be against self analysis. That is GFS
adeck
> >> against
> >> GFS bdeck, CMC adeck against CMC bdeck, and so on and so forth.
> >>
> >> Thanks,
> >> Guang Ping
> >>
> >> On Tue, Jul 27, 2021 at 2:33 PM George McCabe via RT
<met_help at ucar.edu
> >
> >> wrote:
> >>
> >> > Hi Guang Ping,
> >> >
> >> > I looked at this use case more today. I modified the use case
to read
> a
> >> > single month into TCMPRPlotter because it cannot handle all of
the
> >> files at
> >> > once. It looks like you need to set the -bmodel in
> TCMPR_PLOTTER_FILTER
> >> to
> >> > specify the Bdeck to compare the models to. However, each of
the
> output
> >> > files from tc_pairs use the same model for adeck and bdeck, so
the
> >> AMODEL
> >> > and BMODEL values in those files are the same, i.e. both GFSO.
If you
> >> want
> >> > to compare all of the models to a single bdeck, you will have
to
> >> configure
> >> > TCPairs wrapper to use the same bdeck for each model with
> >> > TC_PAIRS_BDECK_TEMPLATE. I tried running using best for the
bdeck and
> it
> >> > produced no pairs. I am rerunning now using GFSO.
> >> >
> >> > - George
> >> >
> >> > On Mon, Jul 26, 2021 at 5:57 PM George McCabe <mccabe at ucar.edu>
> wrote:
> >> >
> >> > > Hi Guang Ping,
> >> > >
> >> > > I copied your config file and made some changes. The new file
can be
> >> > found
> >> > > on hera here:
> >> > >
> >> > > /home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8.conf
> >> > >
> >> > > Using METplus 4.1.0-beta1, you can set TC_PAIRS_RUN_ONCE =
True to
> >> loop
> >> > > over the init times even with LOOP_ORDER = processes.
> >> > >
> >> > > I noticed that you have TC_PAIRS_REFORMAT_DECK = yes, which
isn't
> >> > > necessary because tc_pairs can read the input files without
> >> reformatting.
> >> > > That can be unset or set to False/no.
> >> > >
> >> > > TC_PAIRS_REFORMAT_DECK = no
> >> > >
> >> > > I misread your email about trying to use the wildcard for the
model.
> >> This
> >> > > won't work as you expect because the output template will not
know
> >> how to
> >> > > name the output file with the correct model. Each model will
> overwrite
> >> > the
> >> > > output files. To get around this you can use
> >> TC_PAIRS_CUSTOM_LOOP_LIST.
> >> > > This will run the TCPairs wrapper once for each item in the
custom
> >> loop
> >> > > list. Here you can specify the list of models (lowercase) and
> >> reference
> >> > the
> >> > > {custom} template tag to find the files.
> >> > >
> >> > > TC_PAIRS_CUSTOM_LOOP_LIST = best, cmc, emx, gfso, nam, ngx,
ukx
> >> > >
> >> > > TC_PAIRS_ADECK_TEMPLATE =
> >> > >
> >> >
> >>
>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
> >> > > TC_PAIRS_BDECK_TEMPLATE =
> >> > >
> >> >
> >>
>
{date?fmt=%Y%m}/b{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
> >> > > TC_PAIRS_OUTPUT_TEMPLATE =
> >> > >
> >> >
> >>
>
{date?fmt=%Y%m}/{basin?fmt=%s}q{date?fmt=%Y%m%d%H}.{custom}.{cyclone?fmt=%s}
> >> > >
> >> > > I removed the MODEL values (you had it twice in your config
file)
> >> because
> >> > > we don't need to filter by the model if we are reading all of
them
> in
> >> > > individually.
> >> > >
> >> > > Using this config file I was able to run TCPairs for each
model
> >> > processing
> >> > > each for all 3 months and creating output files from tc_pairs
here:
> >> > >
> >> > > /scratch1/BMC/dtc/George.Mccabe/tcmpr-test2/tc_pairs
> >> > >
> >> > > The error in your log file from TCMPRPlotter mentions that
you do
> not
> >> > have
> >> > > -bmodel set in the filter options. When I ran it, I got an
error
> with
> >> a
> >> > > command that appears to list the files which leads me to
think that
> >> there
> >> > > are too many files in the tc_pairs directory to process at
once. I
> >> need
> >> > to
> >> > > stop working for today, but I will resume debugging this
tomorrow
> and
> >> see
> >> > > what I can find. It may be the case that the plotter cannot
handle
> >> all 7
> >> > > models over a 3 day period all at once. If this is the case,
what
> >> would
> >> > be
> >> > > useful to display as a subset of this data?
> >> > >
> >> > > Thanks,
> >> > > George
> >> > >
> >> > >
> >> > >
> >> > > On Fri, Jul 23, 2021 at 11:46 AM Guang Lou - NOAA Affiliate
via RT <
> >> > > met_help at ucar.edu> wrote:
> >> > >
> >> > >>
> >> > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100602 >
> >> > >>
> >> > >> Hi George,
> >> > >>
> >> > >> Thanks for letting me know you have downloaded the files.
I've
> >> removed
> >> > the
> >> > >> files to save disk space.
> >> > >> Per your suggestion, I was able to generate consecutive
months of
> >> paired
> >> > >> tracks by changing a few lines
> >> > >> and parameters in the configure file. I've also loaded the
module
> >> > >> "metplus/4.1.0-beta1".
> >> > >>
> >> > >> This was done by:
> >> > >> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs
> >> > >> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c
> >> > >> usecase_pairing_8.conf
> >> > >>
> >> > >> However, when I tried to generate plots from the just
generated
> >> pairs,
> >> > it
> >> > >> ran into error:
> >> > >> in /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs
> >> > >> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c
usecase_8.conf
> >> > >>
> >> > >> where I changed the following to:
> >> > >> LOOP_ORDER = processes
> >> > >> PROCESS_LIST = TCMPRPlotter
> >> > >> error messages are:
> >> > >> 07/23 17:38:51.477 metplus INFO: Check the logfile for more
> >> information
> >> > on
> >> > >> why it failed:
> >> > >>
> >> > >>
> >> >
> >>
> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
> >> > >> 07/23 17:38:51.478 metplus INFO: Plotting complete
> >> > >> 07/23 17:38:51.481 metplus.TCMPRPlotter (met_util.py:171)
ERROR:
> >> > >> TCMPRPlotter had 9 errors.
> >> > >> 07/23 17:38:51.481 metplus INFO:
> >> > >>
> >> > >>
> >> >
> >>
> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/metplus_final.conf:
> >> > >> write metplus.conf here
> >> > >> 07/23 17:38:51.486 metplus (met_util.py:208) ERROR: METplus
has
> >> finished
> >> > >> running but had 9 errors.
> >> > >> 07/23 17:38:51.486 metplus INFO: Check the log file for more
> >> > information:
> >> > >>
> >> > >>
> >> >
> >>
> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
> >> > >> -----------------------------------------------------------
> >> > >> The wild card in the place of
> >> > >> TC_PAIRS_ADECK_TEMPLATE =
> >> > >>
>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.nam.{cyclone?fmt=%s}
> >> > >> to
> >> > >> TC_PAIRS_ADECK_TEMPLATE =
> >> > >>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> >> > >> still does not work.
> >> > >>
> >> > >> Thanks,
> >> > >> Guang Ping
> >> > >>
> >> > >> On Wed, Jul 21, 2021 at 2:41 PM George McCabe via RT <
> >> met_help at ucar.edu
> >> > >
> >> > >> wrote:
> >> > >>
> >> > >> > Hi Guang Ping,
> >> > >> >
> >> > >> > I have finished downloading the GFS NetCDF files from Hera
so you
> >> can
> >> > >> > remove them. Thanks again for providing that data.
> >> > >> >
> >> > >> > - George
> >> > >> >
> >> > >> > On Tue, Jul 20, 2021 at 3:35 PM George McCabe
<mccabe at ucar.edu>
> >> > wrote:
> >> > >> >
> >> > >> > > Hi Guang Ping,
> >> > >> > >
> >> > >> > > I mistyped in my previous email. The variable to set to
"times"
> >> is
> >> > >> > > LOOP_ORDER not LOOP_BY.
> >> > >> > >
> >> > >> > > LOOP_ORDER = times
> >> > >> > > INIT_INCREMENT = 1m
> >> > >> > >
> >> > >> > > Sorry for the confusion.
> >> > >> > >
> >> > >> > > - George
> >> > >> > >
> >> > >> > > On Tue, Jul 20, 2021 at 3:21 PM George McCabe
<mccabe at ucar.edu
> >
> >> > >> wrote:
> >> > >> > >
> >> > >> > >> Hi Guang Ping,
> >> > >> > >>
> >> > >> > >> Thank you for providing the NetCDF files. I am
currently
> >> > downloading
> >> > >> one
> >> > >> > >> of the files but it is going very slowly.
> >> > >> > >>
> >> > >> > >> In regards to your use case, the logic for TCPairs
wrapper is
> a
> >> > >> little
> >> > >> > >> strange. It behaves differently based on the value of
LOOP_BY.
> >> If
> >> > >> > LOOP_BY =
> >> > >> > >> processes, then it will run once using the INIT_BEG
time as
> the
> >> run
> >> > >> > time to
> >> > >> > >> find input files. If LOOP_BY = times, then it will loop
over
> >> each
> >> > run
> >> > >> > time
> >> > >> > >> specified between INIT_BEG and INIT_END by
INIT_INCREMENT and
> >> find
> >> > >> > files to
> >> > >> > >> process for each run time.
> >> > >> > >>
> >> > >> > >> If you change your configuration file to include
LOOP_BY =
> >> times,
> >> > >> then
> >> > >> > it
> >> > >> > >> should look over each run time and process for each. It
looks
> >> like
> >> > >> your
> >> > >> > >> INIT_INCREMENT is set to 21600 or 6 hours, so it will
run
> every
> >> 6
> >> > >> hours
> >> > >> > >> from 20200201 to 2020331. If you want to run once per
month,
> you
> >> > will
> >> > >> > need
> >> > >> > >> to set INIT_INCREMENT = 1m so that it will increment by
1
> month
> >> for
> >> > >> each
> >> > >> > >> iteration.
> >> > >> > >>
> >> > >> > >> The caveat is that TCMPRPlotter will only run if
LOOP_BY =
> >> > processes
> >> > >> and
> >> > >> > >> will produce an error if LOOP_BY = times. This is
because we
> >> > >> typically
> >> > >> > want
> >> > >> > >> to process all of the data that is available for all
run
> times,
> >> > then
> >> > >> run
> >> > >> > >> TCMPRPlotter so it can read all of the data. For now,
to
> handle
> >> > this
> >> > >> use
> >> > >> > >> case you will have to call run_metplus.py 2 separate
times,
> once
> >> > >> calling
> >> > >> > >> TCPairs with LOOP_BY = times and once calling
TCMPRPlotter
> with
> >> > >> LOOP_BY
> >> > >> > =
> >> > >> > >> processes.
> >> > >> > >>
> >> > >> > >> I am going to add a configuration option to TCPairs
wrapper so
> >> that
> >> > >> you
> >> > >> > >> can loop over the run times when LOOP_BY = processes is
set if
> >> > >> desired.
> >> > >> > I
> >> > >> > >> will try to get this change into METplus 4.1.0-beta1 so
you
> can
> >> > test
> >> > >> the
> >> > >> > >> change and make sure it works as expected. I can let
you know
> >> when
> >> > >> that
> >> > >> > is
> >> > >> > >> available.
> >> > >> > >>
> >> > >> > >> Thanks,
> >> > >> > >> George
> >> > >> > >>
> >> > >> > >> On Tue, Jul 20, 2021 at 1:27 PM Guang Lou - NOAA
Affiliate via
> >> RT <
> >> > >> > >> met_help at ucar.edu> wrote:
> >> > >> > >>
> >> > >> > >>> Hi John,
> >> > >> > >>>
> >> > >> > >>> Thanks for forwarding George's email answer to me. I
created
> a
> >> > test
> >> > >> > case
> >> > >> > >>> that does the tcpairs only on Hera:
> >> > >> > >>>
> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test3
> >> > >> > >>> Command line:
> >> > >> > >>> run_metplus.py -c
Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c
> >> > >> > >>> usecase_8.conf
> >> > >> > >>> where PROCESS_LIST = TCPairs
> >> > >> > >>>
> >> > >> > >>> It generated the month of 202002, that is because I
changed
> >> > >> > >>> INIT_BEG = 20200101
> >> > >> > >>> INIT_END = 20200331
> >> > >> > >>> to
> >> > >> > >>> INIT_BEG = 20200201
> >> > >> > >>> INIT_END = 20200331
> >> > >> > >>>
> >> > >> > >>> If I use "INIT_BEG = 20200101,INIT_END = 20200331" it
will
> >> > generate
> >> > >> > >>> 20200101 only.
> >> > >> > >>>
> >> > >> > >>> The wild card does not work either for the model's
place:
> >> > >> > >>> TC_PAIRS_ADECK_TEMPLATE =
> >> > >> > >>>
> >> >
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> >> > >> > >>> I populated adeck and bdeck on
> >> > >> > >>>
> >> > >> > >>>
> >> > >> >
> >> > >>
> >> >
> >>
> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/METplus/METplus-
4.0_sample_data/model_applications/medium_range/tracks_all
> >> > >> > >>> (each subdirectory contains all three months' decks.)
> >> > >> > >>>
> >> > >> > >>> I was wondering if you could copy my directory,
> >> > >> > >>>
> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test3
> >> > >> > >>> and test:
> >> > >> > >>> 1. It can generate multiple months instead of just one
month
> >> and
> >> > one
> >> > >> > >>> model.
> >> > >> > >>> 2. The wild card for models in the DECK_TEMPATE for
multiple
> >> > models.
> >> > >> > >>>
> >> > >> > >>> George, I'm downloading NetCDF model data to Hera:
> >> > >> > >>>
> >> >
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071300_nc.tar
> >> > >> > >>>
> >> >
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071306_nc.tar
> >> > >> > >>>
> >> > >> > >>> Please let me know after you downloaded the files and
I will
> >> have
> >> > to
> >> > >> > >>> remove
> >> > >> > >>> them due to tight disk space.
> >> > >> > >>>
> >> > >> > >>> Thanks,
> >> > >> > >>> Guang Ping
> >> > >> > >>>
> >> > >> > >>> On Mon, Jul 19, 2021 at 5:01 PM John Halley Gotway via
RT <
> >> > >> > >>> met_help at ucar.edu>
> >> > >> > >>> wrote:
> >> > >> > >>>
> >> > >> > >>> > Hi Guang Ping,
> >> > >> > >>> >
> >> > >> > >>> > The command that you ran here:
> >> > >> > >>> >
> >> > >> > >>> >
> >> > >> > >>> >
> >> > >> > >>>
> >> > >> >
> >> > >>
> >> >
> >>
>
/home/George.Mccabe/scratch1_George.Mccabe/METplus.feature_926/ush/run_metplus.py
> >> > >> > >>> > -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c
> Plotter_8.conf
> >> > >> > >>> >
> >> > >> > >>> > only runs TCMPRPlotter. This is because PROCESS_LIST
is
> >> defined
> >> > in
> >> > >> > both
> >> > >> > >>> > Plotter_fcstGFS_obsGFS_RPlotting_8.conf and
Plotter_8.conf.
> >> > Since
> >> > >> > >>> > Plotter_8.conf comes last, then the value for
PROCESS_LIST
> >> from
> >> > >> that
> >> > >> > >>> file
> >> > >> > >>> > is used, which only has TCMPRPlotter listed. The
plotter is
> >> > >> reading
> >> > >> > >>> all of
> >> > >> > >>> > the available files generated from TCPairs and since
that
> >> > process
> >> > >> is
> >> > >> > >>> not
> >> > >> > >>> > running again with the new time range, the data that
is
> read
> >> > only
> >> > >> > >>> covers
> >> > >> > >>> > the first month.
> >> > >> > >>> >
> >> > >> > >>> > We are planning on releasing METplus 4.1.0-beta1
this week.
> >> > >> However,
> >> > >> > >>> if I
> >> > >> > >>> > understand what you are trying to do here, it
shouldn't
> >> require
> >> > >> the
> >> > >> > >>> changes
> >> > >> > >>> > described in Molly's issue. It looks like you want
to run
> >> > TCPairs
> >> > >> > once
> >> > >> > >>> and
> >> > >> > >>> > filter the data based on the init and valid times.
While
> this
> >> > >> > >>> configuration
> >> > >> > >>> > should achieve what you are trying to do, the
following
> >> > >> configuration
> >> > >> > >>> is
> >> > >> > >>> > more explicit to what you are actually running:
> >> > >> > >>> >
> >> > >> > >>> > # run once using 20200101 to substitute the init
time in
> your
> >> > >> string
> >> > >> > >>> > substitution, i.e. TC_PAIRS_ADECK_TEMPLATE =
> >> > >> > >>> >
> >> > >>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> >> > >> > >>> > INIT_BEG = 20200101
> >> > >> > >>> > INIT_END = 20200101
> >> > >> > >>> >
> >> > >> > >>> > # filter the init times of the lines read from the
input
> >> files
> >> > in
> >> > >> the
> >> > >> > >>> > TCPairs configuration file
> >> > >> > >>> > TC_PAIRS_INIT_BEG = 20200101
> >> > >> > >>> > TC_PAIRS_INIT_END = 20200331
> >> > >> > >>> >
> >> > >> > >>> > # filter the valid times of the lines read from the
input
> >> files
> >> > in
> >> > >> > the
> >> > >> > >>> > TCPairs configuration file
> >> > >> > >>> > TC_PAIRS_VALID_BEG = 20200101
> >> > >> > >>> > TC_PAIRS_VALID_END = 20200331
> >> > >> > >>> >
> >> > >> > >>> > If any of TC_PAIRS_[INIT/VALID]_[BEG/END] are not
set, then
> >> the
> >> > >> > >>> > corresponding [INIT/VALID]_[BEG/END] values are
used. This
> is
> >> > >> done to
> >> > >> > >>> > preserve backwards compatibility for old METplus
> >> configurations.
> >> > >> The
> >> > >> > >>> above
> >> > >> > >>> > settings are more explicit as to what is actually be
> >> configured.
> >> > >> > >>> >
> >> > >> > >>> > You may want to consider either clearing your output
> >> directory
> >> > or
> >> > >> > >>> creating
> >> > >> > >>> > a new output directory for a new run (especially if
the
> time
> >> > range
> >> > >> > you
> >> > >> > >>> are
> >> > >> > >>> > using has changed) to ensure that the files you are
> generated
> >> > are
> >> > >> > from
> >> > >> > >>> that
> >> > >> > >>> > run and not from a previous run. Doing this would
have
> >> alerted
> >> > you
> >> > >> > that
> >> > >> > >>> > TCPairs was not run in this case.
> >> > >> > >>> >
> >> > >> > >>> > Let me know if you have any other issues getting
this
> >> running.
> >> > >> > >>> >
> >> > >> > >>> > FYI:
> >> > >> > >>> > We switched from providing support through this
email
> >> address to
> >> > >> > >>> providing
> >> > >> > >>> > support through GitHub Discussions. In the future,
you will
> >> need
> >> > >> to
> >> > >> > >>> create
> >> > >> > >>> > a free GitHub account if you don’t have one already
and
> post
> >> > your
> >> > >> > >>> questions
> >> > >> > >>> > to the METplus Components Discussion Forum:
> >> > >> > >>> > https://github.com/dtcenter/METplus/discussions
> >> > >> > >>> > To post a new question, click on the green button in
the
> >> upper
> >> > >> right
> >> > >> > >>> corner
> >> > >> > >>> > of the page labeled "New discussion" and select
"Incoming"
> as
> >> > the
> >> > >> > >>> category.
> >> > >> > >>> >
> >> > >> > >>> > Thanks,
> >> > >> > >>> > George
> >> > >> > >>> >
> >> > >> > >>> >
> >> > >> > >>>
> >> > >> > >>>
> >> > >> > >>
> >> > >> > >> --
> >> > >> > >> George McCabe - Software Engineer III
> >> > >> > >> National Center for Atmospheric Research
> >> > >> > >> Research Applications Laboratory
> >> > >> > >> 303-497-2768
> >> > >> > >> ---
> >> > >> > >> My working day may not be your working day. Please do
not feel
> >> > >> obliged
> >> > >> > to
> >> > >> > >> reply to this email outside of your normal working
hours.
> >> > >> > >>
> >> > >> > >
> >> > >> > >
> >> > >> > > --
> >> > >> > > George McCabe - Software Engineer III
> >> > >> > > National Center for Atmospheric Research
> >> > >> > > Research Applications Laboratory
> >> > >> > > 303-497-2768
> >> > >> > > ---
> >> > >> > > My working day may not be your working day. Please do
not feel
> >> > >> obliged to
> >> > >> > > reply to this email outside of your normal working
hours.
> >> > >> > >
> >> > >> >
> >> > >> >
> >> > >> > --
> >> > >> > George McCabe - Software Engineer III
> >> > >> > National Center for Atmospheric Research
> >> > >> > Research Applications Laboratory
> >> > >> > 303-497-2768
> >> > >> > ---
> >> > >> > My working day may not be your working day. Please do not
feel
> >> obliged
> >> > >> to
> >> > >> > reply to this email outside of your normal working hours.
> >> > >> >
> >> > >> >
> >> > >>
> >> > >>
> >> > >
> >> > > --
> >> > > George McCabe - Software Engineer III
> >> > > National Center for Atmospheric Research
> >> > > Research Applications Laboratory
> >> > > 303-497-2768
> >> > > ---
> >> > > My working day may not be your working day. Please do not
feel
> >> obliged to
> >> > > reply to this email outside of your normal working hours.
> >> > >
> >> >
> >> >
> >> > --
> >> > George McCabe - Software Engineer III
> >> > National Center for Atmospheric Research
> >> > Research Applications Laboratory
> >> > 303-497-2768
> >> > ---
> >> > My working day may not be your working day. Please do not feel
obliged
> >> to
> >> > reply to this email outside of your normal working hours.
> >> >
> >> >
> >>
> >>
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: tcpairs
From: George McCabe
Time: Tue Aug 03 08:00:07 2021
Hi Guang Ping,
I see. Thanks for clarifying. The good news is that John HG was able
to
generate a plot of all of the models. Attached is an image he
generated.
This is the command he ran to generate the images:
Rscript met/scripts/Rscripts/plot_tcmpr.R -lookin `find ./ -name
"*.tcst"`
-no_ee -plot BOXPLOT,MEAN
There are 2 things John mentioned to note here:
* That "find" command found 2705 input files and passed them all in on
the
command line. That's a long command line and might cause shell
problems on
some system.
* The "-no_ee" option DISABLES event equalization, which is needed
since
these cases are not the same. With event equalization turned on, all
the
cases will be tossed out.
The -no_ee argument can be set by setting this in your METplus
configuration file:
TCMPR_PLOTTER_NO_EE = yes
Howeer, his first comment aligns with the error I was seeing when I
tried
to run where the shell couldn't handle the large number of files that
were
passed to the command. The wrapper finds all of the files in the input
directory that end with .tcst and pass in all of the files to the
Rscript
command, which is essentially the same behavior as the inserting the
find
command in John's example. Hera is down today so I am unable to test,
but
it may be the case that Hera cannot handle this command. The
plot_tcmpr.R
script fails on this line (380), which attempts to expand any
wildcards
with an ls command:
file_list = system(paste("ls -1", paste(file_list, collapse=" ")),
intern=TRUE);
The list of files is passed from the R script into the tc_stat MET
tool for
filtering and any wildcards are already expanded by the wrapper. I
suspect
that if I add an option to this script to bypass the wildcard
expansion,
then the call to tc_stat will work. I can make the necessary changes
locally and test it on Hera when it comes back online. If tc_stat
cannot
handle the long command on Hera, then modifications to the tc_stat
tool to
support reading in a text file that contains a list of files and
modifications to the TCMPRPlotter wrapper to generate a file list text
file
may fix the issue.
I will follow up with more information after some testing. It is
reassuring
to know that the images you are trying to create can be generated.
Thanks,
George
On Thu, Jul 29, 2021 at 7:55 AM Guang Lou - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
> Hi George,
>
> Thanks for the update. The purpose of this plot is to compare error
stats
> between different models. So we need to plot all model errors in one
frame
> instead of each model having its own frame. I did not state clearly
in my
> last email.
>
> Thanks,
> Guang Ping
>
> On Wed, Jul 28, 2021 at 4:32 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Guang Ping,
> >
> > I updated the config file to process each model separately. Here
is the
> > config file on hera:
> >
> > /home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8_test.conf
> >
> > The output from my latest run can be found in here:
> >
> > /scratch1/BMC/dtc/George.Mccabe/tcmpr-test4/tcmpr_plots
> >
> > There is a sub-directory for each model. Please take a look at the
output
> > files to see if they look OK. The config file could be cleaned up
a
> little
> > bit if I add support for TCMPR_PLOTTER_CUSTOM_LOOP_LIST or support
for
> > reading the instance name from the TCMPRPlotter instances. I will
make a
> > GitHub issue to implement those changes as they are relatively
easy
> > additions and would make the wrapper consistent with the other
wrappers.
> >
> > Thanks,
> > George
> >
> > On Wed, Jul 28, 2021 at 10:02 AM George McCabe <mccabe at ucar.edu>
wrote:
> >
> > > Hi Guang Ping,
> > >
> > > OK, that is good to know. I have a few meetings this morning but
I will
> > > try to update your configuration file to set this up.
> > >
> > > - George
> > >
> > > On Wed, Jul 28, 2021 at 9:46 AM Guang Lou - NOAA Affiliate via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >> Hi George,
> > >>
> > >> The verification should be against self analysis. That is GFS
adeck
> > >> against
> > >> GFS bdeck, CMC adeck against CMC bdeck, and so on and so forth.
> > >>
> > >> Thanks,
> > >> Guang Ping
> > >>
> > >> On Tue, Jul 27, 2021 at 2:33 PM George McCabe via RT <
> met_help at ucar.edu
> > >
> > >> wrote:
> > >>
> > >> > Hi Guang Ping,
> > >> >
> > >> > I looked at this use case more today. I modified the use case
to
> read
> > a
> > >> > single month into TCMPRPlotter because it cannot handle all
of the
> > >> files at
> > >> > once. It looks like you need to set the -bmodel in
> > TCMPR_PLOTTER_FILTER
> > >> to
> > >> > specify the Bdeck to compare the models to. However, each of
the
> > output
> > >> > files from tc_pairs use the same model for adeck and bdeck,
so the
> > >> AMODEL
> > >> > and BMODEL values in those files are the same, i.e. both
GFSO. If
> you
> > >> want
> > >> > to compare all of the models to a single bdeck, you will have
to
> > >> configure
> > >> > TCPairs wrapper to use the same bdeck for each model with
> > >> > TC_PAIRS_BDECK_TEMPLATE. I tried running using best for the
bdeck
> and
> > it
> > >> > produced no pairs. I am rerunning now using GFSO.
> > >> >
> > >> > - George
> > >> >
> > >> > On Mon, Jul 26, 2021 at 5:57 PM George McCabe
<mccabe at ucar.edu>
> > wrote:
> > >> >
> > >> > > Hi Guang Ping,
> > >> > >
> > >> > > I copied your config file and made some changes. The new
file can
> be
> > >> > found
> > >> > > on hera here:
> > >> > >
> > >> > > /home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8.conf
> > >> > >
> > >> > > Using METplus 4.1.0-beta1, you can set TC_PAIRS_RUN_ONCE =
True to
> > >> loop
> > >> > > over the init times even with LOOP_ORDER = processes.
> > >> > >
> > >> > > I noticed that you have TC_PAIRS_REFORMAT_DECK = yes, which
isn't
> > >> > > necessary because tc_pairs can read the input files without
> > >> reformatting.
> > >> > > That can be unset or set to False/no.
> > >> > >
> > >> > > TC_PAIRS_REFORMAT_DECK = no
> > >> > >
> > >> > > I misread your email about trying to use the wildcard for
the
> model.
> > >> This
> > >> > > won't work as you expect because the output template will
not know
> > >> how to
> > >> > > name the output file with the correct model. Each model
will
> > overwrite
> > >> > the
> > >> > > output files. To get around this you can use
> > >> TC_PAIRS_CUSTOM_LOOP_LIST.
> > >> > > This will run the TCPairs wrapper once for each item in the
custom
> > >> loop
> > >> > > list. Here you can specify the list of models (lowercase)
and
> > >> reference
> > >> > the
> > >> > > {custom} template tag to find the files.
> > >> > >
> > >> > > TC_PAIRS_CUSTOM_LOOP_LIST = best, cmc, emx, gfso, nam, ngx,
ukx
> > >> > >
> > >> > > TC_PAIRS_ADECK_TEMPLATE =
> > >> > >
> > >> >
> > >>
> >
>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
> > >> > > TC_PAIRS_BDECK_TEMPLATE =
> > >> > >
> > >> >
> > >>
> >
>
{date?fmt=%Y%m}/b{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
> > >> > > TC_PAIRS_OUTPUT_TEMPLATE =
> > >> > >
> > >> >
> > >>
> >
>
{date?fmt=%Y%m}/{basin?fmt=%s}q{date?fmt=%Y%m%d%H}.{custom}.{cyclone?fmt=%s}
> > >> > >
> > >> > > I removed the MODEL values (you had it twice in your config
file)
> > >> because
> > >> > > we don't need to filter by the model if we are reading all
of them
> > in
> > >> > > individually.
> > >> > >
> > >> > > Using this config file I was able to run TCPairs for each
model
> > >> > processing
> > >> > > each for all 3 months and creating output files from
tc_pairs
> here:
> > >> > >
> > >> > > /scratch1/BMC/dtc/George.Mccabe/tcmpr-test2/tc_pairs
> > >> > >
> > >> > > The error in your log file from TCMPRPlotter mentions that
you do
> > not
> > >> > have
> > >> > > -bmodel set in the filter options. When I ran it, I got an
error
> > with
> > >> a
> > >> > > command that appears to list the files which leads me to
think
> that
> > >> there
> > >> > > are too many files in the tc_pairs directory to process at
once. I
> > >> need
> > >> > to
> > >> > > stop working for today, but I will resume debugging this
tomorrow
> > and
> > >> see
> > >> > > what I can find. It may be the case that the plotter cannot
handle
> > >> all 7
> > >> > > models over a 3 day period all at once. If this is the
case, what
> > >> would
> > >> > be
> > >> > > useful to display as a subset of this data?
> > >> > >
> > >> > > Thanks,
> > >> > > George
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Fri, Jul 23, 2021 at 11:46 AM Guang Lou - NOAA Affiliate
via
> RT <
> > >> > > met_help at ucar.edu> wrote:
> > >> > >
> > >> > >>
> > >> > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100602 >
> > >> > >>
> > >> > >> Hi George,
> > >> > >>
> > >> > >> Thanks for letting me know you have downloaded the files.
I've
> > >> removed
> > >> > the
> > >> > >> files to save disk space.
> > >> > >> Per your suggestion, I was able to generate consecutive
months of
> > >> paired
> > >> > >> tracks by changing a few lines
> > >> > >> and parameters in the configure file. I've also loaded the
module
> > >> > >> "metplus/4.1.0-beta1".
> > >> > >>
> > >> > >> This was done by:
> > >> > >> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs
> > >> > >> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c
> > >> > >> usecase_pairing_8.conf
> > >> > >>
> > >> > >> However, when I tried to generate plots from the just
generated
> > >> pairs,
> > >> > it
> > >> > >> ran into error:
> > >> > >> in /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs
> > >> > >> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c
> usecase_8.conf
> > >> > >>
> > >> > >> where I changed the following to:
> > >> > >> LOOP_ORDER = processes
> > >> > >> PROCESS_LIST = TCMPRPlotter
> > >> > >> error messages are:
> > >> > >> 07/23 17:38:51.477 metplus INFO: Check the logfile for
more
> > >> information
> > >> > on
> > >> > >> why it failed:
> > >> > >>
> > >> > >>
> > >> >
> > >>
> >
> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
> > >> > >> 07/23 17:38:51.478 metplus INFO: Plotting complete
> > >> > >> 07/23 17:38:51.481 metplus.TCMPRPlotter (met_util.py:171)
ERROR:
> > >> > >> TCMPRPlotter had 9 errors.
> > >> > >> 07/23 17:38:51.481 metplus INFO:
> > >> > >>
> > >> > >>
> > >> >
> > >>
> >
> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/metplus_final.conf:
> > >> > >> write metplus.conf here
> > >> > >> 07/23 17:38:51.486 metplus (met_util.py:208) ERROR:
METplus has
> > >> finished
> > >> > >> running but had 9 errors.
> > >> > >> 07/23 17:38:51.486 metplus INFO: Check the log file for
more
> > >> > information:
> > >> > >>
> > >> > >>
> > >> >
> > >>
> >
> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
> > >> > >>
-----------------------------------------------------------
> > >> > >> The wild card in the place of
> > >> > >> TC_PAIRS_ADECK_TEMPLATE =
> > >> > >>
> >
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.nam.{cyclone?fmt=%s}
> > >> > >> to
> > >> > >> TC_PAIRS_ADECK_TEMPLATE =
> > >> > >>
> {date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> > >> > >> still does not work.
> > >> > >>
> > >> > >> Thanks,
> > >> > >> Guang Ping
> > >> > >>
> > >> > >> On Wed, Jul 21, 2021 at 2:41 PM George McCabe via RT <
> > >> met_help at ucar.edu
> > >> > >
> > >> > >> wrote:
> > >> > >>
> > >> > >> > Hi Guang Ping,
> > >> > >> >
> > >> > >> > I have finished downloading the GFS NetCDF files from
Hera so
> you
> > >> can
> > >> > >> > remove them. Thanks again for providing that data.
> > >> > >> >
> > >> > >> > - George
> > >> > >> >
> > >> > >> > On Tue, Jul 20, 2021 at 3:35 PM George McCabe
<mccabe at ucar.edu
> >
> > >> > wrote:
> > >> > >> >
> > >> > >> > > Hi Guang Ping,
> > >> > >> > >
> > >> > >> > > I mistyped in my previous email. The variable to set
to
> "times"
> > >> is
> > >> > >> > > LOOP_ORDER not LOOP_BY.
> > >> > >> > >
> > >> > >> > > LOOP_ORDER = times
> > >> > >> > > INIT_INCREMENT = 1m
> > >> > >> > >
> > >> > >> > > Sorry for the confusion.
> > >> > >> > >
> > >> > >> > > - George
> > >> > >> > >
> > >> > >> > > On Tue, Jul 20, 2021 at 3:21 PM George McCabe <
> mccabe at ucar.edu
> > >
> > >> > >> wrote:
> > >> > >> > >
> > >> > >> > >> Hi Guang Ping,
> > >> > >> > >>
> > >> > >> > >> Thank you for providing the NetCDF files. I am
currently
> > >> > downloading
> > >> > >> one
> > >> > >> > >> of the files but it is going very slowly.
> > >> > >> > >>
> > >> > >> > >> In regards to your use case, the logic for TCPairs
wrapper
> is
> > a
> > >> > >> little
> > >> > >> > >> strange. It behaves differently based on the value of
> LOOP_BY.
> > >> If
> > >> > >> > LOOP_BY =
> > >> > >> > >> processes, then it will run once using the INIT_BEG
time as
> > the
> > >> run
> > >> > >> > time to
> > >> > >> > >> find input files. If LOOP_BY = times, then it will
loop over
> > >> each
> > >> > run
> > >> > >> > time
> > >> > >> > >> specified between INIT_BEG and INIT_END by
INIT_INCREMENT
> and
> > >> find
> > >> > >> > files to
> > >> > >> > >> process for each run time.
> > >> > >> > >>
> > >> > >> > >> If you change your configuration file to include
LOOP_BY =
> > >> times,
> > >> > >> then
> > >> > >> > it
> > >> > >> > >> should look over each run time and process for each.
It
> looks
> > >> like
> > >> > >> your
> > >> > >> > >> INIT_INCREMENT is set to 21600 or 6 hours, so it will
run
> > every
> > >> 6
> > >> > >> hours
> > >> > >> > >> from 20200201 to 2020331. If you want to run once per
month,
> > you
> > >> > will
> > >> > >> > need
> > >> > >> > >> to set INIT_INCREMENT = 1m so that it will increment
by 1
> > month
> > >> for
> > >> > >> each
> > >> > >> > >> iteration.
> > >> > >> > >>
> > >> > >> > >> The caveat is that TCMPRPlotter will only run if
LOOP_BY =
> > >> > processes
> > >> > >> and
> > >> > >> > >> will produce an error if LOOP_BY = times. This is
because we
> > >> > >> typically
> > >> > >> > want
> > >> > >> > >> to process all of the data that is available for all
run
> > times,
> > >> > then
> > >> > >> run
> > >> > >> > >> TCMPRPlotter so it can read all of the data. For now,
to
> > handle
> > >> > this
> > >> > >> use
> > >> > >> > >> case you will have to call run_metplus.py 2 separate
times,
> > once
> > >> > >> calling
> > >> > >> > >> TCPairs with LOOP_BY = times and once calling
TCMPRPlotter
> > with
> > >> > >> LOOP_BY
> > >> > >> > =
> > >> > >> > >> processes.
> > >> > >> > >>
> > >> > >> > >> I am going to add a configuration option to TCPairs
wrapper
> so
> > >> that
> > >> > >> you
> > >> > >> > >> can loop over the run times when LOOP_BY = processes
is set
> if
> > >> > >> desired.
> > >> > >> > I
> > >> > >> > >> will try to get this change into METplus 4.1.0-beta1
so you
> > can
> > >> > test
> > >> > >> the
> > >> > >> > >> change and make sure it works as expected. I can let
you
> know
> > >> when
> > >> > >> that
> > >> > >> > is
> > >> > >> > >> available.
> > >> > >> > >>
> > >> > >> > >> Thanks,
> > >> > >> > >> George
> > >> > >> > >>
> > >> > >> > >> On Tue, Jul 20, 2021 at 1:27 PM Guang Lou - NOAA
Affiliate
> via
> > >> RT <
> > >> > >> > >> met_help at ucar.edu> wrote:
> > >> > >> > >>
> > >> > >> > >>> Hi John,
> > >> > >> > >>>
> > >> > >> > >>> Thanks for forwarding George's email answer to me. I
> created
> > a
> > >> > test
> > >> > >> > case
> > >> > >> > >>> that does the tcpairs only on Hera:
> > >> > >> > >>>
> > /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test3
> > >> > >> > >>> Command line:
> > >> > >> > >>> run_metplus.py -c
Plotter_fcstGFS_obsGFS_RPlotting_8.conf
> -c
> > >> > >> > >>> usecase_8.conf
> > >> > >> > >>> where PROCESS_LIST = TCPairs
> > >> > >> > >>>
> > >> > >> > >>> It generated the month of 202002, that is because I
changed
> > >> > >> > >>> INIT_BEG = 20200101
> > >> > >> > >>> INIT_END = 20200331
> > >> > >> > >>> to
> > >> > >> > >>> INIT_BEG = 20200201
> > >> > >> > >>> INIT_END = 20200331
> > >> > >> > >>>
> > >> > >> > >>> If I use "INIT_BEG = 20200101,INIT_END = 20200331"
it will
> > >> > generate
> > >> > >> > >>> 20200101 only.
> > >> > >> > >>>
> > >> > >> > >>> The wild card does not work either for the model's
place:
> > >> > >> > >>> TC_PAIRS_ADECK_TEMPLATE =
> > >> > >> > >>>
> > >> >
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> > >> > >> > >>> I populated adeck and bdeck on
> > >> > >> > >>>
> > >> > >> > >>>
> > >> > >> >
> > >> > >>
> > >> >
> > >>
> >
> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/METplus/METplus-
4.0_sample_data/model_applications/medium_range/tracks_all
> > >> > >> > >>> (each subdirectory contains all three months'
decks.)
> > >> > >> > >>>
> > >> > >> > >>> I was wondering if you could copy my directory,
> > >> > >> > >>>
> > /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test3
> > >> > >> > >>> and test:
> > >> > >> > >>> 1. It can generate multiple months instead of just
one
> month
> > >> and
> > >> > one
> > >> > >> > >>> model.
> > >> > >> > >>> 2. The wild card for models in the DECK_TEMPATE for
> multiple
> > >> > models.
> > >> > >> > >>>
> > >> > >> > >>> George, I'm downloading NetCDF model data to Hera:
> > >> > >> > >>>
> > >> >
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071300_nc.tar
> > >> > >> > >>>
> > >> >
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071306_nc.tar
> > >> > >> > >>>
> > >> > >> > >>> Please let me know after you downloaded the files
and I
> will
> > >> have
> > >> > to
> > >> > >> > >>> remove
> > >> > >> > >>> them due to tight disk space.
> > >> > >> > >>>
> > >> > >> > >>> Thanks,
> > >> > >> > >>> Guang Ping
> > >> > >> > >>>
> > >> > >> > >>> On Mon, Jul 19, 2021 at 5:01 PM John Halley Gotway
via RT <
> > >> > >> > >>> met_help at ucar.edu>
> > >> > >> > >>> wrote:
> > >> > >> > >>>
> > >> > >> > >>> > Hi Guang Ping,
> > >> > >> > >>> >
> > >> > >> > >>> > The command that you ran here:
> > >> > >> > >>> >
> > >> > >> > >>> >
> > >> > >> > >>> >
> > >> > >> > >>>
> > >> > >> >
> > >> > >>
> > >> >
> > >>
> >
>
/home/George.Mccabe/scratch1_George.Mccabe/METplus.feature_926/ush/run_metplus.py
> > >> > >> > >>> > -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c
> > Plotter_8.conf
> > >> > >> > >>> >
> > >> > >> > >>> > only runs TCMPRPlotter. This is because
PROCESS_LIST is
> > >> defined
> > >> > in
> > >> > >> > both
> > >> > >> > >>> > Plotter_fcstGFS_obsGFS_RPlotting_8.conf and
> Plotter_8.conf.
> > >> > Since
> > >> > >> > >>> > Plotter_8.conf comes last, then the value for
> PROCESS_LIST
> > >> from
> > >> > >> that
> > >> > >> > >>> file
> > >> > >> > >>> > is used, which only has TCMPRPlotter listed. The
plotter
> is
> > >> > >> reading
> > >> > >> > >>> all of
> > >> > >> > >>> > the available files generated from TCPairs and
since that
> > >> > process
> > >> > >> is
> > >> > >> > >>> not
> > >> > >> > >>> > running again with the new time range, the data
that is
> > read
> > >> > only
> > >> > >> > >>> covers
> > >> > >> > >>> > the first month.
> > >> > >> > >>> >
> > >> > >> > >>> > We are planning on releasing METplus 4.1.0-beta1
this
> week.
> > >> > >> However,
> > >> > >> > >>> if I
> > >> > >> > >>> > understand what you are trying to do here, it
shouldn't
> > >> require
> > >> > >> the
> > >> > >> > >>> changes
> > >> > >> > >>> > described in Molly's issue. It looks like you want
to run
> > >> > TCPairs
> > >> > >> > once
> > >> > >> > >>> and
> > >> > >> > >>> > filter the data based on the init and valid times.
While
> > this
> > >> > >> > >>> configuration
> > >> > >> > >>> > should achieve what you are trying to do, the
following
> > >> > >> configuration
> > >> > >> > >>> is
> > >> > >> > >>> > more explicit to what you are actually running:
> > >> > >> > >>> >
> > >> > >> > >>> > # run once using 20200101 to substitute the init
time in
> > your
> > >> > >> string
> > >> > >> > >>> > substitution, i.e. TC_PAIRS_ADECK_TEMPLATE =
> > >> > >> > >>> >
> > >> > >>
> {date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> > >> > >> > >>> > INIT_BEG = 20200101
> > >> > >> > >>> > INIT_END = 20200101
> > >> > >> > >>> >
> > >> > >> > >>> > # filter the init times of the lines read from the
input
> > >> files
> > >> > in
> > >> > >> the
> > >> > >> > >>> > TCPairs configuration file
> > >> > >> > >>> > TC_PAIRS_INIT_BEG = 20200101
> > >> > >> > >>> > TC_PAIRS_INIT_END = 20200331
> > >> > >> > >>> >
> > >> > >> > >>> > # filter the valid times of the lines read from
the input
> > >> files
> > >> > in
> > >> > >> > the
> > >> > >> > >>> > TCPairs configuration file
> > >> > >> > >>> > TC_PAIRS_VALID_BEG = 20200101
> > >> > >> > >>> > TC_PAIRS_VALID_END = 20200331
> > >> > >> > >>> >
> > >> > >> > >>> > If any of TC_PAIRS_[INIT/VALID]_[BEG/END] are not
set,
> then
> > >> the
> > >> > >> > >>> > corresponding [INIT/VALID]_[BEG/END] values are
used.
> This
> > is
> > >> > >> done to
> > >> > >> > >>> > preserve backwards compatibility for old METplus
> > >> configurations.
> > >> > >> The
> > >> > >> > >>> above
> > >> > >> > >>> > settings are more explicit as to what is actually
be
> > >> configured.
> > >> > >> > >>> >
> > >> > >> > >>> > You may want to consider either clearing your
output
> > >> directory
> > >> > or
> > >> > >> > >>> creating
> > >> > >> > >>> > a new output directory for a new run (especially
if the
> > time
> > >> > range
> > >> > >> > you
> > >> > >> > >>> are
> > >> > >> > >>> > using has changed) to ensure that the files you
are
> > generated
> > >> > are
> > >> > >> > from
> > >> > >> > >>> that
> > >> > >> > >>> > run and not from a previous run. Doing this would
have
> > >> alerted
> > >> > you
> > >> > >> > that
> > >> > >> > >>> > TCPairs was not run in this case.
> > >> > >> > >>> >
> > >> > >> > >>> > Let me know if you have any other issues getting
this
> > >> running.
> > >> > >> > >>> >
> > >> > >> > >>> > FYI:
> > >> > >> > >>> > We switched from providing support through this
email
> > >> address to
> > >> > >> > >>> providing
> > >> > >> > >>> > support through GitHub Discussions. In the future,
you
> will
> > >> need
> > >> > >> to
> > >> > >> > >>> create
> > >> > >> > >>> > a free GitHub account if you don’t have one
already and
> > post
> > >> > your
> > >> > >> > >>> questions
> > >> > >> > >>> > to the METplus Components Discussion Forum:
> > >> > >> > >>> > https://github.com/dtcenter/METplus/discussions
> > >> > >> > >>> > To post a new question, click on the green button
in the
> > >> upper
> > >> > >> right
> > >> > >> > >>> corner
> > >> > >> > >>> > of the page labeled "New discussion" and select
> "Incoming"
> > as
> > >> > the
> > >> > >> > >>> category.
> > >> > >> > >>> >
> > >> > >> > >>> > Thanks,
> > >> > >> > >>> > George
> > >> > >> > >>> >
> > >> > >> > >>> >
> > >> > >> > >>>
> > >> > >> > >>>
> > >> > >> > >>
> > >> > >> > >> --
> > >> > >> > >> George McCabe - Software Engineer III
> > >> > >> > >> National Center for Atmospheric Research
> > >> > >> > >> Research Applications Laboratory
> > >> > >> > >> 303-497-2768
> > >> > >> > >> ---
> > >> > >> > >> My working day may not be your working day. Please do
not
> feel
> > >> > >> obliged
> > >> > >> > to
> > >> > >> > >> reply to this email outside of your normal working
hours.
> > >> > >> > >>
> > >> > >> > >
> > >> > >> > >
> > >> > >> > > --
> > >> > >> > > George McCabe - Software Engineer III
> > >> > >> > > National Center for Atmospheric Research
> > >> > >> > > Research Applications Laboratory
> > >> > >> > > 303-497-2768
> > >> > >> > > ---
> > >> > >> > > My working day may not be your working day. Please do
not
> feel
> > >> > >> obliged to
> > >> > >> > > reply to this email outside of your normal working
hours.
> > >> > >> > >
> > >> > >> >
> > >> > >> >
> > >> > >> > --
> > >> > >> > George McCabe - Software Engineer III
> > >> > >> > National Center for Atmospheric Research
> > >> > >> > Research Applications Laboratory
> > >> > >> > 303-497-2768
> > >> > >> > ---
> > >> > >> > My working day may not be your working day. Please do
not feel
> > >> obliged
> > >> > >> to
> > >> > >> > reply to this email outside of your normal working
hours.
> > >> > >> >
> > >> > >> >
> > >> > >>
> > >> > >>
> > >> > >
> > >> > > --
> > >> > > George McCabe - Software Engineer III
> > >> > > National Center for Atmospheric Research
> > >> > > Research Applications Laboratory
> > >> > > 303-497-2768
> > >> > > ---
> > >> > > My working day may not be your working day. Please do not
feel
> > >> obliged to
> > >> > > reply to this email outside of your normal working hours.
> > >> > >
> > >> >
> > >> >
> > >> > --
> > >> > George McCabe - Software Engineer III
> > >> > National Center for Atmospheric Research
> > >> > Research Applications Laboratory
> > >> > 303-497-2768
> > >> > ---
> > >> > My working day may not be your working day. Please do not
feel
> obliged
> > >> to
> > >> > reply to this email outside of your normal working hours.
> > >> >
> > >> >
> > >>
> > >>
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: tcpairs
From: George McCabe
Time: Mon Aug 16 11:43:07 2021
Hi Guang Ping,
I was able to generate images with all of the models on Hera. There
were
modifications to the METplus wrapper and R script that were needed to
get
it to work properly on Hera. The config file I used is here:
/home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8_test2.conf
I added the following to properly pass in the data directory to the
script:
TCMPR_PLOTTER_READ_ALL_FILES = True
I had to change the value for MET_INSTALL_DIR to another location that
has
the change to the R script:
MET_INSTALL_DIR = /scratch1/BMC/dtc/George.Mccabe/met-tmp
I called run_metplus.py from my feature branch directory to get the
changes
needed:
/scratch1/BMC/dtc/George.Mccabe/METplus.feature_1057_tcmpr_dir_option/ush/run_metplus.py
I am rerunning the use case now. When it finishes the images will be
generated here:
/scratch1/BMC/dtc/George.Mccabe/tcmpr-test5/tcmpr_plots
Please take a look and let me know if the images look good to you.
Thanks,
George
On Tue, Aug 3, 2021 at 7:59 AM George McCabe <mccabe at ucar.edu> wrote:
> Hi Guang Ping,
>
> I see. Thanks for clarifying. The good news is that John HG was able
to
> generate a plot of all of the models. Attached is an image he
generated.
> This is the command he ran to generate the images:
>
> Rscript met/scripts/Rscripts/plot_tcmpr.R -lookin `find ./ -name
"*.tcst"`
> -no_ee -plot BOXPLOT,MEAN
>
> There are 2 things John mentioned to note here:
>
> * That "find" command found 2705 input files and passed them all in
on the
> command line. That's a long command line and might cause shell
problems on
> some system.
> * The "-no_ee" option DISABLES event equalization, which is needed
since
> these cases are not the same. With event equalization turned on, all
the
> cases will be tossed out.
>
> The -no_ee argument can be set by setting this in your METplus
> configuration file:
>
> TCMPR_PLOTTER_NO_EE = yes
>
> Howeer, his first comment aligns with the error I was seeing when I
tried
> to run where the shell couldn't handle the large number of files
that were
> passed to the command. The wrapper finds all of the files in the
input
> directory that end with .tcst and pass in all of the files to the
Rscript
> command, which is essentially the same behavior as the inserting the
find
> command in John's example. Hera is down today so I am unable to
test, but
> it may be the case that Hera cannot handle this command. The
plot_tcmpr.R
> script fails on this line (380), which attempts to expand any
wildcards
> with an ls command:
>
> file_list = system(paste("ls -1", paste(file_list, collapse=" ")),
> intern=TRUE);
>
> The list of files is passed from the R script into the tc_stat MET
tool
> for filtering and any wildcards are already expanded by the wrapper.
I
> suspect that if I add an option to this script to bypass the
wildcard
> expansion, then the call to tc_stat will work. I can make the
necessary
> changes locally and test it on Hera when it comes back online. If
tc_stat
> cannot handle the long command on Hera, then modifications to the
tc_stat
> tool to support reading in a text file that contains a list of files
and
> modifications to the TCMPRPlotter wrapper to generate a file list
text file
> may fix the issue.
>
> I will follow up with more information after some testing. It is
> reassuring to know that the images you are trying to create can be
> generated.
>
> Thanks,
> George
>
> On Thu, Jul 29, 2021 at 7:55 AM Guang Lou - NOAA Affiliate via RT <
> met_help at ucar.edu> wrote:
>
>> Hi George,
>>
>> Thanks for the update. The purpose of this plot is to compare error
stats
>> between different models. So we need to plot all model errors in
one frame
>> instead of each model having its own frame. I did not state clearly
in my
>> last email.
>>
>> Thanks,
>> Guang Ping
>>
>> On Wed, Jul 28, 2021 at 4:32 PM George McCabe via RT
<met_help at ucar.edu>
>> wrote:
>>
>> > Hi Guang Ping,
>> >
>> > I updated the config file to process each model separately. Here
is the
>> > config file on hera:
>> >
>> > /home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8_test.conf
>> >
>> > The output from my latest run can be found in here:
>> >
>> > /scratch1/BMC/dtc/George.Mccabe/tcmpr-test4/tcmpr_plots
>> >
>> > There is a sub-directory for each model. Please take a look at
the
>> output
>> > files to see if they look OK. The config file could be cleaned up
a
>> little
>> > bit if I add support for TCMPR_PLOTTER_CUSTOM_LOOP_LIST or
support for
>> > reading the instance name from the TCMPRPlotter instances. I will
make a
>> > GitHub issue to implement those changes as they are relatively
easy
>> > additions and would make the wrapper consistent with the other
wrappers.
>> >
>> > Thanks,
>> > George
>> >
>> > On Wed, Jul 28, 2021 at 10:02 AM George McCabe <mccabe at ucar.edu>
wrote:
>> >
>> > > Hi Guang Ping,
>> > >
>> > > OK, that is good to know. I have a few meetings this morning
but I
>> will
>> > > try to update your configuration file to set this up.
>> > >
>> > > - George
>> > >
>> > > On Wed, Jul 28, 2021 at 9:46 AM Guang Lou - NOAA Affiliate via
RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > >> Hi George,
>> > >>
>> > >> The verification should be against self analysis. That is GFS
adeck
>> > >> against
>> > >> GFS bdeck, CMC adeck against CMC bdeck, and so on and so
forth.
>> > >>
>> > >> Thanks,
>> > >> Guang Ping
>> > >>
>> > >> On Tue, Jul 27, 2021 at 2:33 PM George McCabe via RT <
>> met_help at ucar.edu
>> > >
>> > >> wrote:
>> > >>
>> > >> > Hi Guang Ping,
>> > >> >
>> > >> > I looked at this use case more today. I modified the use
case to
>> read
>> > a
>> > >> > single month into TCMPRPlotter because it cannot handle all
of the
>> > >> files at
>> > >> > once. It looks like you need to set the -bmodel in
>> > TCMPR_PLOTTER_FILTER
>> > >> to
>> > >> > specify the Bdeck to compare the models to. However, each of
the
>> > output
>> > >> > files from tc_pairs use the same model for adeck and bdeck,
so the
>> > >> AMODEL
>> > >> > and BMODEL values in those files are the same, i.e. both
GFSO. If
>> you
>> > >> want
>> > >> > to compare all of the models to a single bdeck, you will
have to
>> > >> configure
>> > >> > TCPairs wrapper to use the same bdeck for each model with
>> > >> > TC_PAIRS_BDECK_TEMPLATE. I tried running using best for the
bdeck
>> and
>> > it
>> > >> > produced no pairs. I am rerunning now using GFSO.
>> > >> >
>> > >> > - George
>> > >> >
>> > >> > On Mon, Jul 26, 2021 at 5:57 PM George McCabe
<mccabe at ucar.edu>
>> > wrote:
>> > >> >
>> > >> > > Hi Guang Ping,
>> > >> > >
>> > >> > > I copied your config file and made some changes. The new
file
>> can be
>> > >> > found
>> > >> > > on hera here:
>> > >> > >
>> > >> > > /home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8.conf
>> > >> > >
>> > >> > > Using METplus 4.1.0-beta1, you can set TC_PAIRS_RUN_ONCE =
True
>> to
>> > >> loop
>> > >> > > over the init times even with LOOP_ORDER = processes.
>> > >> > >
>> > >> > > I noticed that you have TC_PAIRS_REFORMAT_DECK = yes,
which isn't
>> > >> > > necessary because tc_pairs can read the input files
without
>> > >> reformatting.
>> > >> > > That can be unset or set to False/no.
>> > >> > >
>> > >> > > TC_PAIRS_REFORMAT_DECK = no
>> > >> > >
>> > >> > > I misread your email about trying to use the wildcard for
the
>> model.
>> > >> This
>> > >> > > won't work as you expect because the output template will
not
>> know
>> > >> how to
>> > >> > > name the output file with the correct model. Each model
will
>> > overwrite
>> > >> > the
>> > >> > > output files. To get around this you can use
>> > >> TC_PAIRS_CUSTOM_LOOP_LIST.
>> > >> > > This will run the TCPairs wrapper once for each item in
the
>> custom
>> > >> loop
>> > >> > > list. Here you can specify the list of models (lowercase)
and
>> > >> reference
>> > >> > the
>> > >> > > {custom} template tag to find the files.
>> > >> > >
>> > >> > > TC_PAIRS_CUSTOM_LOOP_LIST = best, cmc, emx, gfso, nam,
ngx, ukx
>> > >> > >
>> > >> > > TC_PAIRS_ADECK_TEMPLATE =
>> > >> > >
>> > >> >
>> > >>
>> >
>>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
>> > >> > > TC_PAIRS_BDECK_TEMPLATE =
>> > >> > >
>> > >> >
>> > >>
>> >
>>
{date?fmt=%Y%m}/b{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
>> > >> > > TC_PAIRS_OUTPUT_TEMPLATE =
>> > >> > >
>> > >> >
>> > >>
>> >
>>
{date?fmt=%Y%m}/{basin?fmt=%s}q{date?fmt=%Y%m%d%H}.{custom}.{cyclone?fmt=%s}
>> > >> > >
>> > >> > > I removed the MODEL values (you had it twice in your
config file)
>> > >> because
>> > >> > > we don't need to filter by the model if we are reading all
of
>> them
>> > in
>> > >> > > individually.
>> > >> > >
>> > >> > > Using this config file I was able to run TCPairs for each
model
>> > >> > processing
>> > >> > > each for all 3 months and creating output files from
tc_pairs
>> here:
>> > >> > >
>> > >> > > /scratch1/BMC/dtc/George.Mccabe/tcmpr-test2/tc_pairs
>> > >> > >
>> > >> > > The error in your log file from TCMPRPlotter mentions that
you do
>> > not
>> > >> > have
>> > >> > > -bmodel set in the filter options. When I ran it, I got an
error
>> > with
>> > >> a
>> > >> > > command that appears to list the files which leads me to
think
>> that
>> > >> there
>> > >> > > are too many files in the tc_pairs directory to process at
once.
>> I
>> > >> need
>> > >> > to
>> > >> > > stop working for today, but I will resume debugging this
tomorrow
>> > and
>> > >> see
>> > >> > > what I can find. It may be the case that the plotter
cannot
>> handle
>> > >> all 7
>> > >> > > models over a 3 day period all at once. If this is the
case, what
>> > >> would
>> > >> > be
>> > >> > > useful to display as a subset of this data?
>> > >> > >
>> > >> > > Thanks,
>> > >> > > George
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > > On Fri, Jul 23, 2021 at 11:46 AM Guang Lou - NOAA
Affiliate via
>> RT <
>> > >> > > met_help at ucar.edu> wrote:
>> > >> > >
>> > >> > >>
>> > >> > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100602
>> >
>> > >> > >>
>> > >> > >> Hi George,
>> > >> > >>
>> > >> > >> Thanks for letting me know you have downloaded the files.
I've
>> > >> removed
>> > >> > the
>> > >> > >> files to save disk space.
>> > >> > >> Per your suggestion, I was able to generate consecutive
months
>> of
>> > >> paired
>> > >> > >> tracks by changing a few lines
>> > >> > >> and parameters in the configure file. I've also loaded
the
>> module
>> > >> > >> "metplus/4.1.0-beta1".
>> > >> > >>
>> > >> > >> This was done by:
>> > >> > >> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs
>> > >> > >> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c
>> > >> > >> usecase_pairing_8.conf
>> > >> > >>
>> > >> > >> However, when I tried to generate plots from the just
generated
>> > >> pairs,
>> > >> > it
>> > >> > >> ran into error:
>> > >> > >> in
>> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_pairs
>> > >> > >> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c
>> usecase_8.conf
>> > >> > >>
>> > >> > >> where I changed the following to:
>> > >> > >> LOOP_ORDER = processes
>> > >> > >> PROCESS_LIST = TCMPRPlotter
>> > >> > >> error messages are:
>> > >> > >> 07/23 17:38:51.477 metplus INFO: Check the logfile for
more
>> > >> information
>> > >> > on
>> > >> > >> why it failed:
>> > >> > >>
>> > >> > >>
>> > >> >
>> > >>
>> >
>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
>> > >> > >> 07/23 17:38:51.478 metplus INFO: Plotting complete
>> > >> > >> 07/23 17:38:51.481 metplus.TCMPRPlotter (met_util.py:171)
ERROR:
>> > >> > >> TCMPRPlotter had 9 errors.
>> > >> > >> 07/23 17:38:51.481 metplus INFO:
>> > >> > >>
>> > >> > >>
>> > >> >
>> > >>
>> >
>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/metplus_final.conf:
>> > >> > >> write metplus.conf here
>> > >> > >> 07/23 17:38:51.486 metplus (met_util.py:208) ERROR:
METplus has
>> > >> finished
>> > >> > >> running but had 9 errors.
>> > >> > >> 07/23 17:38:51.486 metplus INFO: Check the log file for
more
>> > >> > information:
>> > >> > >>
>> > >> > >>
>> > >> >
>> > >>
>> >
>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
>> > >> > >>
-----------------------------------------------------------
>> > >> > >> The wild card in the place of
>> > >> > >> TC_PAIRS_ADECK_TEMPLATE =
>> > >> > >>
>> >
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.nam.{cyclone?fmt=%s}
>> > >> > >> to
>> > >> > >> TC_PAIRS_ADECK_TEMPLATE =
>> > >> > >>
>> {date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
>> > >> > >> still does not work.
>> > >> > >>
>> > >> > >> Thanks,
>> > >> > >> Guang Ping
>> > >> > >>
>> > >> > >> On Wed, Jul 21, 2021 at 2:41 PM George McCabe via RT <
>> > >> met_help at ucar.edu
>> > >> > >
>> > >> > >> wrote:
>> > >> > >>
>> > >> > >> > Hi Guang Ping,
>> > >> > >> >
>> > >> > >> > I have finished downloading the GFS NetCDF files from
Hera so
>> you
>> > >> can
>> > >> > >> > remove them. Thanks again for providing that data.
>> > >> > >> >
>> > >> > >> > - George
>> > >> > >> >
>> > >> > >> > On Tue, Jul 20, 2021 at 3:35 PM George McCabe <
>> mccabe at ucar.edu>
>> > >> > wrote:
>> > >> > >> >
>> > >> > >> > > Hi Guang Ping,
>> > >> > >> > >
>> > >> > >> > > I mistyped in my previous email. The variable to set
to
>> "times"
>> > >> is
>> > >> > >> > > LOOP_ORDER not LOOP_BY.
>> > >> > >> > >
>> > >> > >> > > LOOP_ORDER = times
>> > >> > >> > > INIT_INCREMENT = 1m
>> > >> > >> > >
>> > >> > >> > > Sorry for the confusion.
>> > >> > >> > >
>> > >> > >> > > - George
>> > >> > >> > >
>> > >> > >> > > On Tue, Jul 20, 2021 at 3:21 PM George McCabe <
>> mccabe at ucar.edu
>> > >
>> > >> > >> wrote:
>> > >> > >> > >
>> > >> > >> > >> Hi Guang Ping,
>> > >> > >> > >>
>> > >> > >> > >> Thank you for providing the NetCDF files. I am
currently
>> > >> > downloading
>> > >> > >> one
>> > >> > >> > >> of the files but it is going very slowly.
>> > >> > >> > >>
>> > >> > >> > >> In regards to your use case, the logic for TCPairs
wrapper
>> is
>> > a
>> > >> > >> little
>> > >> > >> > >> strange. It behaves differently based on the value
of
>> LOOP_BY.
>> > >> If
>> > >> > >> > LOOP_BY =
>> > >> > >> > >> processes, then it will run once using the INIT_BEG
time as
>> > the
>> > >> run
>> > >> > >> > time to
>> > >> > >> > >> find input files. If LOOP_BY = times, then it will
loop
>> over
>> > >> each
>> > >> > run
>> > >> > >> > time
>> > >> > >> > >> specified between INIT_BEG and INIT_END by
INIT_INCREMENT
>> and
>> > >> find
>> > >> > >> > files to
>> > >> > >> > >> process for each run time.
>> > >> > >> > >>
>> > >> > >> > >> If you change your configuration file to include
LOOP_BY =
>> > >> times,
>> > >> > >> then
>> > >> > >> > it
>> > >> > >> > >> should look over each run time and process for each.
It
>> looks
>> > >> like
>> > >> > >> your
>> > >> > >> > >> INIT_INCREMENT is set to 21600 or 6 hours, so it
will run
>> > every
>> > >> 6
>> > >> > >> hours
>> > >> > >> > >> from 20200201 to 2020331. If you want to run once
per
>> month,
>> > you
>> > >> > will
>> > >> > >> > need
>> > >> > >> > >> to set INIT_INCREMENT = 1m so that it will increment
by 1
>> > month
>> > >> for
>> > >> > >> each
>> > >> > >> > >> iteration.
>> > >> > >> > >>
>> > >> > >> > >> The caveat is that TCMPRPlotter will only run if
LOOP_BY =
>> > >> > processes
>> > >> > >> and
>> > >> > >> > >> will produce an error if LOOP_BY = times. This is
because
>> we
>> > >> > >> typically
>> > >> > >> > want
>> > >> > >> > >> to process all of the data that is available for all
run
>> > times,
>> > >> > then
>> > >> > >> run
>> > >> > >> > >> TCMPRPlotter so it can read all of the data. For
now, to
>> > handle
>> > >> > this
>> > >> > >> use
>> > >> > >> > >> case you will have to call run_metplus.py 2 separate
times,
>> > once
>> > >> > >> calling
>> > >> > >> > >> TCPairs with LOOP_BY = times and once calling
TCMPRPlotter
>> > with
>> > >> > >> LOOP_BY
>> > >> > >> > =
>> > >> > >> > >> processes.
>> > >> > >> > >>
>> > >> > >> > >> I am going to add a configuration option to TCPairs
>> wrapper so
>> > >> that
>> > >> > >> you
>> > >> > >> > >> can loop over the run times when LOOP_BY = processes
is
>> set if
>> > >> > >> desired.
>> > >> > >> > I
>> > >> > >> > >> will try to get this change into METplus 4.1.0-beta1
so you
>> > can
>> > >> > test
>> > >> > >> the
>> > >> > >> > >> change and make sure it works as expected. I can let
you
>> know
>> > >> when
>> > >> > >> that
>> > >> > >> > is
>> > >> > >> > >> available.
>> > >> > >> > >>
>> > >> > >> > >> Thanks,
>> > >> > >> > >> George
>> > >> > >> > >>
>> > >> > >> > >> On Tue, Jul 20, 2021 at 1:27 PM Guang Lou - NOAA
Affiliate
>> via
>> > >> RT <
>> > >> > >> > >> met_help at ucar.edu> wrote:
>> > >> > >> > >>
>> > >> > >> > >>> Hi John,
>> > >> > >> > >>>
>> > >> > >> > >>> Thanks for forwarding George's email answer to me.
I
>> created
>> > a
>> > >> > test
>> > >> > >> > case
>> > >> > >> > >>> that does the tcpairs only on Hera:
>> > >> > >> > >>>
>> > /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test3
>> > >> > >> > >>> Command line:
>> > >> > >> > >>> run_metplus.py -c
Plotter_fcstGFS_obsGFS_RPlotting_8.conf
>> -c
>> > >> > >> > >>> usecase_8.conf
>> > >> > >> > >>> where PROCESS_LIST = TCPairs
>> > >> > >> > >>>
>> > >> > >> > >>> It generated the month of 202002, that is because I
>> changed
>> > >> > >> > >>> INIT_BEG = 20200101
>> > >> > >> > >>> INIT_END = 20200331
>> > >> > >> > >>> to
>> > >> > >> > >>> INIT_BEG = 20200201
>> > >> > >> > >>> INIT_END = 20200331
>> > >> > >> > >>>
>> > >> > >> > >>> If I use "INIT_BEG = 20200101,INIT_END = 20200331"
it
>> will
>> > >> > generate
>> > >> > >> > >>> 20200101 only.
>> > >> > >> > >>>
>> > >> > >> > >>> The wild card does not work either for the model's
place:
>> > >> > >> > >>> TC_PAIRS_ADECK_TEMPLATE =
>> > >> > >> > >>>
>> > >> >
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
>> > >> > >> > >>> I populated adeck and bdeck on
>> > >> > >> > >>>
>> > >> > >> > >>>
>> > >> > >> >
>> > >> > >>
>> > >> >
>> > >>
>> >
>> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/METplus/METplus-
4.0_sample_data/model_applications/medium_range/tracks_all
>> > >> > >> > >>> (each subdirectory contains all three months'
decks.)
>> > >> > >> > >>>
>> > >> > >> > >>> I was wondering if you could copy my directory,
>> > >> > >> > >>>
>> > /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test3
>> > >> > >> > >>> and test:
>> > >> > >> > >>> 1. It can generate multiple months instead of just
one
>> month
>> > >> and
>> > >> > one
>> > >> > >> > >>> model.
>> > >> > >> > >>> 2. The wild card for models in the DECK_TEMPATE for
>> multiple
>> > >> > models.
>> > >> > >> > >>>
>> > >> > >> > >>> George, I'm downloading NetCDF model data to Hera:
>> > >> > >> > >>>
>> > >> >
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071300_nc.tar
>> > >> > >> > >>>
>> > >> >
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071306_nc.tar
>> > >> > >> > >>>
>> > >> > >> > >>> Please let me know after you downloaded the files
and I
>> will
>> > >> have
>> > >> > to
>> > >> > >> > >>> remove
>> > >> > >> > >>> them due to tight disk space.
>> > >> > >> > >>>
>> > >> > >> > >>> Thanks,
>> > >> > >> > >>> Guang Ping
>> > >> > >> > >>>
>> > >> > >> > >>> On Mon, Jul 19, 2021 at 5:01 PM John Halley Gotway
via RT
>> <
>> > >> > >> > >>> met_help at ucar.edu>
>> > >> > >> > >>> wrote:
>> > >> > >> > >>>
>> > >> > >> > >>> > Hi Guang Ping,
>> > >> > >> > >>> >
>> > >> > >> > >>> > The command that you ran here:
>> > >> > >> > >>> >
>> > >> > >> > >>> >
>> > >> > >> > >>> >
>> > >> > >> > >>>
>> > >> > >> >
>> > >> > >>
>> > >> >
>> > >>
>> >
>>
/home/George.Mccabe/scratch1_George.Mccabe/METplus.feature_926/ush/run_metplus.py
>> > >> > >> > >>> > -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c
>> > Plotter_8.conf
>> > >> > >> > >>> >
>> > >> > >> > >>> > only runs TCMPRPlotter. This is because
PROCESS_LIST is
>> > >> defined
>> > >> > in
>> > >> > >> > both
>> > >> > >> > >>> > Plotter_fcstGFS_obsGFS_RPlotting_8.conf and
>> Plotter_8.conf.
>> > >> > Since
>> > >> > >> > >>> > Plotter_8.conf comes last, then the value for
>> PROCESS_LIST
>> > >> from
>> > >> > >> that
>> > >> > >> > >>> file
>> > >> > >> > >>> > is used, which only has TCMPRPlotter listed. The
>> plotter is
>> > >> > >> reading
>> > >> > >> > >>> all of
>> > >> > >> > >>> > the available files generated from TCPairs and
since
>> that
>> > >> > process
>> > >> > >> is
>> > >> > >> > >>> not
>> > >> > >> > >>> > running again with the new time range, the data
that is
>> > read
>> > >> > only
>> > >> > >> > >>> covers
>> > >> > >> > >>> > the first month.
>> > >> > >> > >>> >
>> > >> > >> > >>> > We are planning on releasing METplus 4.1.0-beta1
this
>> week.
>> > >> > >> However,
>> > >> > >> > >>> if I
>> > >> > >> > >>> > understand what you are trying to do here, it
shouldn't
>> > >> require
>> > >> > >> the
>> > >> > >> > >>> changes
>> > >> > >> > >>> > described in Molly's issue. It looks like you
want to
>> run
>> > >> > TCPairs
>> > >> > >> > once
>> > >> > >> > >>> and
>> > >> > >> > >>> > filter the data based on the init and valid
times. While
>> > this
>> > >> > >> > >>> configuration
>> > >> > >> > >>> > should achieve what you are trying to do, the
following
>> > >> > >> configuration
>> > >> > >> > >>> is
>> > >> > >> > >>> > more explicit to what you are actually running:
>> > >> > >> > >>> >
>> > >> > >> > >>> > # run once using 20200101 to substitute the init
time in
>> > your
>> > >> > >> string
>> > >> > >> > >>> > substitution, i.e. TC_PAIRS_ADECK_TEMPLATE =
>> > >> > >> > >>> >
>> > >> > >>
>> {date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
>> > >> > >> > >>> > INIT_BEG = 20200101
>> > >> > >> > >>> > INIT_END = 20200101
>> > >> > >> > >>> >
>> > >> > >> > >>> > # filter the init times of the lines read from
the input
>> > >> files
>> > >> > in
>> > >> > >> the
>> > >> > >> > >>> > TCPairs configuration file
>> > >> > >> > >>> > TC_PAIRS_INIT_BEG = 20200101
>> > >> > >> > >>> > TC_PAIRS_INIT_END = 20200331
>> > >> > >> > >>> >
>> > >> > >> > >>> > # filter the valid times of the lines read from
the
>> input
>> > >> files
>> > >> > in
>> > >> > >> > the
>> > >> > >> > >>> > TCPairs configuration file
>> > >> > >> > >>> > TC_PAIRS_VALID_BEG = 20200101
>> > >> > >> > >>> > TC_PAIRS_VALID_END = 20200331
>> > >> > >> > >>> >
>> > >> > >> > >>> > If any of TC_PAIRS_[INIT/VALID]_[BEG/END] are not
set,
>> then
>> > >> the
>> > >> > >> > >>> > corresponding [INIT/VALID]_[BEG/END] values are
used.
>> This
>> > is
>> > >> > >> done to
>> > >> > >> > >>> > preserve backwards compatibility for old METplus
>> > >> configurations.
>> > >> > >> The
>> > >> > >> > >>> above
>> > >> > >> > >>> > settings are more explicit as to what is actually
be
>> > >> configured.
>> > >> > >> > >>> >
>> > >> > >> > >>> > You may want to consider either clearing your
output
>> > >> directory
>> > >> > or
>> > >> > >> > >>> creating
>> > >> > >> > >>> > a new output directory for a new run (especially
if the
>> > time
>> > >> > range
>> > >> > >> > you
>> > >> > >> > >>> are
>> > >> > >> > >>> > using has changed) to ensure that the files you
are
>> > generated
>> > >> > are
>> > >> > >> > from
>> > >> > >> > >>> that
>> > >> > >> > >>> > run and not from a previous run. Doing this would
have
>> > >> alerted
>> > >> > you
>> > >> > >> > that
>> > >> > >> > >>> > TCPairs was not run in this case.
>> > >> > >> > >>> >
>> > >> > >> > >>> > Let me know if you have any other issues getting
this
>> > >> running.
>> > >> > >> > >>> >
>> > >> > >> > >>> > FYI:
>> > >> > >> > >>> > We switched from providing support through this
email
>> > >> address to
>> > >> > >> > >>> providing
>> > >> > >> > >>> > support through GitHub Discussions. In the
future, you
>> will
>> > >> need
>> > >> > >> to
>> > >> > >> > >>> create
>> > >> > >> > >>> > a free GitHub account if you don’t have one
already and
>> > post
>> > >> > your
>> > >> > >> > >>> questions
>> > >> > >> > >>> > to the METplus Components Discussion Forum:
>> > >> > >> > >>> > https://github.com/dtcenter/METplus/discussions
>> > >> > >> > >>> > To post a new question, click on the green button
in the
>> > >> upper
>> > >> > >> right
>> > >> > >> > >>> corner
>> > >> > >> > >>> > of the page labeled "New discussion" and select
>> "Incoming"
>> > as
>> > >> > the
>> > >> > >> > >>> category.
>> > >> > >> > >>> >
>> > >> > >> > >>> > Thanks,
>> > >> > >> > >>> > George
>> > >> > >> > >>> >
>> > >> > >> > >>> >
>> > >> > >> > >>>
>> > >> > >> > >>>
>> > >> > >> > >>
>> > >> > >> > >> --
>> > >> > >> > >> George McCabe - Software Engineer III
>> > >> > >> > >> National Center for Atmospheric Research
>> > >> > >> > >> Research Applications Laboratory
>> > >> > >> > >> 303-497-2768
>> > >> > >> > >> ---
>> > >> > >> > >> My working day may not be your working day. Please
do not
>> feel
>> > >> > >> obliged
>> > >> > >> > to
>> > >> > >> > >> reply to this email outside of your normal working
hours.
>> > >> > >> > >>
>> > >> > >> > >
>> > >> > >> > >
>> > >> > >> > > --
>> > >> > >> > > George McCabe - Software Engineer III
>> > >> > >> > > National Center for Atmospheric Research
>> > >> > >> > > Research Applications Laboratory
>> > >> > >> > > 303-497-2768
>> > >> > >> > > ---
>> > >> > >> > > My working day may not be your working day. Please do
not
>> feel
>> > >> > >> obliged to
>> > >> > >> > > reply to this email outside of your normal working
hours.
>> > >> > >> > >
>> > >> > >> >
>> > >> > >> >
>> > >> > >> > --
>> > >> > >> > George McCabe - Software Engineer III
>> > >> > >> > National Center for Atmospheric Research
>> > >> > >> > Research Applications Laboratory
>> > >> > >> > 303-497-2768
>> > >> > >> > ---
>> > >> > >> > My working day may not be your working day. Please do
not feel
>> > >> obliged
>> > >> > >> to
>> > >> > >> > reply to this email outside of your normal working
hours.
>> > >> > >> >
>> > >> > >> >
>> > >> > >>
>> > >> > >>
>> > >> > >
>> > >> > > --
>> > >> > > George McCabe - Software Engineer III
>> > >> > > National Center for Atmospheric Research
>> > >> > > Research Applications Laboratory
>> > >> > > 303-497-2768
>> > >> > > ---
>> > >> > > My working day may not be your working day. Please do not
feel
>> > >> obliged to
>> > >> > > reply to this email outside of your normal working hours.
>> > >> > >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > George McCabe - Software Engineer III
>> > >> > National Center for Atmospheric Research
>> > >> > Research Applications Laboratory
>> > >> > 303-497-2768
>> > >> > ---
>> > >> > My working day may not be your working day. Please do not
feel
>> obliged
>> > >> to
>> > >> > reply to this email outside of your normal working hours.
>> > >> >
>> > >> >
>> > >>
>> > >>
>> > >
>> > > --
>> > > George McCabe - Software Engineer III
>> > > National Center for Atmospheric Research
>> > > Research Applications Laboratory
>> > > 303-497-2768
>> > > ---
>> > > My working day may not be your working day. Please do not feel
>> obliged to
>> > > reply to this email outside of your normal working hours.
>> > >
>> >
>> >
>> > --
>> > George McCabe - Software Engineer III
>> > National Center for Atmospheric Research
>> > Research Applications Laboratory
>> > 303-497-2768
>> > ---
>> > My working day may not be your working day. Please do not feel
obliged
>> to
>> > reply to this email outside of your normal working hours.
>> >
>> >
>>
>>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: tcpairs
From: Guang Lou - NOAA Affiliate
Time: Tue Aug 17 06:28:01 2021
Hi George,
Thank you so much for figuring out the plotting scripts! Following
your
suggestion, I was able to generate plots that put all model errors on
the
same frame. I've generated the 3 months' average values and I will do
a
longer term average.
One question for you, is there a place that I can turn off the error
range
bar (the sigma range from low to high)? When we have many models,
these
bars would crowd out the error lines.
Your help is much appreciated!
Guang Ping
On Mon, Aug 16, 2021 at 1:43 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Guang Ping,
>
> I was able to generate images with all of the models on Hera. There
were
> modifications to the METplus wrapper and R script that were needed
to get
> it to work properly on Hera. The config file I used is here:
>
> /home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8_test2.conf
>
> I added the following to properly pass in the data directory to the
script:
>
> TCMPR_PLOTTER_READ_ALL_FILES = True
>
> I had to change the value for MET_INSTALL_DIR to another location
that has
> the change to the R script:
>
> MET_INSTALL_DIR = /scratch1/BMC/dtc/George.Mccabe/met-tmp
>
> I called run_metplus.py from my feature branch directory to get the
changes
> needed:
>
>
>
/scratch1/BMC/dtc/George.Mccabe/METplus.feature_1057_tcmpr_dir_option/ush/run_metplus.py
>
> I am rerunning the use case now. When it finishes the images will be
> generated here:
>
> /scratch1/BMC/dtc/George.Mccabe/tcmpr-test5/tcmpr_plots
>
> Please take a look and let me know if the images look good to you.
>
> Thanks,
> George
>
> On Tue, Aug 3, 2021 at 7:59 AM George McCabe <mccabe at ucar.edu>
wrote:
>
> > Hi Guang Ping,
> >
> > I see. Thanks for clarifying. The good news is that John HG was
able to
> > generate a plot of all of the models. Attached is an image he
generated.
> > This is the command he ran to generate the images:
> >
> > Rscript met/scripts/Rscripts/plot_tcmpr.R -lookin `find ./ -name
> "*.tcst"`
> > -no_ee -plot BOXPLOT,MEAN
> >
> > There are 2 things John mentioned to note here:
> >
> > * That "find" command found 2705 input files and passed them all
in on
> the
> > command line. That's a long command line and might cause shell
problems
> on
> > some system.
> > * The "-no_ee" option DISABLES event equalization, which is needed
since
> > these cases are not the same. With event equalization turned on,
all the
> > cases will be tossed out.
> >
> > The -no_ee argument can be set by setting this in your METplus
> > configuration file:
> >
> > TCMPR_PLOTTER_NO_EE = yes
> >
> > Howeer, his first comment aligns with the error I was seeing when
I tried
> > to run where the shell couldn't handle the large number of files
that
> were
> > passed to the command. The wrapper finds all of the files in the
input
> > directory that end with .tcst and pass in all of the files to the
Rscript
> > command, which is essentially the same behavior as the inserting
the find
> > command in John's example. Hera is down today so I am unable to
test, but
> > it may be the case that Hera cannot handle this command. The
plot_tcmpr.R
> > script fails on this line (380), which attempts to expand any
wildcards
> > with an ls command:
> >
> > file_list = system(paste("ls -1", paste(file_list, collapse=" ")),
> > intern=TRUE);
> >
> > The list of files is passed from the R script into the tc_stat MET
tool
> > for filtering and any wildcards are already expanded by the
wrapper. I
> > suspect that if I add an option to this script to bypass the
wildcard
> > expansion, then the call to tc_stat will work. I can make the
necessary
> > changes locally and test it on Hera when it comes back online. If
tc_stat
> > cannot handle the long command on Hera, then modifications to the
tc_stat
> > tool to support reading in a text file that contains a list of
files and
> > modifications to the TCMPRPlotter wrapper to generate a file list
text
> file
> > may fix the issue.
> >
> > I will follow up with more information after some testing. It is
> > reassuring to know that the images you are trying to create can be
> > generated.
> >
> > Thanks,
> > George
> >
> > On Thu, Jul 29, 2021 at 7:55 AM Guang Lou - NOAA Affiliate via RT
<
> > met_help at ucar.edu> wrote:
> >
> >> Hi George,
> >>
> >> Thanks for the update. The purpose of this plot is to compare
error
> stats
> >> between different models. So we need to plot all model errors in
one
> frame
> >> instead of each model having its own frame. I did not state
clearly in
> my
> >> last email.
> >>
> >> Thanks,
> >> Guang Ping
> >>
> >> On Wed, Jul 28, 2021 at 4:32 PM George McCabe via RT
<met_help at ucar.edu
> >
> >> wrote:
> >>
> >> > Hi Guang Ping,
> >> >
> >> > I updated the config file to process each model separately.
Here is
> the
> >> > config file on hera:
> >> >
> >> > /home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8_test.conf
> >> >
> >> > The output from my latest run can be found in here:
> >> >
> >> > /scratch1/BMC/dtc/George.Mccabe/tcmpr-test4/tcmpr_plots
> >> >
> >> > There is a sub-directory for each model. Please take a look at
the
> >> output
> >> > files to see if they look OK. The config file could be cleaned
up a
> >> little
> >> > bit if I add support for TCMPR_PLOTTER_CUSTOM_LOOP_LIST or
support for
> >> > reading the instance name from the TCMPRPlotter instances. I
will
> make a
> >> > GitHub issue to implement those changes as they are relatively
easy
> >> > additions and would make the wrapper consistent with the other
> wrappers.
> >> >
> >> > Thanks,
> >> > George
> >> >
> >> > On Wed, Jul 28, 2021 at 10:02 AM George McCabe
<mccabe at ucar.edu>
> wrote:
> >> >
> >> > > Hi Guang Ping,
> >> > >
> >> > > OK, that is good to know. I have a few meetings this morning
but I
> >> will
> >> > > try to update your configuration file to set this up.
> >> > >
> >> > > - George
> >> > >
> >> > > On Wed, Jul 28, 2021 at 9:46 AM Guang Lou - NOAA Affiliate
via RT <
> >> > > met_help at ucar.edu> wrote:
> >> > >
> >> > >> Hi George,
> >> > >>
> >> > >> The verification should be against self analysis. That is
GFS adeck
> >> > >> against
> >> > >> GFS bdeck, CMC adeck against CMC bdeck, and so on and so
forth.
> >> > >>
> >> > >> Thanks,
> >> > >> Guang Ping
> >> > >>
> >> > >> On Tue, Jul 27, 2021 at 2:33 PM George McCabe via RT <
> >> met_help at ucar.edu
> >> > >
> >> > >> wrote:
> >> > >>
> >> > >> > Hi Guang Ping,
> >> > >> >
> >> > >> > I looked at this use case more today. I modified the use
case to
> >> read
> >> > a
> >> > >> > single month into TCMPRPlotter because it cannot handle
all of
> the
> >> > >> files at
> >> > >> > once. It looks like you need to set the -bmodel in
> >> > TCMPR_PLOTTER_FILTER
> >> > >> to
> >> > >> > specify the Bdeck to compare the models to. However, each
of the
> >> > output
> >> > >> > files from tc_pairs use the same model for adeck and
bdeck, so
> the
> >> > >> AMODEL
> >> > >> > and BMODEL values in those files are the same, i.e. both
GFSO. If
> >> you
> >> > >> want
> >> > >> > to compare all of the models to a single bdeck, you will
have to
> >> > >> configure
> >> > >> > TCPairs wrapper to use the same bdeck for each model with
> >> > >> > TC_PAIRS_BDECK_TEMPLATE. I tried running using best for
the bdeck
> >> and
> >> > it
> >> > >> > produced no pairs. I am rerunning now using GFSO.
> >> > >> >
> >> > >> > - George
> >> > >> >
> >> > >> > On Mon, Jul 26, 2021 at 5:57 PM George McCabe
<mccabe at ucar.edu>
> >> > wrote:
> >> > >> >
> >> > >> > > Hi Guang Ping,
> >> > >> > >
> >> > >> > > I copied your config file and made some changes. The new
file
> >> can be
> >> > >> > found
> >> > >> > > on hera here:
> >> > >> > >
> >> > >> > > /home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8.conf
> >> > >> > >
> >> > >> > > Using METplus 4.1.0-beta1, you can set TC_PAIRS_RUN_ONCE
= True
> >> to
> >> > >> loop
> >> > >> > > over the init times even with LOOP_ORDER = processes.
> >> > >> > >
> >> > >> > > I noticed that you have TC_PAIRS_REFORMAT_DECK = yes,
which
> isn't
> >> > >> > > necessary because tc_pairs can read the input files
without
> >> > >> reformatting.
> >> > >> > > That can be unset or set to False/no.
> >> > >> > >
> >> > >> > > TC_PAIRS_REFORMAT_DECK = no
> >> > >> > >
> >> > >> > > I misread your email about trying to use the wildcard
for the
> >> model.
> >> > >> This
> >> > >> > > won't work as you expect because the output template
will not
> >> know
> >> > >> how to
> >> > >> > > name the output file with the correct model. Each model
will
> >> > overwrite
> >> > >> > the
> >> > >> > > output files. To get around this you can use
> >> > >> TC_PAIRS_CUSTOM_LOOP_LIST.
> >> > >> > > This will run the TCPairs wrapper once for each item in
the
> >> custom
> >> > >> loop
> >> > >> > > list. Here you can specify the list of models
(lowercase) and
> >> > >> reference
> >> > >> > the
> >> > >> > > {custom} template tag to find the files.
> >> > >> > >
> >> > >> > > TC_PAIRS_CUSTOM_LOOP_LIST = best, cmc, emx, gfso, nam,
ngx, ukx
> >> > >> > >
> >> > >> > > TC_PAIRS_ADECK_TEMPLATE =
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
> >> > >> > > TC_PAIRS_BDECK_TEMPLATE =
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
>
{date?fmt=%Y%m}/b{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
> >> > >> > > TC_PAIRS_OUTPUT_TEMPLATE =
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
>
{date?fmt=%Y%m}/{basin?fmt=%s}q{date?fmt=%Y%m%d%H}.{custom}.{cyclone?fmt=%s}
> >> > >> > >
> >> > >> > > I removed the MODEL values (you had it twice in your
config
> file)
> >> > >> because
> >> > >> > > we don't need to filter by the model if we are reading
all of
> >> them
> >> > in
> >> > >> > > individually.
> >> > >> > >
> >> > >> > > Using this config file I was able to run TCPairs for
each model
> >> > >> > processing
> >> > >> > > each for all 3 months and creating output files from
tc_pairs
> >> here:
> >> > >> > >
> >> > >> > > /scratch1/BMC/dtc/George.Mccabe/tcmpr-test2/tc_pairs
> >> > >> > >
> >> > >> > > The error in your log file from TCMPRPlotter mentions
that you
> do
> >> > not
> >> > >> > have
> >> > >> > > -bmodel set in the filter options. When I ran it, I got
an
> error
> >> > with
> >> > >> a
> >> > >> > > command that appears to list the files which leads me to
think
> >> that
> >> > >> there
> >> > >> > > are too many files in the tc_pairs directory to process
at
> once.
> >> I
> >> > >> need
> >> > >> > to
> >> > >> > > stop working for today, but I will resume debugging this
> tomorrow
> >> > and
> >> > >> see
> >> > >> > > what I can find. It may be the case that the plotter
cannot
> >> handle
> >> > >> all 7
> >> > >> > > models over a 3 day period all at once. If this is the
case,
> what
> >> > >> would
> >> > >> > be
> >> > >> > > useful to display as a subset of this data?
> >> > >> > >
> >> > >> > > Thanks,
> >> > >> > > George
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > > On Fri, Jul 23, 2021 at 11:46 AM Guang Lou - NOAA
Affiliate via
> >> RT <
> >> > >> > > met_help at ucar.edu> wrote:
> >> > >> > >
> >> > >> > >>
> >> > >> > >> <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100602
> >> >
> >> > >> > >>
> >> > >> > >> Hi George,
> >> > >> > >>
> >> > >> > >> Thanks for letting me know you have downloaded the
files. I've
> >> > >> removed
> >> > >> > the
> >> > >> > >> files to save disk space.
> >> > >> > >> Per your suggestion, I was able to generate consecutive
months
> >> of
> >> > >> paired
> >> > >> > >> tracks by changing a few lines
> >> > >> > >> and parameters in the configure file. I've also loaded
the
> >> module
> >> > >> > >> "metplus/4.1.0-beta1".
> >> > >> > >>
> >> > >> > >> This was done by:
> >> > >> > >> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs
> >> > >> > >> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c
> >> > >> > >> usecase_pairing_8.conf
> >> > >> > >>
> >> > >> > >> However, when I tried to generate plots from the just
> generated
> >> > >> pairs,
> >> > >> > it
> >> > >> > >> ran into error:
> >> > >> > >> in
> >> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_pairs
> >> > >> > >> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c
> >> usecase_8.conf
> >> > >> > >>
> >> > >> > >> where I changed the following to:
> >> > >> > >> LOOP_ORDER = processes
> >> > >> > >> PROCESS_LIST = TCMPRPlotter
> >> > >> > >> error messages are:
> >> > >> > >> 07/23 17:38:51.477 metplus INFO: Check the logfile for
more
> >> > >> information
> >> > >> > on
> >> > >> > >> why it failed:
> >> > >> > >>
> >> > >> > >>
> >> > >> >
> >> > >>
> >> >
> >>
> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
> >> > >> > >> 07/23 17:38:51.478 metplus INFO: Plotting complete
> >> > >> > >> 07/23 17:38:51.481 metplus.TCMPRPlotter
(met_util.py:171)
> ERROR:
> >> > >> > >> TCMPRPlotter had 9 errors.
> >> > >> > >> 07/23 17:38:51.481 metplus INFO:
> >> > >> > >>
> >> > >> > >>
> >> > >> >
> >> > >>
> >> >
> >>
> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/metplus_final.conf:
> >> > >> > >> write metplus.conf here
> >> > >> > >> 07/23 17:38:51.486 metplus (met_util.py:208) ERROR:
METplus
> has
> >> > >> finished
> >> > >> > >> running but had 9 errors.
> >> > >> > >> 07/23 17:38:51.486 metplus INFO: Check the log file for
more
> >> > >> > information:
> >> > >> > >>
> >> > >> > >>
> >> > >> >
> >> > >>
> >> >
> >>
> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
> >> > >> > >>
-----------------------------------------------------------
> >> > >> > >> The wild card in the place of
> >> > >> > >> TC_PAIRS_ADECK_TEMPLATE =
> >> > >> > >>
> >> >
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.nam.{cyclone?fmt=%s}
> >> > >> > >> to
> >> > >> > >> TC_PAIRS_ADECK_TEMPLATE =
> >> > >> > >>
> >>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> >> > >> > >> still does not work.
> >> > >> > >>
> >> > >> > >> Thanks,
> >> > >> > >> Guang Ping
> >> > >> > >>
> >> > >> > >> On Wed, Jul 21, 2021 at 2:41 PM George McCabe via RT <
> >> > >> met_help at ucar.edu
> >> > >> > >
> >> > >> > >> wrote:
> >> > >> > >>
> >> > >> > >> > Hi Guang Ping,
> >> > >> > >> >
> >> > >> > >> > I have finished downloading the GFS NetCDF files from
Hera
> so
> >> you
> >> > >> can
> >> > >> > >> > remove them. Thanks again for providing that data.
> >> > >> > >> >
> >> > >> > >> > - George
> >> > >> > >> >
> >> > >> > >> > On Tue, Jul 20, 2021 at 3:35 PM George McCabe <
> >> mccabe at ucar.edu>
> >> > >> > wrote:
> >> > >> > >> >
> >> > >> > >> > > Hi Guang Ping,
> >> > >> > >> > >
> >> > >> > >> > > I mistyped in my previous email. The variable to
set to
> >> "times"
> >> > >> is
> >> > >> > >> > > LOOP_ORDER not LOOP_BY.
> >> > >> > >> > >
> >> > >> > >> > > LOOP_ORDER = times
> >> > >> > >> > > INIT_INCREMENT = 1m
> >> > >> > >> > >
> >> > >> > >> > > Sorry for the confusion.
> >> > >> > >> > >
> >> > >> > >> > > - George
> >> > >> > >> > >
> >> > >> > >> > > On Tue, Jul 20, 2021 at 3:21 PM George McCabe <
> >> mccabe at ucar.edu
> >> > >
> >> > >> > >> wrote:
> >> > >> > >> > >
> >> > >> > >> > >> Hi Guang Ping,
> >> > >> > >> > >>
> >> > >> > >> > >> Thank you for providing the NetCDF files. I am
currently
> >> > >> > downloading
> >> > >> > >> one
> >> > >> > >> > >> of the files but it is going very slowly.
> >> > >> > >> > >>
> >> > >> > >> > >> In regards to your use case, the logic for TCPairs
> wrapper
> >> is
> >> > a
> >> > >> > >> little
> >> > >> > >> > >> strange. It behaves differently based on the value
of
> >> LOOP_BY.
> >> > >> If
> >> > >> > >> > LOOP_BY =
> >> > >> > >> > >> processes, then it will run once using the
INIT_BEG time
> as
> >> > the
> >> > >> run
> >> > >> > >> > time to
> >> > >> > >> > >> find input files. If LOOP_BY = times, then it will
loop
> >> over
> >> > >> each
> >> > >> > run
> >> > >> > >> > time
> >> > >> > >> > >> specified between INIT_BEG and INIT_END by
INIT_INCREMENT
> >> and
> >> > >> find
> >> > >> > >> > files to
> >> > >> > >> > >> process for each run time.
> >> > >> > >> > >>
> >> > >> > >> > >> If you change your configuration file to include
LOOP_BY
> =
> >> > >> times,
> >> > >> > >> then
> >> > >> > >> > it
> >> > >> > >> > >> should look over each run time and process for
each. It
> >> looks
> >> > >> like
> >> > >> > >> your
> >> > >> > >> > >> INIT_INCREMENT is set to 21600 or 6 hours, so it
will run
> >> > every
> >> > >> 6
> >> > >> > >> hours
> >> > >> > >> > >> from 20200201 to 2020331. If you want to run once
per
> >> month,
> >> > you
> >> > >> > will
> >> > >> > >> > need
> >> > >> > >> > >> to set INIT_INCREMENT = 1m so that it will
increment by 1
> >> > month
> >> > >> for
> >> > >> > >> each
> >> > >> > >> > >> iteration.
> >> > >> > >> > >>
> >> > >> > >> > >> The caveat is that TCMPRPlotter will only run if
LOOP_BY
> =
> >> > >> > processes
> >> > >> > >> and
> >> > >> > >> > >> will produce an error if LOOP_BY = times. This is
because
> >> we
> >> > >> > >> typically
> >> > >> > >> > want
> >> > >> > >> > >> to process all of the data that is available for
all run
> >> > times,
> >> > >> > then
> >> > >> > >> run
> >> > >> > >> > >> TCMPRPlotter so it can read all of the data. For
now, to
> >> > handle
> >> > >> > this
> >> > >> > >> use
> >> > >> > >> > >> case you will have to call run_metplus.py 2
separate
> times,
> >> > once
> >> > >> > >> calling
> >> > >> > >> > >> TCPairs with LOOP_BY = times and once calling
> TCMPRPlotter
> >> > with
> >> > >> > >> LOOP_BY
> >> > >> > >> > =
> >> > >> > >> > >> processes.
> >> > >> > >> > >>
> >> > >> > >> > >> I am going to add a configuration option to
TCPairs
> >> wrapper so
> >> > >> that
> >> > >> > >> you
> >> > >> > >> > >> can loop over the run times when LOOP_BY =
processes is
> >> set if
> >> > >> > >> desired.
> >> > >> > >> > I
> >> > >> > >> > >> will try to get this change into METplus 4.1.0-
beta1 so
> you
> >> > can
> >> > >> > test
> >> > >> > >> the
> >> > >> > >> > >> change and make sure it works as expected. I can
let you
> >> know
> >> > >> when
> >> > >> > >> that
> >> > >> > >> > is
> >> > >> > >> > >> available.
> >> > >> > >> > >>
> >> > >> > >> > >> Thanks,
> >> > >> > >> > >> George
> >> > >> > >> > >>
> >> > >> > >> > >> On Tue, Jul 20, 2021 at 1:27 PM Guang Lou - NOAA
> Affiliate
> >> via
> >> > >> RT <
> >> > >> > >> > >> met_help at ucar.edu> wrote:
> >> > >> > >> > >>
> >> > >> > >> > >>> Hi John,
> >> > >> > >> > >>>
> >> > >> > >> > >>> Thanks for forwarding George's email answer to
me. I
> >> created
> >> > a
> >> > >> > test
> >> > >> > >> > case
> >> > >> > >> > >>> that does the tcpairs only on Hera:
> >> > >> > >> > >>>
> >> > /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test3
> >> > >> > >> > >>> Command line:
> >> > >> > >> > >>> run_metplus.py -c
> Plotter_fcstGFS_obsGFS_RPlotting_8.conf
> >> -c
> >> > >> > >> > >>> usecase_8.conf
> >> > >> > >> > >>> where PROCESS_LIST = TCPairs
> >> > >> > >> > >>>
> >> > >> > >> > >>> It generated the month of 202002, that is because
I
> >> changed
> >> > >> > >> > >>> INIT_BEG = 20200101
> >> > >> > >> > >>> INIT_END = 20200331
> >> > >> > >> > >>> to
> >> > >> > >> > >>> INIT_BEG = 20200201
> >> > >> > >> > >>> INIT_END = 20200331
> >> > >> > >> > >>>
> >> > >> > >> > >>> If I use "INIT_BEG = 20200101,INIT_END =
20200331" it
> >> will
> >> > >> > generate
> >> > >> > >> > >>> 20200101 only.
> >> > >> > >> > >>>
> >> > >> > >> > >>> The wild card does not work either for the
model's
> place:
> >> > >> > >> > >>> TC_PAIRS_ADECK_TEMPLATE =
> >> > >> > >> > >>>
> >> > >> >
> {date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> >> > >> > >> > >>> I populated adeck and bdeck on
> >> > >> > >> > >>>
> >> > >> > >> > >>>
> >> > >> > >> >
> >> > >> > >>
> >> > >> >
> >> > >>
> >> >
> >>
> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/METplus/METplus-
4.0_sample_data/model_applications/medium_range/tracks_all
> >> > >> > >> > >>> (each subdirectory contains all three months'
decks.)
> >> > >> > >> > >>>
> >> > >> > >> > >>> I was wondering if you could copy my directory,
> >> > >> > >> > >>>
> >> > /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test3
> >> > >> > >> > >>> and test:
> >> > >> > >> > >>> 1. It can generate multiple months instead of
just one
> >> month
> >> > >> and
> >> > >> > one
> >> > >> > >> > >>> model.
> >> > >> > >> > >>> 2. The wild card for models in the DECK_TEMPATE
for
> >> multiple
> >> > >> > models.
> >> > >> > >> > >>>
> >> > >> > >> > >>> George, I'm downloading NetCDF model data to
Hera:
> >> > >> > >> > >>>
> >> > >> >
> /scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071300_nc.tar
> >> > >> > >> > >>>
> >> > >> >
> /scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071306_nc.tar
> >> > >> > >> > >>>
> >> > >> > >> > >>> Please let me know after you downloaded the files
and I
> >> will
> >> > >> have
> >> > >> > to
> >> > >> > >> > >>> remove
> >> > >> > >> > >>> them due to tight disk space.
> >> > >> > >> > >>>
> >> > >> > >> > >>> Thanks,
> >> > >> > >> > >>> Guang Ping
> >> > >> > >> > >>>
> >> > >> > >> > >>> On Mon, Jul 19, 2021 at 5:01 PM John Halley
Gotway via
> RT
> >> <
> >> > >> > >> > >>> met_help at ucar.edu>
> >> > >> > >> > >>> wrote:
> >> > >> > >> > >>>
> >> > >> > >> > >>> > Hi Guang Ping,
> >> > >> > >> > >>> >
> >> > >> > >> > >>> > The command that you ran here:
> >> > >> > >> > >>> >
> >> > >> > >> > >>> >
> >> > >> > >> > >>> >
> >> > >> > >> > >>>
> >> > >> > >> >
> >> > >> > >>
> >> > >> >
> >> > >>
> >> >
> >>
>
/home/George.Mccabe/scratch1_George.Mccabe/METplus.feature_926/ush/run_metplus.py
> >> > >> > >> > >>> > -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c
> >> > Plotter_8.conf
> >> > >> > >> > >>> >
> >> > >> > >> > >>> > only runs TCMPRPlotter. This is because
PROCESS_LIST
> is
> >> > >> defined
> >> > >> > in
> >> > >> > >> > both
> >> > >> > >> > >>> > Plotter_fcstGFS_obsGFS_RPlotting_8.conf and
> >> Plotter_8.conf.
> >> > >> > Since
> >> > >> > >> > >>> > Plotter_8.conf comes last, then the value for
> >> PROCESS_LIST
> >> > >> from
> >> > >> > >> that
> >> > >> > >> > >>> file
> >> > >> > >> > >>> > is used, which only has TCMPRPlotter listed.
The
> >> plotter is
> >> > >> > >> reading
> >> > >> > >> > >>> all of
> >> > >> > >> > >>> > the available files generated from TCPairs and
since
> >> that
> >> > >> > process
> >> > >> > >> is
> >> > >> > >> > >>> not
> >> > >> > >> > >>> > running again with the new time range, the data
that
> is
> >> > read
> >> > >> > only
> >> > >> > >> > >>> covers
> >> > >> > >> > >>> > the first month.
> >> > >> > >> > >>> >
> >> > >> > >> > >>> > We are planning on releasing METplus 4.1.0-
beta1 this
> >> week.
> >> > >> > >> However,
> >> > >> > >> > >>> if I
> >> > >> > >> > >>> > understand what you are trying to do here, it
> shouldn't
> >> > >> require
> >> > >> > >> the
> >> > >> > >> > >>> changes
> >> > >> > >> > >>> > described in Molly's issue. It looks like you
want to
> >> run
> >> > >> > TCPairs
> >> > >> > >> > once
> >> > >> > >> > >>> and
> >> > >> > >> > >>> > filter the data based on the init and valid
times.
> While
> >> > this
> >> > >> > >> > >>> configuration
> >> > >> > >> > >>> > should achieve what you are trying to do, the
> following
> >> > >> > >> configuration
> >> > >> > >> > >>> is
> >> > >> > >> > >>> > more explicit to what you are actually running:
> >> > >> > >> > >>> >
> >> > >> > >> > >>> > # run once using 20200101 to substitute the
init time
> in
> >> > your
> >> > >> > >> string
> >> > >> > >> > >>> > substitution, i.e. TC_PAIRS_ADECK_TEMPLATE =
> >> > >> > >> > >>> >
> >> > >> > >>
> >>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> >> > >> > >> > >>> > INIT_BEG = 20200101
> >> > >> > >> > >>> > INIT_END = 20200101
> >> > >> > >> > >>> >
> >> > >> > >> > >>> > # filter the init times of the lines read from
the
> input
> >> > >> files
> >> > >> > in
> >> > >> > >> the
> >> > >> > >> > >>> > TCPairs configuration file
> >> > >> > >> > >>> > TC_PAIRS_INIT_BEG = 20200101
> >> > >> > >> > >>> > TC_PAIRS_INIT_END = 20200331
> >> > >> > >> > >>> >
> >> > >> > >> > >>> > # filter the valid times of the lines read from
the
> >> input
> >> > >> files
> >> > >> > in
> >> > >> > >> > the
> >> > >> > >> > >>> > TCPairs configuration file
> >> > >> > >> > >>> > TC_PAIRS_VALID_BEG = 20200101
> >> > >> > >> > >>> > TC_PAIRS_VALID_END = 20200331
> >> > >> > >> > >>> >
> >> > >> > >> > >>> > If any of TC_PAIRS_[INIT/VALID]_[BEG/END] are
not set,
> >> then
> >> > >> the
> >> > >> > >> > >>> > corresponding [INIT/VALID]_[BEG/END] values are
used.
> >> This
> >> > is
> >> > >> > >> done to
> >> > >> > >> > >>> > preserve backwards compatibility for old
METplus
> >> > >> configurations.
> >> > >> > >> The
> >> > >> > >> > >>> above
> >> > >> > >> > >>> > settings are more explicit as to what is
actually be
> >> > >> configured.
> >> > >> > >> > >>> >
> >> > >> > >> > >>> > You may want to consider either clearing your
output
> >> > >> directory
> >> > >> > or
> >> > >> > >> > >>> creating
> >> > >> > >> > >>> > a new output directory for a new run
(especially if
> the
> >> > time
> >> > >> > range
> >> > >> > >> > you
> >> > >> > >> > >>> are
> >> > >> > >> > >>> > using has changed) to ensure that the files you
are
> >> > generated
> >> > >> > are
> >> > >> > >> > from
> >> > >> > >> > >>> that
> >> > >> > >> > >>> > run and not from a previous run. Doing this
would have
> >> > >> alerted
> >> > >> > you
> >> > >> > >> > that
> >> > >> > >> > >>> > TCPairs was not run in this case.
> >> > >> > >> > >>> >
> >> > >> > >> > >>> > Let me know if you have any other issues
getting this
> >> > >> running.
> >> > >> > >> > >>> >
> >> > >> > >> > >>> > FYI:
> >> > >> > >> > >>> > We switched from providing support through this
email
> >> > >> address to
> >> > >> > >> > >>> providing
> >> > >> > >> > >>> > support through GitHub Discussions. In the
future, you
> >> will
> >> > >> need
> >> > >> > >> to
> >> > >> > >> > >>> create
> >> > >> > >> > >>> > a free GitHub account if you don’t have one
already
> and
> >> > post
> >> > >> > your
> >> > >> > >> > >>> questions
> >> > >> > >> > >>> > to the METplus Components Discussion Forum:
> >> > >> > >> > >>> > https://github.com/dtcenter/METplus/discussions
> >> > >> > >> > >>> > To post a new question, click on the green
button in
> the
> >> > >> upper
> >> > >> > >> right
> >> > >> > >> > >>> corner
> >> > >> > >> > >>> > of the page labeled "New discussion" and select
> >> "Incoming"
> >> > as
> >> > >> > the
> >> > >> > >> > >>> category.
> >> > >> > >> > >>> >
> >> > >> > >> > >>> > Thanks,
> >> > >> > >> > >>> > George
> >> > >> > >> > >>> >
> >> > >> > >> > >>> >
> >> > >> > >> > >>>
> >> > >> > >> > >>>
> >> > >> > >> > >>
> >> > >> > >> > >> --
> >> > >> > >> > >> George McCabe - Software Engineer III
> >> > >> > >> > >> National Center for Atmospheric Research
> >> > >> > >> > >> Research Applications Laboratory
> >> > >> > >> > >> 303-497-2768
> >> > >> > >> > >> ---
> >> > >> > >> > >> My working day may not be your working day. Please
do not
> >> feel
> >> > >> > >> obliged
> >> > >> > >> > to
> >> > >> > >> > >> reply to this email outside of your normal working
hours.
> >> > >> > >> > >>
> >> > >> > >> > >
> >> > >> > >> > >
> >> > >> > >> > > --
> >> > >> > >> > > George McCabe - Software Engineer III
> >> > >> > >> > > National Center for Atmospheric Research
> >> > >> > >> > > Research Applications Laboratory
> >> > >> > >> > > 303-497-2768
> >> > >> > >> > > ---
> >> > >> > >> > > My working day may not be your working day. Please
do not
> >> feel
> >> > >> > >> obliged to
> >> > >> > >> > > reply to this email outside of your normal working
hours.
> >> > >> > >> > >
> >> > >> > >> >
> >> > >> > >> >
> >> > >> > >> > --
> >> > >> > >> > George McCabe - Software Engineer III
> >> > >> > >> > National Center for Atmospheric Research
> >> > >> > >> > Research Applications Laboratory
> >> > >> > >> > 303-497-2768
> >> > >> > >> > ---
> >> > >> > >> > My working day may not be your working day. Please do
not
> feel
> >> > >> obliged
> >> > >> > >> to
> >> > >> > >> > reply to this email outside of your normal working
hours.
> >> > >> > >> >
> >> > >> > >> >
> >> > >> > >>
> >> > >> > >>
> >> > >> > >
> >> > >> > > --
> >> > >> > > George McCabe - Software Engineer III
> >> > >> > > National Center for Atmospheric Research
> >> > >> > > Research Applications Laboratory
> >> > >> > > 303-497-2768
> >> > >> > > ---
> >> > >> > > My working day may not be your working day. Please do
not feel
> >> > >> obliged to
> >> > >> > > reply to this email outside of your normal working
hours.
> >> > >> > >
> >> > >> >
> >> > >> >
> >> > >> > --
> >> > >> > George McCabe - Software Engineer III
> >> > >> > National Center for Atmospheric Research
> >> > >> > Research Applications Laboratory
> >> > >> > 303-497-2768
> >> > >> > ---
> >> > >> > My working day may not be your working day. Please do not
feel
> >> obliged
> >> > >> to
> >> > >> > reply to this email outside of your normal working hours.
> >> > >> >
> >> > >> >
> >> > >>
> >> > >>
> >> > >
> >> > > --
> >> > > George McCabe - Software Engineer III
> >> > > National Center for Atmospheric Research
> >> > > Research Applications Laboratory
> >> > > 303-497-2768
> >> > > ---
> >> > > My working day may not be your working day. Please do not
feel
> >> obliged to
> >> > > reply to this email outside of your normal working hours.
> >> > >
> >> >
> >> >
> >> > --
> >> > George McCabe - Software Engineer III
> >> > National Center for Atmospheric Research
> >> > Research Applications Laboratory
> >> > 303-497-2768
> >> > ---
> >> > My working day may not be your working day. Please do not feel
obliged
> >> to
> >> > reply to this email outside of your normal working hours.
> >> >
> >> >
> >>
> >>
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
------------------------------------------------
Subject: tcpairs
From: Guang Lou - NOAA Affiliate
Time: Tue Aug 17 09:48:38 2021
Hi George,
One more question: if the tracks have converted to .tcst format, I
just
wanted to do the plotting, what parameters I need to set?
I tried these:
PROCESS_LIST = TCMPRPlotter
#PROCESS_LIST = TCPairs, TCMPRPlotter
It gives error messages.
I tried a few other changes and none seemed to work.
The reason I wanted to do this is because it will save time once the
format
is converted and saved. I can just plot out whatever length (yearly,
seasonally and monthly) at a much faster speed, instead of everytime I
have
to convert the format (it takes a long time.)
Thanks again,
Guang Ping
On Tue, Aug 17, 2021 at 8:27 AM Guang Lou - NOAA Affiliate <
guang.ping.lou at noaa.gov> wrote:
> Hi George,
>
> Thank you so much for figuring out the plotting scripts! Following
your
> suggestion, I was able to generate plots that put all model errors
on the
> same frame. I've generated the 3 months' average values and I will
do a
> longer term average.
>
> One question for you, is there a place that I can turn off the error
range
> bar (the sigma range from low to high)? When we have many models,
these
> bars would crowd out the error lines.
>
> Your help is much appreciated!
> Guang Ping
>
> On Mon, Aug 16, 2021 at 1:43 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
>> Hi Guang Ping,
>>
>> I was able to generate images with all of the models on Hera. There
were
>> modifications to the METplus wrapper and R script that were needed
to get
>> it to work properly on Hera. The config file I used is here:
>>
>> /home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8_test2.conf
>>
>> I added the following to properly pass in the data directory to the
>> script:
>>
>> TCMPR_PLOTTER_READ_ALL_FILES = True
>>
>> I had to change the value for MET_INSTALL_DIR to another location
that has
>> the change to the R script:
>>
>> MET_INSTALL_DIR = /scratch1/BMC/dtc/George.Mccabe/met-tmp
>>
>> I called run_metplus.py from my feature branch directory to get the
>> changes
>> needed:
>>
>>
>>
/scratch1/BMC/dtc/George.Mccabe/METplus.feature_1057_tcmpr_dir_option/ush/run_metplus.py
>>
>> I am rerunning the use case now. When it finishes the images will
be
>> generated here:
>>
>> /scratch1/BMC/dtc/George.Mccabe/tcmpr-test5/tcmpr_plots
>>
>> Please take a look and let me know if the images look good to you.
>>
>> Thanks,
>> George
>>
>> On Tue, Aug 3, 2021 at 7:59 AM George McCabe <mccabe at ucar.edu>
wrote:
>>
>> > Hi Guang Ping,
>> >
>> > I see. Thanks for clarifying. The good news is that John HG was
able to
>> > generate a plot of all of the models. Attached is an image he
generated.
>> > This is the command he ran to generate the images:
>> >
>> > Rscript met/scripts/Rscripts/plot_tcmpr.R -lookin `find ./ -name
>> "*.tcst"`
>> > -no_ee -plot BOXPLOT,MEAN
>> >
>> > There are 2 things John mentioned to note here:
>> >
>> > * That "find" command found 2705 input files and passed them all
in on
>> the
>> > command line. That's a long command line and might cause shell
problems
>> on
>> > some system.
>> > * The "-no_ee" option DISABLES event equalization, which is
needed since
>> > these cases are not the same. With event equalization turned on,
all the
>> > cases will be tossed out.
>> >
>> > The -no_ee argument can be set by setting this in your METplus
>> > configuration file:
>> >
>> > TCMPR_PLOTTER_NO_EE = yes
>> >
>> > Howeer, his first comment aligns with the error I was seeing when
I
>> tried
>> > to run where the shell couldn't handle the large number of files
that
>> were
>> > passed to the command. The wrapper finds all of the files in the
input
>> > directory that end with .tcst and pass in all of the files to the
>> Rscript
>> > command, which is essentially the same behavior as the inserting
the
>> find
>> > command in John's example. Hera is down today so I am unable to
test,
>> but
>> > it may be the case that Hera cannot handle this command. The
>> plot_tcmpr.R
>> > script fails on this line (380), which attempts to expand any
wildcards
>> > with an ls command:
>> >
>> > file_list = system(paste("ls -1", paste(file_list, collapse="
")),
>> > intern=TRUE);
>> >
>> > The list of files is passed from the R script into the tc_stat
MET tool
>> > for filtering and any wildcards are already expanded by the
wrapper. I
>> > suspect that if I add an option to this script to bypass the
wildcard
>> > expansion, then the call to tc_stat will work. I can make the
necessary
>> > changes locally and test it on Hera when it comes back online. If
>> tc_stat
>> > cannot handle the long command on Hera, then modifications to the
>> tc_stat
>> > tool to support reading in a text file that contains a list of
files and
>> > modifications to the TCMPRPlotter wrapper to generate a file list
text
>> file
>> > may fix the issue.
>> >
>> > I will follow up with more information after some testing. It is
>> > reassuring to know that the images you are trying to create can
be
>> > generated.
>> >
>> > Thanks,
>> > George
>> >
>> > On Thu, Jul 29, 2021 at 7:55 AM Guang Lou - NOAA Affiliate via RT
<
>> > met_help at ucar.edu> wrote:
>> >
>> >> Hi George,
>> >>
>> >> Thanks for the update. The purpose of this plot is to compare
error
>> stats
>> >> between different models. So we need to plot all model errors in
one
>> frame
>> >> instead of each model having its own frame. I did not state
clearly in
>> my
>> >> last email.
>> >>
>> >> Thanks,
>> >> Guang Ping
>> >>
>> >> On Wed, Jul 28, 2021 at 4:32 PM George McCabe via RT <
>> met_help at ucar.edu>
>> >> wrote:
>> >>
>> >> > Hi Guang Ping,
>> >> >
>> >> > I updated the config file to process each model separately.
Here is
>> the
>> >> > config file on hera:
>> >> >
>> >> > /home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8_test.conf
>> >> >
>> >> > The output from my latest run can be found in here:
>> >> >
>> >> > /scratch1/BMC/dtc/George.Mccabe/tcmpr-test4/tcmpr_plots
>> >> >
>> >> > There is a sub-directory for each model. Please take a look at
the
>> >> output
>> >> > files to see if they look OK. The config file could be cleaned
up a
>> >> little
>> >> > bit if I add support for TCMPR_PLOTTER_CUSTOM_LOOP_LIST or
support
>> for
>> >> > reading the instance name from the TCMPRPlotter instances. I
will
>> make a
>> >> > GitHub issue to implement those changes as they are relatively
easy
>> >> > additions and would make the wrapper consistent with the other
>> wrappers.
>> >> >
>> >> > Thanks,
>> >> > George
>> >> >
>> >> > On Wed, Jul 28, 2021 at 10:02 AM George McCabe
<mccabe at ucar.edu>
>> wrote:
>> >> >
>> >> > > Hi Guang Ping,
>> >> > >
>> >> > > OK, that is good to know. I have a few meetings this morning
but I
>> >> will
>> >> > > try to update your configuration file to set this up.
>> >> > >
>> >> > > - George
>> >> > >
>> >> > > On Wed, Jul 28, 2021 at 9:46 AM Guang Lou - NOAA Affiliate
via RT <
>> >> > > met_help at ucar.edu> wrote:
>> >> > >
>> >> > >> Hi George,
>> >> > >>
>> >> > >> The verification should be against self analysis. That is
GFS
>> adeck
>> >> > >> against
>> >> > >> GFS bdeck, CMC adeck against CMC bdeck, and so on and so
forth.
>> >> > >>
>> >> > >> Thanks,
>> >> > >> Guang Ping
>> >> > >>
>> >> > >> On Tue, Jul 27, 2021 at 2:33 PM George McCabe via RT <
>> >> met_help at ucar.edu
>> >> > >
>> >> > >> wrote:
>> >> > >>
>> >> > >> > Hi Guang Ping,
>> >> > >> >
>> >> > >> > I looked at this use case more today. I modified the use
case to
>> >> read
>> >> > a
>> >> > >> > single month into TCMPRPlotter because it cannot handle
all of
>> the
>> >> > >> files at
>> >> > >> > once. It looks like you need to set the -bmodel in
>> >> > TCMPR_PLOTTER_FILTER
>> >> > >> to
>> >> > >> > specify the Bdeck to compare the models to. However, each
of the
>> >> > output
>> >> > >> > files from tc_pairs use the same model for adeck and
bdeck, so
>> the
>> >> > >> AMODEL
>> >> > >> > and BMODEL values in those files are the same, i.e. both
GFSO.
>> If
>> >> you
>> >> > >> want
>> >> > >> > to compare all of the models to a single bdeck, you will
have to
>> >> > >> configure
>> >> > >> > TCPairs wrapper to use the same bdeck for each model with
>> >> > >> > TC_PAIRS_BDECK_TEMPLATE. I tried running using best for
the
>> bdeck
>> >> and
>> >> > it
>> >> > >> > produced no pairs. I am rerunning now using GFSO.
>> >> > >> >
>> >> > >> > - George
>> >> > >> >
>> >> > >> > On Mon, Jul 26, 2021 at 5:57 PM George McCabe
<mccabe at ucar.edu>
>> >> > wrote:
>> >> > >> >
>> >> > >> > > Hi Guang Ping,
>> >> > >> > >
>> >> > >> > > I copied your config file and made some changes. The
new file
>> >> can be
>> >> > >> > found
>> >> > >> > > on hera here:
>> >> > >> > >
>> >> > >> > > /home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8.conf
>> >> > >> > >
>> >> > >> > > Using METplus 4.1.0-beta1, you can set
TC_PAIRS_RUN_ONCE =
>> True
>> >> to
>> >> > >> loop
>> >> > >> > > over the init times even with LOOP_ORDER = processes.
>> >> > >> > >
>> >> > >> > > I noticed that you have TC_PAIRS_REFORMAT_DECK = yes,
which
>> isn't
>> >> > >> > > necessary because tc_pairs can read the input files
without
>> >> > >> reformatting.
>> >> > >> > > That can be unset or set to False/no.
>> >> > >> > >
>> >> > >> > > TC_PAIRS_REFORMAT_DECK = no
>> >> > >> > >
>> >> > >> > > I misread your email about trying to use the wildcard
for the
>> >> model.
>> >> > >> This
>> >> > >> > > won't work as you expect because the output template
will not
>> >> know
>> >> > >> how to
>> >> > >> > > name the output file with the correct model. Each model
will
>> >> > overwrite
>> >> > >> > the
>> >> > >> > > output files. To get around this you can use
>> >> > >> TC_PAIRS_CUSTOM_LOOP_LIST.
>> >> > >> > > This will run the TCPairs wrapper once for each item in
the
>> >> custom
>> >> > >> loop
>> >> > >> > > list. Here you can specify the list of models
(lowercase) and
>> >> > >> reference
>> >> > >> > the
>> >> > >> > > {custom} template tag to find the files.
>> >> > >> > >
>> >> > >> > > TC_PAIRS_CUSTOM_LOOP_LIST = best, cmc, emx, gfso, nam,
ngx,
>> ukx
>> >> > >> > >
>> >> > >> > > TC_PAIRS_ADECK_TEMPLATE =
>> >> > >> > >
>> >> > >> >
>> >> > >>
>> >> >
>> >>
>>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
>> >> > >> > > TC_PAIRS_BDECK_TEMPLATE =
>> >> > >> > >
>> >> > >> >
>> >> > >>
>> >> >
>> >>
>>
{date?fmt=%Y%m}/b{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
>> >> > >> > > TC_PAIRS_OUTPUT_TEMPLATE =
>> >> > >> > >
>> >> > >> >
>> >> > >>
>> >> >
>> >>
>>
{date?fmt=%Y%m}/{basin?fmt=%s}q{date?fmt=%Y%m%d%H}.{custom}.{cyclone?fmt=%s}
>> >> > >> > >
>> >> > >> > > I removed the MODEL values (you had it twice in your
config
>> file)
>> >> > >> because
>> >> > >> > > we don't need to filter by the model if we are reading
all of
>> >> them
>> >> > in
>> >> > >> > > individually.
>> >> > >> > >
>> >> > >> > > Using this config file I was able to run TCPairs for
each
>> model
>> >> > >> > processing
>> >> > >> > > each for all 3 months and creating output files from
tc_pairs
>> >> here:
>> >> > >> > >
>> >> > >> > > /scratch1/BMC/dtc/George.Mccabe/tcmpr-test2/tc_pairs
>> >> > >> > >
>> >> > >> > > The error in your log file from TCMPRPlotter mentions
that
>> you do
>> >> > not
>> >> > >> > have
>> >> > >> > > -bmodel set in the filter options. When I ran it, I got
an
>> error
>> >> > with
>> >> > >> a
>> >> > >> > > command that appears to list the files which leads me
to think
>> >> that
>> >> > >> there
>> >> > >> > > are too many files in the tc_pairs directory to process
at
>> once.
>> >> I
>> >> > >> need
>> >> > >> > to
>> >> > >> > > stop working for today, but I will resume debugging
this
>> tomorrow
>> >> > and
>> >> > >> see
>> >> > >> > > what I can find. It may be the case that the plotter
cannot
>> >> handle
>> >> > >> all 7
>> >> > >> > > models over a 3 day period all at once. If this is the
case,
>> what
>> >> > >> would
>> >> > >> > be
>> >> > >> > > useful to display as a subset of this data?
>> >> > >> > >
>> >> > >> > > Thanks,
>> >> > >> > > George
>> >> > >> > >
>> >> > >> > >
>> >> > >> > >
>> >> > >> > > On Fri, Jul 23, 2021 at 11:46 AM Guang Lou - NOAA
Affiliate
>> via
>> >> RT <
>> >> > >> > > met_help at ucar.edu> wrote:
>> >> > >> > >
>> >> > >> > >>
>> >> > >> > >> <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100602
>> >> >
>> >> > >> > >>
>> >> > >> > >> Hi George,
>> >> > >> > >>
>> >> > >> > >> Thanks for letting me know you have downloaded the
files.
>> I've
>> >> > >> removed
>> >> > >> > the
>> >> > >> > >> files to save disk space.
>> >> > >> > >> Per your suggestion, I was able to generate
consecutive
>> months
>> >> of
>> >> > >> paired
>> >> > >> > >> tracks by changing a few lines
>> >> > >> > >> and parameters in the configure file. I've also loaded
the
>> >> module
>> >> > >> > >> "metplus/4.1.0-beta1".
>> >> > >> > >>
>> >> > >> > >> This was done by:
>> >> > >> > >>
>> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_pairs
>> >> > >> > >> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c
>> >> > >> > >> usecase_pairing_8.conf
>> >> > >> > >>
>> >> > >> > >> However, when I tried to generate plots from the just
>> generated
>> >> > >> pairs,
>> >> > >> > it
>> >> > >> > >> ran into error:
>> >> > >> > >> in
>> >> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_pairs
>> >> > >> > >> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c
>> >> usecase_8.conf
>> >> > >> > >>
>> >> > >> > >> where I changed the following to:
>> >> > >> > >> LOOP_ORDER = processes
>> >> > >> > >> PROCESS_LIST = TCMPRPlotter
>> >> > >> > >> error messages are:
>> >> > >> > >> 07/23 17:38:51.477 metplus INFO: Check the logfile for
more
>> >> > >> information
>> >> > >> > on
>> >> > >> > >> why it failed:
>> >> > >> > >>
>> >> > >> > >>
>> >> > >> >
>> >> > >>
>> >> >
>> >>
>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
>> >> > >> > >> 07/23 17:38:51.478 metplus INFO: Plotting complete
>> >> > >> > >> 07/23 17:38:51.481 metplus.TCMPRPlotter
(met_util.py:171)
>> ERROR:
>> >> > >> > >> TCMPRPlotter had 9 errors.
>> >> > >> > >> 07/23 17:38:51.481 metplus INFO:
>> >> > >> > >>
>> >> > >> > >>
>> >> > >> >
>> >> > >>
>> >> >
>> >>
>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/metplus_final.conf:
>> >> > >> > >> write metplus.conf here
>> >> > >> > >> 07/23 17:38:51.486 metplus (met_util.py:208) ERROR:
METplus
>> has
>> >> > >> finished
>> >> > >> > >> running but had 9 errors.
>> >> > >> > >> 07/23 17:38:51.486 metplus INFO: Check the log file
for more
>> >> > >> > information:
>> >> > >> > >>
>> >> > >> > >>
>> >> > >> >
>> >> > >>
>> >> >
>> >>
>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
>> >> > >> > >>
-----------------------------------------------------------
>> >> > >> > >> The wild card in the place of
>> >> > >> > >> TC_PAIRS_ADECK_TEMPLATE =
>> >> > >> > >>
>> >> >
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.nam.{cyclone?fmt=%s}
>> >> > >> > >> to
>> >> > >> > >> TC_PAIRS_ADECK_TEMPLATE =
>> >> > >> > >>
>> >>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
>> >> > >> > >> still does not work.
>> >> > >> > >>
>> >> > >> > >> Thanks,
>> >> > >> > >> Guang Ping
>> >> > >> > >>
>> >> > >> > >> On Wed, Jul 21, 2021 at 2:41 PM George McCabe via RT <
>> >> > >> met_help at ucar.edu
>> >> > >> > >
>> >> > >> > >> wrote:
>> >> > >> > >>
>> >> > >> > >> > Hi Guang Ping,
>> >> > >> > >> >
>> >> > >> > >> > I have finished downloading the GFS NetCDF files
from Hera
>> so
>> >> you
>> >> > >> can
>> >> > >> > >> > remove them. Thanks again for providing that data.
>> >> > >> > >> >
>> >> > >> > >> > - George
>> >> > >> > >> >
>> >> > >> > >> > On Tue, Jul 20, 2021 at 3:35 PM George McCabe <
>> >> mccabe at ucar.edu>
>> >> > >> > wrote:
>> >> > >> > >> >
>> >> > >> > >> > > Hi Guang Ping,
>> >> > >> > >> > >
>> >> > >> > >> > > I mistyped in my previous email. The variable to
set to
>> >> "times"
>> >> > >> is
>> >> > >> > >> > > LOOP_ORDER not LOOP_BY.
>> >> > >> > >> > >
>> >> > >> > >> > > LOOP_ORDER = times
>> >> > >> > >> > > INIT_INCREMENT = 1m
>> >> > >> > >> > >
>> >> > >> > >> > > Sorry for the confusion.
>> >> > >> > >> > >
>> >> > >> > >> > > - George
>> >> > >> > >> > >
>> >> > >> > >> > > On Tue, Jul 20, 2021 at 3:21 PM George McCabe <
>> >> mccabe at ucar.edu
>> >> > >
>> >> > >> > >> wrote:
>> >> > >> > >> > >
>> >> > >> > >> > >> Hi Guang Ping,
>> >> > >> > >> > >>
>> >> > >> > >> > >> Thank you for providing the NetCDF files. I am
currently
>> >> > >> > downloading
>> >> > >> > >> one
>> >> > >> > >> > >> of the files but it is going very slowly.
>> >> > >> > >> > >>
>> >> > >> > >> > >> In regards to your use case, the logic for
TCPairs
>> wrapper
>> >> is
>> >> > a
>> >> > >> > >> little
>> >> > >> > >> > >> strange. It behaves differently based on the
value of
>> >> LOOP_BY.
>> >> > >> If
>> >> > >> > >> > LOOP_BY =
>> >> > >> > >> > >> processes, then it will run once using the
INIT_BEG
>> time as
>> >> > the
>> >> > >> run
>> >> > >> > >> > time to
>> >> > >> > >> > >> find input files. If LOOP_BY = times, then it
will loop
>> >> over
>> >> > >> each
>> >> > >> > run
>> >> > >> > >> > time
>> >> > >> > >> > >> specified between INIT_BEG and INIT_END by
>> INIT_INCREMENT
>> >> and
>> >> > >> find
>> >> > >> > >> > files to
>> >> > >> > >> > >> process for each run time.
>> >> > >> > >> > >>
>> >> > >> > >> > >> If you change your configuration file to include
>> LOOP_BY =
>> >> > >> times,
>> >> > >> > >> then
>> >> > >> > >> > it
>> >> > >> > >> > >> should look over each run time and process for
each. It
>> >> looks
>> >> > >> like
>> >> > >> > >> your
>> >> > >> > >> > >> INIT_INCREMENT is set to 21600 or 6 hours, so it
will
>> run
>> >> > every
>> >> > >> 6
>> >> > >> > >> hours
>> >> > >> > >> > >> from 20200201 to 2020331. If you want to run once
per
>> >> month,
>> >> > you
>> >> > >> > will
>> >> > >> > >> > need
>> >> > >> > >> > >> to set INIT_INCREMENT = 1m so that it will
increment by
>> 1
>> >> > month
>> >> > >> for
>> >> > >> > >> each
>> >> > >> > >> > >> iteration.
>> >> > >> > >> > >>
>> >> > >> > >> > >> The caveat is that TCMPRPlotter will only run if
>> LOOP_BY =
>> >> > >> > processes
>> >> > >> > >> and
>> >> > >> > >> > >> will produce an error if LOOP_BY = times. This is
>> because
>> >> we
>> >> > >> > >> typically
>> >> > >> > >> > want
>> >> > >> > >> > >> to process all of the data that is available for
all run
>> >> > times,
>> >> > >> > then
>> >> > >> > >> run
>> >> > >> > >> > >> TCMPRPlotter so it can read all of the data. For
now, to
>> >> > handle
>> >> > >> > this
>> >> > >> > >> use
>> >> > >> > >> > >> case you will have to call run_metplus.py 2
separate
>> times,
>> >> > once
>> >> > >> > >> calling
>> >> > >> > >> > >> TCPairs with LOOP_BY = times and once calling
>> TCMPRPlotter
>> >> > with
>> >> > >> > >> LOOP_BY
>> >> > >> > >> > =
>> >> > >> > >> > >> processes.
>> >> > >> > >> > >>
>> >> > >> > >> > >> I am going to add a configuration option to
TCPairs
>> >> wrapper so
>> >> > >> that
>> >> > >> > >> you
>> >> > >> > >> > >> can loop over the run times when LOOP_BY =
processes is
>> >> set if
>> >> > >> > >> desired.
>> >> > >> > >> > I
>> >> > >> > >> > >> will try to get this change into METplus 4.1.0-
beta1 so
>> you
>> >> > can
>> >> > >> > test
>> >> > >> > >> the
>> >> > >> > >> > >> change and make sure it works as expected. I can
let you
>> >> know
>> >> > >> when
>> >> > >> > >> that
>> >> > >> > >> > is
>> >> > >> > >> > >> available.
>> >> > >> > >> > >>
>> >> > >> > >> > >> Thanks,
>> >> > >> > >> > >> George
>> >> > >> > >> > >>
>> >> > >> > >> > >> On Tue, Jul 20, 2021 at 1:27 PM Guang Lou - NOAA
>> Affiliate
>> >> via
>> >> > >> RT <
>> >> > >> > >> > >> met_help at ucar.edu> wrote:
>> >> > >> > >> > >>
>> >> > >> > >> > >>> Hi John,
>> >> > >> > >> > >>>
>> >> > >> > >> > >>> Thanks for forwarding George's email answer to
me. I
>> >> created
>> >> > a
>> >> > >> > test
>> >> > >> > >> > case
>> >> > >> > >> > >>> that does the tcpairs only on Hera:
>> >> > >> > >> > >>>
>> >> > /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test3
>> >> > >> > >> > >>> Command line:
>> >> > >> > >> > >>> run_metplus.py -c
>> Plotter_fcstGFS_obsGFS_RPlotting_8.conf
>> >> -c
>> >> > >> > >> > >>> usecase_8.conf
>> >> > >> > >> > >>> where PROCESS_LIST = TCPairs
>> >> > >> > >> > >>>
>> >> > >> > >> > >>> It generated the month of 202002, that is
because I
>> >> changed
>> >> > >> > >> > >>> INIT_BEG = 20200101
>> >> > >> > >> > >>> INIT_END = 20200331
>> >> > >> > >> > >>> to
>> >> > >> > >> > >>> INIT_BEG = 20200201
>> >> > >> > >> > >>> INIT_END = 20200331
>> >> > >> > >> > >>>
>> >> > >> > >> > >>> If I use "INIT_BEG = 20200101,INIT_END =
20200331" it
>> >> will
>> >> > >> > generate
>> >> > >> > >> > >>> 20200101 only.
>> >> > >> > >> > >>>
>> >> > >> > >> > >>> The wild card does not work either for the
model's
>> place:
>> >> > >> > >> > >>> TC_PAIRS_ADECK_TEMPLATE =
>> >> > >> > >> > >>>
>> >> > >> >
>> {date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
>> >> > >> > >> > >>> I populated adeck and bdeck on
>> >> > >> > >> > >>>
>> >> > >> > >> > >>>
>> >> > >> > >> >
>> >> > >> > >>
>> >> > >> >
>> >> > >>
>> >> >
>> >>
>> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/METplus/METplus-
4.0_sample_data/model_applications/medium_range/tracks_all
>> >> > >> > >> > >>> (each subdirectory contains all three months'
decks.)
>> >> > >> > >> > >>>
>> >> > >> > >> > >>> I was wondering if you could copy my directory,
>> >> > >> > >> > >>>
>> >> > /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_test3
>> >> > >> > >> > >>> and test:
>> >> > >> > >> > >>> 1. It can generate multiple months instead of
just one
>> >> month
>> >> > >> and
>> >> > >> > one
>> >> > >> > >> > >>> model.
>> >> > >> > >> > >>> 2. The wild card for models in the DECK_TEMPATE
for
>> >> multiple
>> >> > >> > models.
>> >> > >> > >> > >>>
>> >> > >> > >> > >>> George, I'm downloading NetCDF model data to
Hera:
>> >> > >> > >> > >>>
>> >> > >> >
>> /scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071300_nc.tar
>> >> > >> > >> > >>>
>> >> > >> >
>> /scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071306_nc.tar
>> >> > >> > >> > >>>
>> >> > >> > >> > >>> Please let me know after you downloaded the
files and I
>> >> will
>> >> > >> have
>> >> > >> > to
>> >> > >> > >> > >>> remove
>> >> > >> > >> > >>> them due to tight disk space.
>> >> > >> > >> > >>>
>> >> > >> > >> > >>> Thanks,
>> >> > >> > >> > >>> Guang Ping
>> >> > >> > >> > >>>
>> >> > >> > >> > >>> On Mon, Jul 19, 2021 at 5:01 PM John Halley
Gotway via
>> RT
>> >> <
>> >> > >> > >> > >>> met_help at ucar.edu>
>> >> > >> > >> > >>> wrote:
>> >> > >> > >> > >>>
>> >> > >> > >> > >>> > Hi Guang Ping,
>> >> > >> > >> > >>> >
>> >> > >> > >> > >>> > The command that you ran here:
>> >> > >> > >> > >>> >
>> >> > >> > >> > >>> >
>> >> > >> > >> > >>> >
>> >> > >> > >> > >>>
>> >> > >> > >> >
>> >> > >> > >>
>> >> > >> >
>> >> > >>
>> >> >
>> >>
>>
/home/George.Mccabe/scratch1_George.Mccabe/METplus.feature_926/ush/run_metplus.py
>> >> > >> > >> > >>> > -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c
>> >> > Plotter_8.conf
>> >> > >> > >> > >>> >
>> >> > >> > >> > >>> > only runs TCMPRPlotter. This is because
PROCESS_LIST
>> is
>> >> > >> defined
>> >> > >> > in
>> >> > >> > >> > both
>> >> > >> > >> > >>> > Plotter_fcstGFS_obsGFS_RPlotting_8.conf and
>> >> Plotter_8.conf.
>> >> > >> > Since
>> >> > >> > >> > >>> > Plotter_8.conf comes last, then the value for
>> >> PROCESS_LIST
>> >> > >> from
>> >> > >> > >> that
>> >> > >> > >> > >>> file
>> >> > >> > >> > >>> > is used, which only has TCMPRPlotter listed.
The
>> >> plotter is
>> >> > >> > >> reading
>> >> > >> > >> > >>> all of
>> >> > >> > >> > >>> > the available files generated from TCPairs and
since
>> >> that
>> >> > >> > process
>> >> > >> > >> is
>> >> > >> > >> > >>> not
>> >> > >> > >> > >>> > running again with the new time range, the
data that
>> is
>> >> > read
>> >> > >> > only
>> >> > >> > >> > >>> covers
>> >> > >> > >> > >>> > the first month.
>> >> > >> > >> > >>> >
>> >> > >> > >> > >>> > We are planning on releasing METplus 4.1.0-
beta1 this
>> >> week.
>> >> > >> > >> However,
>> >> > >> > >> > >>> if I
>> >> > >> > >> > >>> > understand what you are trying to do here, it
>> shouldn't
>> >> > >> require
>> >> > >> > >> the
>> >> > >> > >> > >>> changes
>> >> > >> > >> > >>> > described in Molly's issue. It looks like you
want to
>> >> run
>> >> > >> > TCPairs
>> >> > >> > >> > once
>> >> > >> > >> > >>> and
>> >> > >> > >> > >>> > filter the data based on the init and valid
times.
>> While
>> >> > this
>> >> > >> > >> > >>> configuration
>> >> > >> > >> > >>> > should achieve what you are trying to do, the
>> following
>> >> > >> > >> configuration
>> >> > >> > >> > >>> is
>> >> > >> > >> > >>> > more explicit to what you are actually
running:
>> >> > >> > >> > >>> >
>> >> > >> > >> > >>> > # run once using 20200101 to substitute the
init
>> time in
>> >> > your
>> >> > >> > >> string
>> >> > >> > >> > >>> > substitution, i.e. TC_PAIRS_ADECK_TEMPLATE =
>> >> > >> > >> > >>> >
>> >> > >> > >>
>> >>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
>> >> > >> > >> > >>> > INIT_BEG = 20200101
>> >> > >> > >> > >>> > INIT_END = 20200101
>> >> > >> > >> > >>> >
>> >> > >> > >> > >>> > # filter the init times of the lines read from
the
>> input
>> >> > >> files
>> >> > >> > in
>> >> > >> > >> the
>> >> > >> > >> > >>> > TCPairs configuration file
>> >> > >> > >> > >>> > TC_PAIRS_INIT_BEG = 20200101
>> >> > >> > >> > >>> > TC_PAIRS_INIT_END = 20200331
>> >> > >> > >> > >>> >
>> >> > >> > >> > >>> > # filter the valid times of the lines read
from the
>> >> input
>> >> > >> files
>> >> > >> > in
>> >> > >> > >> > the
>> >> > >> > >> > >>> > TCPairs configuration file
>> >> > >> > >> > >>> > TC_PAIRS_VALID_BEG = 20200101
>> >> > >> > >> > >>> > TC_PAIRS_VALID_END = 20200331
>> >> > >> > >> > >>> >
>> >> > >> > >> > >>> > If any of TC_PAIRS_[INIT/VALID]_[BEG/END] are
not
>> set,
>> >> then
>> >> > >> the
>> >> > >> > >> > >>> > corresponding [INIT/VALID]_[BEG/END] values
are used.
>> >> This
>> >> > is
>> >> > >> > >> done to
>> >> > >> > >> > >>> > preserve backwards compatibility for old
METplus
>> >> > >> configurations.
>> >> > >> > >> The
>> >> > >> > >> > >>> above
>> >> > >> > >> > >>> > settings are more explicit as to what is
actually be
>> >> > >> configured.
>> >> > >> > >> > >>> >
>> >> > >> > >> > >>> > You may want to consider either clearing your
output
>> >> > >> directory
>> >> > >> > or
>> >> > >> > >> > >>> creating
>> >> > >> > >> > >>> > a new output directory for a new run
(especially if
>> the
>> >> > time
>> >> > >> > range
>> >> > >> > >> > you
>> >> > >> > >> > >>> are
>> >> > >> > >> > >>> > using has changed) to ensure that the files
you are
>> >> > generated
>> >> > >> > are
>> >> > >> > >> > from
>> >> > >> > >> > >>> that
>> >> > >> > >> > >>> > run and not from a previous run. Doing this
would
>> have
>> >> > >> alerted
>> >> > >> > you
>> >> > >> > >> > that
>> >> > >> > >> > >>> > TCPairs was not run in this case.
>> >> > >> > >> > >>> >
>> >> > >> > >> > >>> > Let me know if you have any other issues
getting this
>> >> > >> running.
>> >> > >> > >> > >>> >
>> >> > >> > >> > >>> > FYI:
>> >> > >> > >> > >>> > We switched from providing support through
this email
>> >> > >> address to
>> >> > >> > >> > >>> providing
>> >> > >> > >> > >>> > support through GitHub Discussions. In the
future,
>> you
>> >> will
>> >> > >> need
>> >> > >> > >> to
>> >> > >> > >> > >>> create
>> >> > >> > >> > >>> > a free GitHub account if you don’t have one
already
>> and
>> >> > post
>> >> > >> > your
>> >> > >> > >> > >>> questions
>> >> > >> > >> > >>> > to the METplus Components Discussion Forum:
>> >> > >> > >> > >>> >
https://github.com/dtcenter/METplus/discussions
>> >> > >> > >> > >>> > To post a new question, click on the green
button in
>> the
>> >> > >> upper
>> >> > >> > >> right
>> >> > >> > >> > >>> corner
>> >> > >> > >> > >>> > of the page labeled "New discussion" and
select
>> >> "Incoming"
>> >> > as
>> >> > >> > the
>> >> > >> > >> > >>> category.
>> >> > >> > >> > >>> >
>> >> > >> > >> > >>> > Thanks,
>> >> > >> > >> > >>> > George
>> >> > >> > >> > >>> >
>> >> > >> > >> > >>> >
>> >> > >> > >> > >>>
>> >> > >> > >> > >>>
>> >> > >> > >> > >>
>> >> > >> > >> > >> --
>> >> > >> > >> > >> George McCabe - Software Engineer III
>> >> > >> > >> > >> National Center for Atmospheric Research
>> >> > >> > >> > >> Research Applications Laboratory
>> >> > >> > >> > >> 303-497-2768
>> >> > >> > >> > >> ---
>> >> > >> > >> > >> My working day may not be your working day.
Please do
>> not
>> >> feel
>> >> > >> > >> obliged
>> >> > >> > >> > to
>> >> > >> > >> > >> reply to this email outside of your normal
working
>> hours.
>> >> > >> > >> > >>
>> >> > >> > >> > >
>> >> > >> > >> > >
>> >> > >> > >> > > --
>> >> > >> > >> > > George McCabe - Software Engineer III
>> >> > >> > >> > > National Center for Atmospheric Research
>> >> > >> > >> > > Research Applications Laboratory
>> >> > >> > >> > > 303-497-2768
>> >> > >> > >> > > ---
>> >> > >> > >> > > My working day may not be your working day. Please
do not
>> >> feel
>> >> > >> > >> obliged to
>> >> > >> > >> > > reply to this email outside of your normal working
hours.
>> >> > >> > >> > >
>> >> > >> > >> >
>> >> > >> > >> >
>> >> > >> > >> > --
>> >> > >> > >> > George McCabe - Software Engineer III
>> >> > >> > >> > National Center for Atmospheric Research
>> >> > >> > >> > Research Applications Laboratory
>> >> > >> > >> > 303-497-2768
>> >> > >> > >> > ---
>> >> > >> > >> > My working day may not be your working day. Please
do not
>> feel
>> >> > >> obliged
>> >> > >> > >> to
>> >> > >> > >> > reply to this email outside of your normal working
hours.
>> >> > >> > >> >
>> >> > >> > >> >
>> >> > >> > >>
>> >> > >> > >>
>> >> > >> > >
>> >> > >> > > --
>> >> > >> > > George McCabe - Software Engineer III
>> >> > >> > > National Center for Atmospheric Research
>> >> > >> > > Research Applications Laboratory
>> >> > >> > > 303-497-2768
>> >> > >> > > ---
>> >> > >> > > My working day may not be your working day. Please do
not feel
>> >> > >> obliged to
>> >> > >> > > reply to this email outside of your normal working
hours.
>> >> > >> > >
>> >> > >> >
>> >> > >> >
>> >> > >> > --
>> >> > >> > George McCabe - Software Engineer III
>> >> > >> > National Center for Atmospheric Research
>> >> > >> > Research Applications Laboratory
>> >> > >> > 303-497-2768
>> >> > >> > ---
>> >> > >> > My working day may not be your working day. Please do not
feel
>> >> obliged
>> >> > >> to
>> >> > >> > reply to this email outside of your normal working hours.
>> >> > >> >
>> >> > >> >
>> >> > >>
>> >> > >>
>> >> > >
>> >> > > --
>> >> > > George McCabe - Software Engineer III
>> >> > > National Center for Atmospheric Research
>> >> > > Research Applications Laboratory
>> >> > > 303-497-2768
>> >> > > ---
>> >> > > My working day may not be your working day. Please do not
feel
>> >> obliged to
>> >> > > reply to this email outside of your normal working hours.
>> >> > >
>> >> >
>> >> >
>> >> > --
>> >> > George McCabe - Software Engineer III
>> >> > National Center for Atmospheric Research
>> >> > Research Applications Laboratory
>> >> > 303-497-2768
>> >> > ---
>> >> > My working day may not be your working day. Please do not feel
>> obliged
>> >> to
>> >> > reply to this email outside of your normal working hours.
>> >> >
>> >> >
>> >>
>> >>
>> >
>> > --
>> > George McCabe - Software Engineer III
>> > National Center for Atmospheric Research
>> > Research Applications Laboratory
>> > 303-497-2768
>> > ---
>> > My working day may not be your working day. Please do not feel
obliged
>> to
>> > reply to this email outside of your normal working hours.
>> >
>>
>>
>> --
>> George McCabe - Software Engineer III
>> National Center for Atmospheric Research
>> Research Applications Laboratory
>> 303-497-2768
>> ---
>> My working day may not be your working day. Please do not feel
obliged to
>> reply to this email outside of your normal working hours.
>>
>
------------------------------------------------
Subject: tcpairs
From: Guang Lou - NOAA Affiliate
Time: Tue Aug 17 11:35:29 2021
Hi George,
I've managed to do the plotter only with existing reformatted .tcst
files.
So this is resolved. But the help on the option for removing the error
range bar is still needed.
Thanks,
Guang Ping
On Tue, Aug 17, 2021 at 11:47 AM Guang Lou - NOAA Affiliate <
guang.ping.lou at noaa.gov> wrote:
> Hi George,
>
> One more question: if the tracks have converted to .tcst format, I
just
> wanted to do the plotting, what parameters I need to set?
> I tried these:
> PROCESS_LIST = TCMPRPlotter
> #PROCESS_LIST = TCPairs, TCMPRPlotter
> It gives error messages.
>
> I tried a few other changes and none seemed to work.
> The reason I wanted to do this is because it will save time once the
> format is converted and saved. I can just plot out whatever length
(yearly,
> seasonally and monthly) at a much faster speed, instead of everytime
I have
> to convert the format (it takes a long time.)
>
> Thanks again,
> Guang Ping
>
>
> On Tue, Aug 17, 2021 at 8:27 AM Guang Lou - NOAA Affiliate <
> guang.ping.lou at noaa.gov> wrote:
>
>> Hi George,
>>
>> Thank you so much for figuring out the plotting scripts! Following
your
>> suggestion, I was able to generate plots that put all model errors
on the
>> same frame. I've generated the 3 months' average values and I will
do a
>> longer term average.
>>
>> One question for you, is there a place that I can turn off the
error
>> range bar (the sigma range from low to high)? When we have many
models,
>> these bars would crowd out the error lines.
>>
>> Your help is much appreciated!
>> Guang Ping
>>
>> On Mon, Aug 16, 2021 at 1:43 PM George McCabe via RT
<met_help at ucar.edu>
>> wrote:
>>
>>> Hi Guang Ping,
>>>
>>> I was able to generate images with all of the models on Hera.
There were
>>> modifications to the METplus wrapper and R script that were needed
to get
>>> it to work properly on Hera. The config file I used is here:
>>>
>>> /home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8_test2.conf
>>>
>>> I added the following to properly pass in the data directory to
the
>>> script:
>>>
>>> TCMPR_PLOTTER_READ_ALL_FILES = True
>>>
>>> I had to change the value for MET_INSTALL_DIR to another location
that
>>> has
>>> the change to the R script:
>>>
>>> MET_INSTALL_DIR = /scratch1/BMC/dtc/George.Mccabe/met-tmp
>>>
>>> I called run_metplus.py from my feature branch directory to get
the
>>> changes
>>> needed:
>>>
>>>
>>>
/scratch1/BMC/dtc/George.Mccabe/METplus.feature_1057_tcmpr_dir_option/ush/run_metplus.py
>>>
>>> I am rerunning the use case now. When it finishes the images will
be
>>> generated here:
>>>
>>> /scratch1/BMC/dtc/George.Mccabe/tcmpr-test5/tcmpr_plots
>>>
>>> Please take a look and let me know if the images look good to you.
>>>
>>> Thanks,
>>> George
>>>
>>> On Tue, Aug 3, 2021 at 7:59 AM George McCabe <mccabe at ucar.edu>
wrote:
>>>
>>> > Hi Guang Ping,
>>> >
>>> > I see. Thanks for clarifying. The good news is that John HG was
able to
>>> > generate a plot of all of the models. Attached is an image he
>>> generated.
>>> > This is the command he ran to generate the images:
>>> >
>>> > Rscript met/scripts/Rscripts/plot_tcmpr.R -lookin `find ./ -name
>>> "*.tcst"`
>>> > -no_ee -plot BOXPLOT,MEAN
>>> >
>>> > There are 2 things John mentioned to note here:
>>> >
>>> > * That "find" command found 2705 input files and passed them all
in on
>>> the
>>> > command line. That's a long command line and might cause shell
>>> problems on
>>> > some system.
>>> > * The "-no_ee" option DISABLES event equalization, which is
needed
>>> since
>>> > these cases are not the same. With event equalization turned on,
all
>>> the
>>> > cases will be tossed out.
>>> >
>>> > The -no_ee argument can be set by setting this in your METplus
>>> > configuration file:
>>> >
>>> > TCMPR_PLOTTER_NO_EE = yes
>>> >
>>> > Howeer, his first comment aligns with the error I was seeing
when I
>>> tried
>>> > to run where the shell couldn't handle the large number of files
that
>>> were
>>> > passed to the command. The wrapper finds all of the files in the
input
>>> > directory that end with .tcst and pass in all of the files to
the
>>> Rscript
>>> > command, which is essentially the same behavior as the inserting
the
>>> find
>>> > command in John's example. Hera is down today so I am unable to
test,
>>> but
>>> > it may be the case that Hera cannot handle this command. The
>>> plot_tcmpr.R
>>> > script fails on this line (380), which attempts to expand any
wildcards
>>> > with an ls command:
>>> >
>>> > file_list = system(paste("ls -1", paste(file_list, collapse="
")),
>>> > intern=TRUE);
>>> >
>>> > The list of files is passed from the R script into the tc_stat
MET tool
>>> > for filtering and any wildcards are already expanded by the
wrapper. I
>>> > suspect that if I add an option to this script to bypass the
wildcard
>>> > expansion, then the call to tc_stat will work. I can make the
necessary
>>> > changes locally and test it on Hera when it comes back online.
If
>>> tc_stat
>>> > cannot handle the long command on Hera, then modifications to
the
>>> tc_stat
>>> > tool to support reading in a text file that contains a list of
files
>>> and
>>> > modifications to the TCMPRPlotter wrapper to generate a file
list text
>>> file
>>> > may fix the issue.
>>> >
>>> > I will follow up with more information after some testing. It is
>>> > reassuring to know that the images you are trying to create can
be
>>> > generated.
>>> >
>>> > Thanks,
>>> > George
>>> >
>>> > On Thu, Jul 29, 2021 at 7:55 AM Guang Lou - NOAA Affiliate via
RT <
>>> > met_help at ucar.edu> wrote:
>>> >
>>> >> Hi George,
>>> >>
>>> >> Thanks for the update. The purpose of this plot is to compare
error
>>> stats
>>> >> between different models. So we need to plot all model errors
in one
>>> frame
>>> >> instead of each model having its own frame. I did not state
clearly
>>> in my
>>> >> last email.
>>> >>
>>> >> Thanks,
>>> >> Guang Ping
>>> >>
>>> >> On Wed, Jul 28, 2021 at 4:32 PM George McCabe via RT <
>>> met_help at ucar.edu>
>>> >> wrote:
>>> >>
>>> >> > Hi Guang Ping,
>>> >> >
>>> >> > I updated the config file to process each model separately.
Here is
>>> the
>>> >> > config file on hera:
>>> >> >
>>> >> > /home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8_test.conf
>>> >> >
>>> >> > The output from my latest run can be found in here:
>>> >> >
>>> >> > /scratch1/BMC/dtc/George.Mccabe/tcmpr-test4/tcmpr_plots
>>> >> >
>>> >> > There is a sub-directory for each model. Please take a look
at the
>>> >> output
>>> >> > files to see if they look OK. The config file could be
cleaned up a
>>> >> little
>>> >> > bit if I add support for TCMPR_PLOTTER_CUSTOM_LOOP_LIST or
support
>>> for
>>> >> > reading the instance name from the TCMPRPlotter instances. I
will
>>> make a
>>> >> > GitHub issue to implement those changes as they are
relatively easy
>>> >> > additions and would make the wrapper consistent with the
other
>>> wrappers.
>>> >> >
>>> >> > Thanks,
>>> >> > George
>>> >> >
>>> >> > On Wed, Jul 28, 2021 at 10:02 AM George McCabe
<mccabe at ucar.edu>
>>> wrote:
>>> >> >
>>> >> > > Hi Guang Ping,
>>> >> > >
>>> >> > > OK, that is good to know. I have a few meetings this
morning but I
>>> >> will
>>> >> > > try to update your configuration file to set this up.
>>> >> > >
>>> >> > > - George
>>> >> > >
>>> >> > > On Wed, Jul 28, 2021 at 9:46 AM Guang Lou - NOAA Affiliate
via RT
>>> <
>>> >> > > met_help at ucar.edu> wrote:
>>> >> > >
>>> >> > >> Hi George,
>>> >> > >>
>>> >> > >> The verification should be against self analysis. That is
GFS
>>> adeck
>>> >> > >> against
>>> >> > >> GFS bdeck, CMC adeck against CMC bdeck, and so on and so
forth.
>>> >> > >>
>>> >> > >> Thanks,
>>> >> > >> Guang Ping
>>> >> > >>
>>> >> > >> On Tue, Jul 27, 2021 at 2:33 PM George McCabe via RT <
>>> >> met_help at ucar.edu
>>> >> > >
>>> >> > >> wrote:
>>> >> > >>
>>> >> > >> > Hi Guang Ping,
>>> >> > >> >
>>> >> > >> > I looked at this use case more today. I modified the use
case
>>> to
>>> >> read
>>> >> > a
>>> >> > >> > single month into TCMPRPlotter because it cannot handle
all of
>>> the
>>> >> > >> files at
>>> >> > >> > once. It looks like you need to set the -bmodel in
>>> >> > TCMPR_PLOTTER_FILTER
>>> >> > >> to
>>> >> > >> > specify the Bdeck to compare the models to. However,
each of
>>> the
>>> >> > output
>>> >> > >> > files from tc_pairs use the same model for adeck and
bdeck, so
>>> the
>>> >> > >> AMODEL
>>> >> > >> > and BMODEL values in those files are the same, i.e. both
GFSO.
>>> If
>>> >> you
>>> >> > >> want
>>> >> > >> > to compare all of the models to a single bdeck, you will
have
>>> to
>>> >> > >> configure
>>> >> > >> > TCPairs wrapper to use the same bdeck for each model
with
>>> >> > >> > TC_PAIRS_BDECK_TEMPLATE. I tried running using best for
the
>>> bdeck
>>> >> and
>>> >> > it
>>> >> > >> > produced no pairs. I am rerunning now using GFSO.
>>> >> > >> >
>>> >> > >> > - George
>>> >> > >> >
>>> >> > >> > On Mon, Jul 26, 2021 at 5:57 PM George McCabe
<mccabe at ucar.edu
>>> >
>>> >> > wrote:
>>> >> > >> >
>>> >> > >> > > Hi Guang Ping,
>>> >> > >> > >
>>> >> > >> > > I copied your config file and made some changes. The
new file
>>> >> can be
>>> >> > >> > found
>>> >> > >> > > on hera here:
>>> >> > >> > >
>>> >> > >> > > /home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8.conf
>>> >> > >> > >
>>> >> > >> > > Using METplus 4.1.0-beta1, you can set
TC_PAIRS_RUN_ONCE =
>>> True
>>> >> to
>>> >> > >> loop
>>> >> > >> > > over the init times even with LOOP_ORDER = processes.
>>> >> > >> > >
>>> >> > >> > > I noticed that you have TC_PAIRS_REFORMAT_DECK = yes,
which
>>> isn't
>>> >> > >> > > necessary because tc_pairs can read the input files
without
>>> >> > >> reformatting.
>>> >> > >> > > That can be unset or set to False/no.
>>> >> > >> > >
>>> >> > >> > > TC_PAIRS_REFORMAT_DECK = no
>>> >> > >> > >
>>> >> > >> > > I misread your email about trying to use the wildcard
for the
>>> >> model.
>>> >> > >> This
>>> >> > >> > > won't work as you expect because the output template
will not
>>> >> know
>>> >> > >> how to
>>> >> > >> > > name the output file with the correct model. Each
model will
>>> >> > overwrite
>>> >> > >> > the
>>> >> > >> > > output files. To get around this you can use
>>> >> > >> TC_PAIRS_CUSTOM_LOOP_LIST.
>>> >> > >> > > This will run the TCPairs wrapper once for each item
in the
>>> >> custom
>>> >> > >> loop
>>> >> > >> > > list. Here you can specify the list of models
(lowercase) and
>>> >> > >> reference
>>> >> > >> > the
>>> >> > >> > > {custom} template tag to find the files.
>>> >> > >> > >
>>> >> > >> > > TC_PAIRS_CUSTOM_LOOP_LIST = best, cmc, emx, gfso, nam,
ngx,
>>> ukx
>>> >> > >> > >
>>> >> > >> > > TC_PAIRS_ADECK_TEMPLATE =
>>> >> > >> > >
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >>
>>>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
>>> >> > >> > > TC_PAIRS_BDECK_TEMPLATE =
>>> >> > >> > >
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >>
>>>
{date?fmt=%Y%m}/b{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
>>> >> > >> > > TC_PAIRS_OUTPUT_TEMPLATE =
>>> >> > >> > >
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >>
>>>
{date?fmt=%Y%m}/{basin?fmt=%s}q{date?fmt=%Y%m%d%H}.{custom}.{cyclone?fmt=%s}
>>> >> > >> > >
>>> >> > >> > > I removed the MODEL values (you had it twice in your
config
>>> file)
>>> >> > >> because
>>> >> > >> > > we don't need to filter by the model if we are reading
all of
>>> >> them
>>> >> > in
>>> >> > >> > > individually.
>>> >> > >> > >
>>> >> > >> > > Using this config file I was able to run TCPairs for
each
>>> model
>>> >> > >> > processing
>>> >> > >> > > each for all 3 months and creating output files from
tc_pairs
>>> >> here:
>>> >> > >> > >
>>> >> > >> > > /scratch1/BMC/dtc/George.Mccabe/tcmpr-test2/tc_pairs
>>> >> > >> > >
>>> >> > >> > > The error in your log file from TCMPRPlotter mentions
that
>>> you do
>>> >> > not
>>> >> > >> > have
>>> >> > >> > > -bmodel set in the filter options. When I ran it, I
got an
>>> error
>>> >> > with
>>> >> > >> a
>>> >> > >> > > command that appears to list the files which leads me
to
>>> think
>>> >> that
>>> >> > >> there
>>> >> > >> > > are too many files in the tc_pairs directory to
process at
>>> once.
>>> >> I
>>> >> > >> need
>>> >> > >> > to
>>> >> > >> > > stop working for today, but I will resume debugging
this
>>> tomorrow
>>> >> > and
>>> >> > >> see
>>> >> > >> > > what I can find. It may be the case that the plotter
cannot
>>> >> handle
>>> >> > >> all 7
>>> >> > >> > > models over a 3 day period all at once. If this is the
case,
>>> what
>>> >> > >> would
>>> >> > >> > be
>>> >> > >> > > useful to display as a subset of this data?
>>> >> > >> > >
>>> >> > >> > > Thanks,
>>> >> > >> > > George
>>> >> > >> > >
>>> >> > >> > >
>>> >> > >> > >
>>> >> > >> > > On Fri, Jul 23, 2021 at 11:46 AM Guang Lou - NOAA
Affiliate
>>> via
>>> >> RT <
>>> >> > >> > > met_help at ucar.edu> wrote:
>>> >> > >> > >
>>> >> > >> > >>
>>> >> > >> > >> <URL:
>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100602
>>> >> >
>>> >> > >> > >>
>>> >> > >> > >> Hi George,
>>> >> > >> > >>
>>> >> > >> > >> Thanks for letting me know you have downloaded the
files.
>>> I've
>>> >> > >> removed
>>> >> > >> > the
>>> >> > >> > >> files to save disk space.
>>> >> > >> > >> Per your suggestion, I was able to generate
consecutive
>>> months
>>> >> of
>>> >> > >> paired
>>> >> > >> > >> tracks by changing a few lines
>>> >> > >> > >> and parameters in the configure file. I've also
loaded the
>>> >> module
>>> >> > >> > >> "metplus/4.1.0-beta1".
>>> >> > >> > >>
>>> >> > >> > >> This was done by:
>>> >> > >> > >>
>>> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_pairs
>>> >> > >> > >> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c
>>> >> > >> > >> usecase_pairing_8.conf
>>> >> > >> > >>
>>> >> > >> > >> However, when I tried to generate plots from the just
>>> generated
>>> >> > >> pairs,
>>> >> > >> > it
>>> >> > >> > >> ran into error:
>>> >> > >> > >> in
>>> >> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_pairs
>>> >> > >> > >> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf -c
>>> >> usecase_8.conf
>>> >> > >> > >>
>>> >> > >> > >> where I changed the following to:
>>> >> > >> > >> LOOP_ORDER = processes
>>> >> > >> > >> PROCESS_LIST = TCMPRPlotter
>>> >> > >> > >> error messages are:
>>> >> > >> > >> 07/23 17:38:51.477 metplus INFO: Check the logfile
for more
>>> >> > >> information
>>> >> > >> > on
>>> >> > >> > >> why it failed:
>>> >> > >> > >>
>>> >> > >> > >>
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >>
>>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
>>> >> > >> > >> 07/23 17:38:51.478 metplus INFO: Plotting complete
>>> >> > >> > >> 07/23 17:38:51.481 metplus.TCMPRPlotter
(met_util.py:171)
>>> ERROR:
>>> >> > >> > >> TCMPRPlotter had 9 errors.
>>> >> > >> > >> 07/23 17:38:51.481 metplus INFO:
>>> >> > >> > >>
>>> >> > >> > >>
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >>
>>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/metplus_final.conf:
>>> >> > >> > >> write metplus.conf here
>>> >> > >> > >> 07/23 17:38:51.486 metplus (met_util.py:208) ERROR:
METplus
>>> has
>>> >> > >> finished
>>> >> > >> > >> running but had 9 errors.
>>> >> > >> > >> 07/23 17:38:51.486 metplus INFO: Check the log file
for more
>>> >> > >> > information:
>>> >> > >> > >>
>>> >> > >> > >>
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >>
>>> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
>>> >> > >> > >>
-----------------------------------------------------------
>>> >> > >> > >> The wild card in the place of
>>> >> > >> > >> TC_PAIRS_ADECK_TEMPLATE =
>>> >> > >> > >>
>>> >> >
>>>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.nam.{cyclone?fmt=%s}
>>> >> > >> > >> to
>>> >> > >> > >> TC_PAIRS_ADECK_TEMPLATE =
>>> >> > >> > >>
>>> >>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
>>> >> > >> > >> still does not work.
>>> >> > >> > >>
>>> >> > >> > >> Thanks,
>>> >> > >> > >> Guang Ping
>>> >> > >> > >>
>>> >> > >> > >> On Wed, Jul 21, 2021 at 2:41 PM George McCabe via RT
<
>>> >> > >> met_help at ucar.edu
>>> >> > >> > >
>>> >> > >> > >> wrote:
>>> >> > >> > >>
>>> >> > >> > >> > Hi Guang Ping,
>>> >> > >> > >> >
>>> >> > >> > >> > I have finished downloading the GFS NetCDF files
from
>>> Hera so
>>> >> you
>>> >> > >> can
>>> >> > >> > >> > remove them. Thanks again for providing that data.
>>> >> > >> > >> >
>>> >> > >> > >> > - George
>>> >> > >> > >> >
>>> >> > >> > >> > On Tue, Jul 20, 2021 at 3:35 PM George McCabe <
>>> >> mccabe at ucar.edu>
>>> >> > >> > wrote:
>>> >> > >> > >> >
>>> >> > >> > >> > > Hi Guang Ping,
>>> >> > >> > >> > >
>>> >> > >> > >> > > I mistyped in my previous email. The variable to
set to
>>> >> "times"
>>> >> > >> is
>>> >> > >> > >> > > LOOP_ORDER not LOOP_BY.
>>> >> > >> > >> > >
>>> >> > >> > >> > > LOOP_ORDER = times
>>> >> > >> > >> > > INIT_INCREMENT = 1m
>>> >> > >> > >> > >
>>> >> > >> > >> > > Sorry for the confusion.
>>> >> > >> > >> > >
>>> >> > >> > >> > > - George
>>> >> > >> > >> > >
>>> >> > >> > >> > > On Tue, Jul 20, 2021 at 3:21 PM George McCabe <
>>> >> mccabe at ucar.edu
>>> >> > >
>>> >> > >> > >> wrote:
>>> >> > >> > >> > >
>>> >> > >> > >> > >> Hi Guang Ping,
>>> >> > >> > >> > >>
>>> >> > >> > >> > >> Thank you for providing the NetCDF files. I am
>>> currently
>>> >> > >> > downloading
>>> >> > >> > >> one
>>> >> > >> > >> > >> of the files but it is going very slowly.
>>> >> > >> > >> > >>
>>> >> > >> > >> > >> In regards to your use case, the logic for
TCPairs
>>> wrapper
>>> >> is
>>> >> > a
>>> >> > >> > >> little
>>> >> > >> > >> > >> strange. It behaves differently based on the
value of
>>> >> LOOP_BY.
>>> >> > >> If
>>> >> > >> > >> > LOOP_BY =
>>> >> > >> > >> > >> processes, then it will run once using the
INIT_BEG
>>> time as
>>> >> > the
>>> >> > >> run
>>> >> > >> > >> > time to
>>> >> > >> > >> > >> find input files. If LOOP_BY = times, then it
will loop
>>> >> over
>>> >> > >> each
>>> >> > >> > run
>>> >> > >> > >> > time
>>> >> > >> > >> > >> specified between INIT_BEG and INIT_END by
>>> INIT_INCREMENT
>>> >> and
>>> >> > >> find
>>> >> > >> > >> > files to
>>> >> > >> > >> > >> process for each run time.
>>> >> > >> > >> > >>
>>> >> > >> > >> > >> If you change your configuration file to include
>>> LOOP_BY =
>>> >> > >> times,
>>> >> > >> > >> then
>>> >> > >> > >> > it
>>> >> > >> > >> > >> should look over each run time and process for
each. It
>>> >> looks
>>> >> > >> like
>>> >> > >> > >> your
>>> >> > >> > >> > >> INIT_INCREMENT is set to 21600 or 6 hours, so it
will
>>> run
>>> >> > every
>>> >> > >> 6
>>> >> > >> > >> hours
>>> >> > >> > >> > >> from 20200201 to 2020331. If you want to run
once per
>>> >> month,
>>> >> > you
>>> >> > >> > will
>>> >> > >> > >> > need
>>> >> > >> > >> > >> to set INIT_INCREMENT = 1m so that it will
increment
>>> by 1
>>> >> > month
>>> >> > >> for
>>> >> > >> > >> each
>>> >> > >> > >> > >> iteration.
>>> >> > >> > >> > >>
>>> >> > >> > >> > >> The caveat is that TCMPRPlotter will only run if
>>> LOOP_BY =
>>> >> > >> > processes
>>> >> > >> > >> and
>>> >> > >> > >> > >> will produce an error if LOOP_BY = times. This
is
>>> because
>>> >> we
>>> >> > >> > >> typically
>>> >> > >> > >> > want
>>> >> > >> > >> > >> to process all of the data that is available for
all
>>> run
>>> >> > times,
>>> >> > >> > then
>>> >> > >> > >> run
>>> >> > >> > >> > >> TCMPRPlotter so it can read all of the data. For
now,
>>> to
>>> >> > handle
>>> >> > >> > this
>>> >> > >> > >> use
>>> >> > >> > >> > >> case you will have to call run_metplus.py 2
separate
>>> times,
>>> >> > once
>>> >> > >> > >> calling
>>> >> > >> > >> > >> TCPairs with LOOP_BY = times and once calling
>>> TCMPRPlotter
>>> >> > with
>>> >> > >> > >> LOOP_BY
>>> >> > >> > >> > =
>>> >> > >> > >> > >> processes.
>>> >> > >> > >> > >>
>>> >> > >> > >> > >> I am going to add a configuration option to
TCPairs
>>> >> wrapper so
>>> >> > >> that
>>> >> > >> > >> you
>>> >> > >> > >> > >> can loop over the run times when LOOP_BY =
processes is
>>> >> set if
>>> >> > >> > >> desired.
>>> >> > >> > >> > I
>>> >> > >> > >> > >> will try to get this change into METplus 4.1.0-
beta1
>>> so you
>>> >> > can
>>> >> > >> > test
>>> >> > >> > >> the
>>> >> > >> > >> > >> change and make sure it works as expected. I can
let
>>> you
>>> >> know
>>> >> > >> when
>>> >> > >> > >> that
>>> >> > >> > >> > is
>>> >> > >> > >> > >> available.
>>> >> > >> > >> > >>
>>> >> > >> > >> > >> Thanks,
>>> >> > >> > >> > >> George
>>> >> > >> > >> > >>
>>> >> > >> > >> > >> On Tue, Jul 20, 2021 at 1:27 PM Guang Lou - NOAA
>>> Affiliate
>>> >> via
>>> >> > >> RT <
>>> >> > >> > >> > >> met_help at ucar.edu> wrote:
>>> >> > >> > >> > >>
>>> >> > >> > >> > >>> Hi John,
>>> >> > >> > >> > >>>
>>> >> > >> > >> > >>> Thanks for forwarding George's email answer to
me. I
>>> >> created
>>> >> > a
>>> >> > >> > test
>>> >> > >> > >> > case
>>> >> > >> > >> > >>> that does the tcpairs only on Hera:
>>> >> > >> > >> > >>>
>>> >> > /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_test3
>>> >> > >> > >> > >>> Command line:
>>> >> > >> > >> > >>> run_metplus.py -c
>>> Plotter_fcstGFS_obsGFS_RPlotting_8.conf
>>> >> -c
>>> >> > >> > >> > >>> usecase_8.conf
>>> >> > >> > >> > >>> where PROCESS_LIST = TCPairs
>>> >> > >> > >> > >>>
>>> >> > >> > >> > >>> It generated the month of 202002, that is
because I
>>> >> changed
>>> >> > >> > >> > >>> INIT_BEG = 20200101
>>> >> > >> > >> > >>> INIT_END = 20200331
>>> >> > >> > >> > >>> to
>>> >> > >> > >> > >>> INIT_BEG = 20200201
>>> >> > >> > >> > >>> INIT_END = 20200331
>>> >> > >> > >> > >>>
>>> >> > >> > >> > >>> If I use "INIT_BEG = 20200101,INIT_END =
20200331" it
>>> >> will
>>> >> > >> > generate
>>> >> > >> > >> > >>> 20200101 only.
>>> >> > >> > >> > >>>
>>> >> > >> > >> > >>> The wild card does not work either for the
model's
>>> place:
>>> >> > >> > >> > >>> TC_PAIRS_ADECK_TEMPLATE =
>>> >> > >> > >> > >>>
>>> >> > >> >
>>>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
>>> >> > >> > >> > >>> I populated adeck and bdeck on
>>> >> > >> > >> > >>>
>>> >> > >> > >> > >>>
>>> >> > >> > >> >
>>> >> > >> > >>
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >>
>>> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/METplus/METplus-
4.0_sample_data/model_applications/medium_range/tracks_all
>>> >> > >> > >> > >>> (each subdirectory contains all three months'
decks.)
>>> >> > >> > >> > >>>
>>> >> > >> > >> > >>> I was wondering if you could copy my directory,
>>> >> > >> > >> > >>>
>>> >> > /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_test3
>>> >> > >> > >> > >>> and test:
>>> >> > >> > >> > >>> 1. It can generate multiple months instead of
just one
>>> >> month
>>> >> > >> and
>>> >> > >> > one
>>> >> > >> > >> > >>> model.
>>> >> > >> > >> > >>> 2. The wild card for models in the DECK_TEMPATE
for
>>> >> multiple
>>> >> > >> > models.
>>> >> > >> > >> > >>>
>>> >> > >> > >> > >>> George, I'm downloading NetCDF model data to
Hera:
>>> >> > >> > >> > >>>
>>> >> > >> >
>>>
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071300_nc.tar
>>> >> > >> > >> > >>>
>>> >> > >> >
>>>
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071306_nc.tar
>>> >> > >> > >> > >>>
>>> >> > >> > >> > >>> Please let me know after you downloaded the
files and
>>> I
>>> >> will
>>> >> > >> have
>>> >> > >> > to
>>> >> > >> > >> > >>> remove
>>> >> > >> > >> > >>> them due to tight disk space.
>>> >> > >> > >> > >>>
>>> >> > >> > >> > >>> Thanks,
>>> >> > >> > >> > >>> Guang Ping
>>> >> > >> > >> > >>>
>>> >> > >> > >> > >>> On Mon, Jul 19, 2021 at 5:01 PM John Halley
Gotway
>>> via RT
>>> >> <
>>> >> > >> > >> > >>> met_help at ucar.edu>
>>> >> > >> > >> > >>> wrote:
>>> >> > >> > >> > >>>
>>> >> > >> > >> > >>> > Hi Guang Ping,
>>> >> > >> > >> > >>> >
>>> >> > >> > >> > >>> > The command that you ran here:
>>> >> > >> > >> > >>> >
>>> >> > >> > >> > >>> >
>>> >> > >> > >> > >>> >
>>> >> > >> > >> > >>>
>>> >> > >> > >> >
>>> >> > >> > >>
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >>
>>>
/home/George.Mccabe/scratch1_George.Mccabe/METplus.feature_926/ush/run_metplus.py
>>> >> > >> > >> > >>> > -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf -c
>>> >> > Plotter_8.conf
>>> >> > >> > >> > >>> >
>>> >> > >> > >> > >>> > only runs TCMPRPlotter. This is because
>>> PROCESS_LIST is
>>> >> > >> defined
>>> >> > >> > in
>>> >> > >> > >> > both
>>> >> > >> > >> > >>> > Plotter_fcstGFS_obsGFS_RPlotting_8.conf and
>>> >> Plotter_8.conf.
>>> >> > >> > Since
>>> >> > >> > >> > >>> > Plotter_8.conf comes last, then the value for
>>> >> PROCESS_LIST
>>> >> > >> from
>>> >> > >> > >> that
>>> >> > >> > >> > >>> file
>>> >> > >> > >> > >>> > is used, which only has TCMPRPlotter listed.
The
>>> >> plotter is
>>> >> > >> > >> reading
>>> >> > >> > >> > >>> all of
>>> >> > >> > >> > >>> > the available files generated from TCPairs
and since
>>> >> that
>>> >> > >> > process
>>> >> > >> > >> is
>>> >> > >> > >> > >>> not
>>> >> > >> > >> > >>> > running again with the new time range, the
data
>>> that is
>>> >> > read
>>> >> > >> > only
>>> >> > >> > >> > >>> covers
>>> >> > >> > >> > >>> > the first month.
>>> >> > >> > >> > >>> >
>>> >> > >> > >> > >>> > We are planning on releasing METplus 4.1.0-
beta1
>>> this
>>> >> week.
>>> >> > >> > >> However,
>>> >> > >> > >> > >>> if I
>>> >> > >> > >> > >>> > understand what you are trying to do here, it
>>> shouldn't
>>> >> > >> require
>>> >> > >> > >> the
>>> >> > >> > >> > >>> changes
>>> >> > >> > >> > >>> > described in Molly's issue. It looks like you
want
>>> to
>>> >> run
>>> >> > >> > TCPairs
>>> >> > >> > >> > once
>>> >> > >> > >> > >>> and
>>> >> > >> > >> > >>> > filter the data based on the init and valid
times.
>>> While
>>> >> > this
>>> >> > >> > >> > >>> configuration
>>> >> > >> > >> > >>> > should achieve what you are trying to do, the
>>> following
>>> >> > >> > >> configuration
>>> >> > >> > >> > >>> is
>>> >> > >> > >> > >>> > more explicit to what you are actually
running:
>>> >> > >> > >> > >>> >
>>> >> > >> > >> > >>> > # run once using 20200101 to substitute the
init
>>> time in
>>> >> > your
>>> >> > >> > >> string
>>> >> > >> > >> > >>> > substitution, i.e. TC_PAIRS_ADECK_TEMPLATE =
>>> >> > >> > >> > >>> >
>>> >> > >> > >>
>>> >>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
>>> >> > >> > >> > >>> > INIT_BEG = 20200101
>>> >> > >> > >> > >>> > INIT_END = 20200101
>>> >> > >> > >> > >>> >
>>> >> > >> > >> > >>> > # filter the init times of the lines read
from the
>>> input
>>> >> > >> files
>>> >> > >> > in
>>> >> > >> > >> the
>>> >> > >> > >> > >>> > TCPairs configuration file
>>> >> > >> > >> > >>> > TC_PAIRS_INIT_BEG = 20200101
>>> >> > >> > >> > >>> > TC_PAIRS_INIT_END = 20200331
>>> >> > >> > >> > >>> >
>>> >> > >> > >> > >>> > # filter the valid times of the lines read
from the
>>> >> input
>>> >> > >> files
>>> >> > >> > in
>>> >> > >> > >> > the
>>> >> > >> > >> > >>> > TCPairs configuration file
>>> >> > >> > >> > >>> > TC_PAIRS_VALID_BEG = 20200101
>>> >> > >> > >> > >>> > TC_PAIRS_VALID_END = 20200331
>>> >> > >> > >> > >>> >
>>> >> > >> > >> > >>> > If any of TC_PAIRS_[INIT/VALID]_[BEG/END] are
not
>>> set,
>>> >> then
>>> >> > >> the
>>> >> > >> > >> > >>> > corresponding [INIT/VALID]_[BEG/END] values
are
>>> used.
>>> >> This
>>> >> > is
>>> >> > >> > >> done to
>>> >> > >> > >> > >>> > preserve backwards compatibility for old
METplus
>>> >> > >> configurations.
>>> >> > >> > >> The
>>> >> > >> > >> > >>> above
>>> >> > >> > >> > >>> > settings are more explicit as to what is
actually be
>>> >> > >> configured.
>>> >> > >> > >> > >>> >
>>> >> > >> > >> > >>> > You may want to consider either clearing your
output
>>> >> > >> directory
>>> >> > >> > or
>>> >> > >> > >> > >>> creating
>>> >> > >> > >> > >>> > a new output directory for a new run
(especially if
>>> the
>>> >> > time
>>> >> > >> > range
>>> >> > >> > >> > you
>>> >> > >> > >> > >>> are
>>> >> > >> > >> > >>> > using has changed) to ensure that the files
you are
>>> >> > generated
>>> >> > >> > are
>>> >> > >> > >> > from
>>> >> > >> > >> > >>> that
>>> >> > >> > >> > >>> > run and not from a previous run. Doing this
would
>>> have
>>> >> > >> alerted
>>> >> > >> > you
>>> >> > >> > >> > that
>>> >> > >> > >> > >>> > TCPairs was not run in this case.
>>> >> > >> > >> > >>> >
>>> >> > >> > >> > >>> > Let me know if you have any other issues
getting
>>> this
>>> >> > >> running.
>>> >> > >> > >> > >>> >
>>> >> > >> > >> > >>> > FYI:
>>> >> > >> > >> > >>> > We switched from providing support through
this
>>> email
>>> >> > >> address to
>>> >> > >> > >> > >>> providing
>>> >> > >> > >> > >>> > support through GitHub Discussions. In the
future,
>>> you
>>> >> will
>>> >> > >> need
>>> >> > >> > >> to
>>> >> > >> > >> > >>> create
>>> >> > >> > >> > >>> > a free GitHub account if you don’t have one
already
>>> and
>>> >> > post
>>> >> > >> > your
>>> >> > >> > >> > >>> questions
>>> >> > >> > >> > >>> > to the METplus Components Discussion Forum:
>>> >> > >> > >> > >>> >
https://github.com/dtcenter/METplus/discussions
>>> >> > >> > >> > >>> > To post a new question, click on the green
button
>>> in the
>>> >> > >> upper
>>> >> > >> > >> right
>>> >> > >> > >> > >>> corner
>>> >> > >> > >> > >>> > of the page labeled "New discussion" and
select
>>> >> "Incoming"
>>> >> > as
>>> >> > >> > the
>>> >> > >> > >> > >>> category.
>>> >> > >> > >> > >>> >
>>> >> > >> > >> > >>> > Thanks,
>>> >> > >> > >> > >>> > George
>>> >> > >> > >> > >>> >
>>> >> > >> > >> > >>> >
>>> >> > >> > >> > >>>
>>> >> > >> > >> > >>>
>>> >> > >> > >> > >>
>>> >> > >> > >> > >> --
>>> >> > >> > >> > >> George McCabe - Software Engineer III
>>> >> > >> > >> > >> National Center for Atmospheric Research
>>> >> > >> > >> > >> Research Applications Laboratory
>>> >> > >> > >> > >> 303-497-2768
>>> >> > >> > >> > >> ---
>>> >> > >> > >> > >> My working day may not be your working day.
Please do
>>> not
>>> >> feel
>>> >> > >> > >> obliged
>>> >> > >> > >> > to
>>> >> > >> > >> > >> reply to this email outside of your normal
working
>>> hours.
>>> >> > >> > >> > >>
>>> >> > >> > >> > >
>>> >> > >> > >> > >
>>> >> > >> > >> > > --
>>> >> > >> > >> > > George McCabe - Software Engineer III
>>> >> > >> > >> > > National Center for Atmospheric Research
>>> >> > >> > >> > > Research Applications Laboratory
>>> >> > >> > >> > > 303-497-2768
>>> >> > >> > >> > > ---
>>> >> > >> > >> > > My working day may not be your working day.
Please do
>>> not
>>> >> feel
>>> >> > >> > >> obliged to
>>> >> > >> > >> > > reply to this email outside of your normal
working
>>> hours.
>>> >> > >> > >> > >
>>> >> > >> > >> >
>>> >> > >> > >> >
>>> >> > >> > >> > --
>>> >> > >> > >> > George McCabe - Software Engineer III
>>> >> > >> > >> > National Center for Atmospheric Research
>>> >> > >> > >> > Research Applications Laboratory
>>> >> > >> > >> > 303-497-2768
>>> >> > >> > >> > ---
>>> >> > >> > >> > My working day may not be your working day. Please
do not
>>> feel
>>> >> > >> obliged
>>> >> > >> > >> to
>>> >> > >> > >> > reply to this email outside of your normal working
hours.
>>> >> > >> > >> >
>>> >> > >> > >> >
>>> >> > >> > >>
>>> >> > >> > >>
>>> >> > >> > >
>>> >> > >> > > --
>>> >> > >> > > George McCabe - Software Engineer III
>>> >> > >> > > National Center for Atmospheric Research
>>> >> > >> > > Research Applications Laboratory
>>> >> > >> > > 303-497-2768
>>> >> > >> > > ---
>>> >> > >> > > My working day may not be your working day. Please do
not
>>> feel
>>> >> > >> obliged to
>>> >> > >> > > reply to this email outside of your normal working
hours.
>>> >> > >> > >
>>> >> > >> >
>>> >> > >> >
>>> >> > >> > --
>>> >> > >> > George McCabe - Software Engineer III
>>> >> > >> > National Center for Atmospheric Research
>>> >> > >> > Research Applications Laboratory
>>> >> > >> > 303-497-2768
>>> >> > >> > ---
>>> >> > >> > My working day may not be your working day. Please do
not feel
>>> >> obliged
>>> >> > >> to
>>> >> > >> > reply to this email outside of your normal working
hours.
>>> >> > >> >
>>> >> > >> >
>>> >> > >>
>>> >> > >>
>>> >> > >
>>> >> > > --
>>> >> > > George McCabe - Software Engineer III
>>> >> > > National Center for Atmospheric Research
>>> >> > > Research Applications Laboratory
>>> >> > > 303-497-2768
>>> >> > > ---
>>> >> > > My working day may not be your working day. Please do not
feel
>>> >> obliged to
>>> >> > > reply to this email outside of your normal working hours.
>>> >> > >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > George McCabe - Software Engineer III
>>> >> > National Center for Atmospheric Research
>>> >> > Research Applications Laboratory
>>> >> > 303-497-2768
>>> >> > ---
>>> >> > My working day may not be your working day. Please do not
feel
>>> obliged
>>> >> to
>>> >> > reply to this email outside of your normal working hours.
>>> >> >
>>> >> >
>>> >>
>>> >>
>>> >
>>> > --
>>> > George McCabe - Software Engineer III
>>> > National Center for Atmospheric Research
>>> > Research Applications Laboratory
>>> > 303-497-2768
>>> > ---
>>> > My working day may not be your working day. Please do not feel
obliged
>>> to
>>> > reply to this email outside of your normal working hours.
>>> >
>>>
>>>
>>> --
>>> George McCabe - Software Engineer III
>>> National Center for Atmospheric Research
>>> Research Applications Laboratory
>>> 303-497-2768
>>> ---
>>> My working day may not be your working day. Please do not feel
obliged to
>>> reply to this email outside of your normal working hours.
>>>
>>
------------------------------------------------
Subject: tcpairs
From: George McCabe
Time: Wed Aug 18 09:12:42 2021
Hi Guang Ping,
I'm glad you were able to resolve some of your issues. I'll admit that
I am
not entirely familiar with the plot_tcmpr.R script and the plots that
is
generates, but the script doesn't appear to have any option to remove
the
error range bars. I have reached out to other METplus team members
that may
know something that I am missing just in case. You can run the script
without any arguments to view the usage statement which lists the
available
options. I would recommend reviewing this list, as there may be
something
that you can set to reduce the clutter in the images.
Thanks,
George
On Tue, Aug 17, 2021 at 11:36 AM Guang Lou - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
> Hi George,
>
> I've managed to do the plotter only with existing reformatted .tcst
files.
> So this is resolved. But the help on the option for removing the
error
> range bar is still needed.
>
> Thanks,
> Guang Ping
>
> On Tue, Aug 17, 2021 at 11:47 AM Guang Lou - NOAA Affiliate <
> guang.ping.lou at noaa.gov> wrote:
>
> > Hi George,
> >
> > One more question: if the tracks have converted to .tcst format, I
just
> > wanted to do the plotting, what parameters I need to set?
> > I tried these:
> > PROCESS_LIST = TCMPRPlotter
> > #PROCESS_LIST = TCPairs, TCMPRPlotter
> > It gives error messages.
> >
> > I tried a few other changes and none seemed to work.
> > The reason I wanted to do this is because it will save time once
the
> > format is converted and saved. I can just plot out whatever length
> (yearly,
> > seasonally and monthly) at a much faster speed, instead of
everytime I
> have
> > to convert the format (it takes a long time.)
> >
> > Thanks again,
> > Guang Ping
> >
> >
> > On Tue, Aug 17, 2021 at 8:27 AM Guang Lou - NOAA Affiliate <
> > guang.ping.lou at noaa.gov> wrote:
> >
> >> Hi George,
> >>
> >> Thank you so much for figuring out the plotting scripts!
Following your
> >> suggestion, I was able to generate plots that put all model
errors on
> the
> >> same frame. I've generated the 3 months' average values and I
will do a
> >> longer term average.
> >>
> >> One question for you, is there a place that I can turn off the
error
> >> range bar (the sigma range from low to high)? When we have many
models,
> >> these bars would crowd out the error lines.
> >>
> >> Your help is much appreciated!
> >> Guang Ping
> >>
> >> On Mon, Aug 16, 2021 at 1:43 PM George McCabe via RT
<met_help at ucar.edu
> >
> >> wrote:
> >>
> >>> Hi Guang Ping,
> >>>
> >>> I was able to generate images with all of the models on Hera.
There
> were
> >>> modifications to the METplus wrapper and R script that were
needed to
> get
> >>> it to work properly on Hera. The config file I used is here:
> >>>
> >>> /home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8_test2.conf
> >>>
> >>> I added the following to properly pass in the data directory to
the
> >>> script:
> >>>
> >>> TCMPR_PLOTTER_READ_ALL_FILES = True
> >>>
> >>> I had to change the value for MET_INSTALL_DIR to another
location that
> >>> has
> >>> the change to the R script:
> >>>
> >>> MET_INSTALL_DIR = /scratch1/BMC/dtc/George.Mccabe/met-tmp
> >>>
> >>> I called run_metplus.py from my feature branch directory to get
the
> >>> changes
> >>> needed:
> >>>
> >>>
> >>>
>
/scratch1/BMC/dtc/George.Mccabe/METplus.feature_1057_tcmpr_dir_option/ush/run_metplus.py
> >>>
> >>> I am rerunning the use case now. When it finishes the images
will be
> >>> generated here:
> >>>
> >>> /scratch1/BMC/dtc/George.Mccabe/tcmpr-test5/tcmpr_plots
> >>>
> >>> Please take a look and let me know if the images look good to
you.
> >>>
> >>> Thanks,
> >>> George
> >>>
> >>> On Tue, Aug 3, 2021 at 7:59 AM George McCabe <mccabe at ucar.edu>
wrote:
> >>>
> >>> > Hi Guang Ping,
> >>> >
> >>> > I see. Thanks for clarifying. The good news is that John HG
was able
> to
> >>> > generate a plot of all of the models. Attached is an image he
> >>> generated.
> >>> > This is the command he ran to generate the images:
> >>> >
> >>> > Rscript met/scripts/Rscripts/plot_tcmpr.R -lookin `find ./
-name
> >>> "*.tcst"`
> >>> > -no_ee -plot BOXPLOT,MEAN
> >>> >
> >>> > There are 2 things John mentioned to note here:
> >>> >
> >>> > * That "find" command found 2705 input files and passed them
all in
> on
> >>> the
> >>> > command line. That's a long command line and might cause shell
> >>> problems on
> >>> > some system.
> >>> > * The "-no_ee" option DISABLES event equalization, which is
needed
> >>> since
> >>> > these cases are not the same. With event equalization turned
on, all
> >>> the
> >>> > cases will be tossed out.
> >>> >
> >>> > The -no_ee argument can be set by setting this in your METplus
> >>> > configuration file:
> >>> >
> >>> > TCMPR_PLOTTER_NO_EE = yes
> >>> >
> >>> > Howeer, his first comment aligns with the error I was seeing
when I
> >>> tried
> >>> > to run where the shell couldn't handle the large number of
files that
> >>> were
> >>> > passed to the command. The wrapper finds all of the files in
the
> input
> >>> > directory that end with .tcst and pass in all of the files to
the
> >>> Rscript
> >>> > command, which is essentially the same behavior as the
inserting the
> >>> find
> >>> > command in John's example. Hera is down today so I am unable
to test,
> >>> but
> >>> > it may be the case that Hera cannot handle this command. The
> >>> plot_tcmpr.R
> >>> > script fails on this line (380), which attempts to expand any
> wildcards
> >>> > with an ls command:
> >>> >
> >>> > file_list = system(paste("ls -1", paste(file_list, collapse="
")),
> >>> > intern=TRUE);
> >>> >
> >>> > The list of files is passed from the R script into the tc_stat
MET
> tool
> >>> > for filtering and any wildcards are already expanded by the
wrapper.
> I
> >>> > suspect that if I add an option to this script to bypass the
wildcard
> >>> > expansion, then the call to tc_stat will work. I can make the
> necessary
> >>> > changes locally and test it on Hera when it comes back online.
If
> >>> tc_stat
> >>> > cannot handle the long command on Hera, then modifications to
the
> >>> tc_stat
> >>> > tool to support reading in a text file that contains a list of
files
> >>> and
> >>> > modifications to the TCMPRPlotter wrapper to generate a file
list
> text
> >>> file
> >>> > may fix the issue.
> >>> >
> >>> > I will follow up with more information after some testing. It
is
> >>> > reassuring to know that the images you are trying to create
can be
> >>> > generated.
> >>> >
> >>> > Thanks,
> >>> > George
> >>> >
> >>> > On Thu, Jul 29, 2021 at 7:55 AM Guang Lou - NOAA Affiliate via
RT <
> >>> > met_help at ucar.edu> wrote:
> >>> >
> >>> >> Hi George,
> >>> >>
> >>> >> Thanks for the update. The purpose of this plot is to compare
error
> >>> stats
> >>> >> between different models. So we need to plot all model errors
in one
> >>> frame
> >>> >> instead of each model having its own frame. I did not state
clearly
> >>> in my
> >>> >> last email.
> >>> >>
> >>> >> Thanks,
> >>> >> Guang Ping
> >>> >>
> >>> >> On Wed, Jul 28, 2021 at 4:32 PM George McCabe via RT <
> >>> met_help at ucar.edu>
> >>> >> wrote:
> >>> >>
> >>> >> > Hi Guang Ping,
> >>> >> >
> >>> >> > I updated the config file to process each model separately.
Here
> is
> >>> the
> >>> >> > config file on hera:
> >>> >> >
> >>> >> > /home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8_test.conf
> >>> >> >
> >>> >> > The output from my latest run can be found in here:
> >>> >> >
> >>> >> > /scratch1/BMC/dtc/George.Mccabe/tcmpr-test4/tcmpr_plots
> >>> >> >
> >>> >> > There is a sub-directory for each model. Please take a look
at the
> >>> >> output
> >>> >> > files to see if they look OK. The config file could be
cleaned up
> a
> >>> >> little
> >>> >> > bit if I add support for TCMPR_PLOTTER_CUSTOM_LOOP_LIST or
support
> >>> for
> >>> >> > reading the instance name from the TCMPRPlotter instances.
I will
> >>> make a
> >>> >> > GitHub issue to implement those changes as they are
relatively
> easy
> >>> >> > additions and would make the wrapper consistent with the
other
> >>> wrappers.
> >>> >> >
> >>> >> > Thanks,
> >>> >> > George
> >>> >> >
> >>> >> > On Wed, Jul 28, 2021 at 10:02 AM George McCabe
<mccabe at ucar.edu>
> >>> wrote:
> >>> >> >
> >>> >> > > Hi Guang Ping,
> >>> >> > >
> >>> >> > > OK, that is good to know. I have a few meetings this
morning
> but I
> >>> >> will
> >>> >> > > try to update your configuration file to set this up.
> >>> >> > >
> >>> >> > > - George
> >>> >> > >
> >>> >> > > On Wed, Jul 28, 2021 at 9:46 AM Guang Lou - NOAA
Affiliate via
> RT
> >>> <
> >>> >> > > met_help at ucar.edu> wrote:
> >>> >> > >
> >>> >> > >> Hi George,
> >>> >> > >>
> >>> >> > >> The verification should be against self analysis. That
is GFS
> >>> adeck
> >>> >> > >> against
> >>> >> > >> GFS bdeck, CMC adeck against CMC bdeck, and so on and so
forth.
> >>> >> > >>
> >>> >> > >> Thanks,
> >>> >> > >> Guang Ping
> >>> >> > >>
> >>> >> > >> On Tue, Jul 27, 2021 at 2:33 PM George McCabe via RT <
> >>> >> met_help at ucar.edu
> >>> >> > >
> >>> >> > >> wrote:
> >>> >> > >>
> >>> >> > >> > Hi Guang Ping,
> >>> >> > >> >
> >>> >> > >> > I looked at this use case more today. I modified the
use case
> >>> to
> >>> >> read
> >>> >> > a
> >>> >> > >> > single month into TCMPRPlotter because it cannot
handle all
> of
> >>> the
> >>> >> > >> files at
> >>> >> > >> > once. It looks like you need to set the -bmodel in
> >>> >> > TCMPR_PLOTTER_FILTER
> >>> >> > >> to
> >>> >> > >> > specify the Bdeck to compare the models to. However,
each of
> >>> the
> >>> >> > output
> >>> >> > >> > files from tc_pairs use the same model for adeck and
bdeck,
> so
> >>> the
> >>> >> > >> AMODEL
> >>> >> > >> > and BMODEL values in those files are the same, i.e.
both
> GFSO.
> >>> If
> >>> >> you
> >>> >> > >> want
> >>> >> > >> > to compare all of the models to a single bdeck, you
will have
> >>> to
> >>> >> > >> configure
> >>> >> > >> > TCPairs wrapper to use the same bdeck for each model
with
> >>> >> > >> > TC_PAIRS_BDECK_TEMPLATE. I tried running using best
for the
> >>> bdeck
> >>> >> and
> >>> >> > it
> >>> >> > >> > produced no pairs. I am rerunning now using GFSO.
> >>> >> > >> >
> >>> >> > >> > - George
> >>> >> > >> >
> >>> >> > >> > On Mon, Jul 26, 2021 at 5:57 PM George McCabe <
> mccabe at ucar.edu
> >>> >
> >>> >> > wrote:
> >>> >> > >> >
> >>> >> > >> > > Hi Guang Ping,
> >>> >> > >> > >
> >>> >> > >> > > I copied your config file and made some changes. The
new
> file
> >>> >> can be
> >>> >> > >> > found
> >>> >> > >> > > on hera here:
> >>> >> > >> > >
> >>> >> > >> > >
/home/George.Mccabe/Plotter_fcstGFS_obsGFS_plot_8.conf
> >>> >> > >> > >
> >>> >> > >> > > Using METplus 4.1.0-beta1, you can set
TC_PAIRS_RUN_ONCE =
> >>> True
> >>> >> to
> >>> >> > >> loop
> >>> >> > >> > > over the init times even with LOOP_ORDER =
processes.
> >>> >> > >> > >
> >>> >> > >> > > I noticed that you have TC_PAIRS_REFORMAT_DECK =
yes, which
> >>> isn't
> >>> >> > >> > > necessary because tc_pairs can read the input files
without
> >>> >> > >> reformatting.
> >>> >> > >> > > That can be unset or set to False/no.
> >>> >> > >> > >
> >>> >> > >> > > TC_PAIRS_REFORMAT_DECK = no
> >>> >> > >> > >
> >>> >> > >> > > I misread your email about trying to use the
wildcard for
> the
> >>> >> model.
> >>> >> > >> This
> >>> >> > >> > > won't work as you expect because the output template
will
> not
> >>> >> know
> >>> >> > >> how to
> >>> >> > >> > > name the output file with the correct model. Each
model
> will
> >>> >> > overwrite
> >>> >> > >> > the
> >>> >> > >> > > output files. To get around this you can use
> >>> >> > >> TC_PAIRS_CUSTOM_LOOP_LIST.
> >>> >> > >> > > This will run the TCPairs wrapper once for each item
in the
> >>> >> custom
> >>> >> > >> loop
> >>> >> > >> > > list. Here you can specify the list of models
(lowercase)
> and
> >>> >> > >> reference
> >>> >> > >> > the
> >>> >> > >> > > {custom} template tag to find the files.
> >>> >> > >> > >
> >>> >> > >> > > TC_PAIRS_CUSTOM_LOOP_LIST = best, cmc, emx, gfso,
nam, ngx,
> >>> ukx
> >>> >> > >> > >
> >>> >> > >> > > TC_PAIRS_ADECK_TEMPLATE =
> >>> >> > >> > >
> >>> >> > >> >
> >>> >> > >>
> >>> >> >
> >>> >>
> >>>
>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
> >>> >> > >> > > TC_PAIRS_BDECK_TEMPLATE =
> >>> >> > >> > >
> >>> >> > >> >
> >>> >> > >>
> >>> >> >
> >>> >>
> >>>
>
{date?fmt=%Y%m}/b{basin?fmt=%s}q{date?fmt=%Y%m}*.{custom}.{cyclone?fmt=%s}
> >>> >> > >> > > TC_PAIRS_OUTPUT_TEMPLATE =
> >>> >> > >> > >
> >>> >> > >> >
> >>> >> > >>
> >>> >> >
> >>> >>
> >>>
>
{date?fmt=%Y%m}/{basin?fmt=%s}q{date?fmt=%Y%m%d%H}.{custom}.{cyclone?fmt=%s}
> >>> >> > >> > >
> >>> >> > >> > > I removed the MODEL values (you had it twice in your
config
> >>> file)
> >>> >> > >> because
> >>> >> > >> > > we don't need to filter by the model if we are
reading all
> of
> >>> >> them
> >>> >> > in
> >>> >> > >> > > individually.
> >>> >> > >> > >
> >>> >> > >> > > Using this config file I was able to run TCPairs for
each
> >>> model
> >>> >> > >> > processing
> >>> >> > >> > > each for all 3 months and creating output files from
> tc_pairs
> >>> >> here:
> >>> >> > >> > >
> >>> >> > >> > > /scratch1/BMC/dtc/George.Mccabe/tcmpr-test2/tc_pairs
> >>> >> > >> > >
> >>> >> > >> > > The error in your log file from TCMPRPlotter
mentions that
> >>> you do
> >>> >> > not
> >>> >> > >> > have
> >>> >> > >> > > -bmodel set in the filter options. When I ran it, I
got an
> >>> error
> >>> >> > with
> >>> >> > >> a
> >>> >> > >> > > command that appears to list the files which leads
me to
> >>> think
> >>> >> that
> >>> >> > >> there
> >>> >> > >> > > are too many files in the tc_pairs directory to
process at
> >>> once.
> >>> >> I
> >>> >> > >> need
> >>> >> > >> > to
> >>> >> > >> > > stop working for today, but I will resume debugging
this
> >>> tomorrow
> >>> >> > and
> >>> >> > >> see
> >>> >> > >> > > what I can find. It may be the case that the plotter
cannot
> >>> >> handle
> >>> >> > >> all 7
> >>> >> > >> > > models over a 3 day period all at once. If this is
the
> case,
> >>> what
> >>> >> > >> would
> >>> >> > >> > be
> >>> >> > >> > > useful to display as a subset of this data?
> >>> >> > >> > >
> >>> >> > >> > > Thanks,
> >>> >> > >> > > George
> >>> >> > >> > >
> >>> >> > >> > >
> >>> >> > >> > >
> >>> >> > >> > > On Fri, Jul 23, 2021 at 11:46 AM Guang Lou - NOAA
Affiliate
> >>> via
> >>> >> RT <
> >>> >> > >> > > met_help at ucar.edu> wrote:
> >>> >> > >> > >
> >>> >> > >> > >>
> >>> >> > >> > >> <URL:
> >>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=100602
> >>> >> >
> >>> >> > >> > >>
> >>> >> > >> > >> Hi George,
> >>> >> > >> > >>
> >>> >> > >> > >> Thanks for letting me know you have downloaded the
files.
> >>> I've
> >>> >> > >> removed
> >>> >> > >> > the
> >>> >> > >> > >> files to save disk space.
> >>> >> > >> > >> Per your suggestion, I was able to generate
consecutive
> >>> months
> >>> >> of
> >>> >> > >> paired
> >>> >> > >> > >> tracks by changing a few lines
> >>> >> > >> > >> and parameters in the configure file. I've also
loaded the
> >>> >> module
> >>> >> > >> > >> "metplus/4.1.0-beta1".
> >>> >> > >> > >>
> >>> >> > >> > >> This was done by:
> >>> >> > >> > >>
> >>> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/metplus_pairs
> >>> >> > >> > >> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf
-c
> >>> >> > >> > >> usecase_pairing_8.conf
> >>> >> > >> > >>
> >>> >> > >> > >> However, when I tried to generate plots from the
just
> >>> generated
> >>> >> > >> pairs,
> >>> >> > >> > it
> >>> >> > >> > >> ran into error:
> >>> >> > >> > >> in
> >>> >> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs
> >>> >> > >> > >> run_metplus.py Plotter_fcstGFS_obsGFS_plot_8.conf
-c
> >>> >> usecase_8.conf
> >>> >> > >> > >>
> >>> >> > >> > >> where I changed the following to:
> >>> >> > >> > >> LOOP_ORDER = processes
> >>> >> > >> > >> PROCESS_LIST = TCMPRPlotter
> >>> >> > >> > >> error messages are:
> >>> >> > >> > >> 07/23 17:38:51.477 metplus INFO: Check the logfile
for
> more
> >>> >> > >> information
> >>> >> > >> > on
> >>> >> > >> > >> why it failed:
> >>> >> > >> > >>
> >>> >> > >> > >>
> >>> >> > >> >
> >>> >> > >>
> >>> >> >
> >>> >>
> >>>
> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
> >>> >> > >> > >> 07/23 17:38:51.478 metplus INFO: Plotting complete
> >>> >> > >> > >> 07/23 17:38:51.481 metplus.TCMPRPlotter
(met_util.py:171)
> >>> ERROR:
> >>> >> > >> > >> TCMPRPlotter had 9 errors.
> >>> >> > >> > >> 07/23 17:38:51.481 metplus INFO:
> >>> >> > >> > >>
> >>> >> > >> > >>
> >>> >> > >> >
> >>> >> > >>
> >>> >> >
> >>> >>
> >>>
> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/metplus_final.conf:
> >>> >> > >> > >> write metplus.conf here
> >>> >> > >> > >> 07/23 17:38:51.486 metplus (met_util.py:208) ERROR:
> METplus
> >>> has
> >>> >> > >> finished
> >>> >> > >> > >> running but had 9 errors.
> >>> >> > >> > >> 07/23 17:38:51.486 metplus INFO: Check the log file
for
> more
> >>> >> > >> > information:
> >>> >> > >> > >>
> >>> >> > >> > >>
> >>> >> > >> >
> >>> >> > >>
> >>> >> >
> >>> >>
> >>>
> /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_pairs/met_out/logs/metplus.log.20210723173221
> >>> >> > >> > >>
> -----------------------------------------------------------
> >>> >> > >> > >> The wild card in the place of
> >>> >> > >> > >> TC_PAIRS_ADECK_TEMPLATE =
> >>> >> > >> > >>
> >>> >> >
> >>>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.nam.{cyclone?fmt=%s}
> >>> >> > >> > >> to
> >>> >> > >> > >> TC_PAIRS_ADECK_TEMPLATE =
> >>> >> > >> > >>
> >>> >>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> >>> >> > >> > >> still does not work.
> >>> >> > >> > >>
> >>> >> > >> > >> Thanks,
> >>> >> > >> > >> Guang Ping
> >>> >> > >> > >>
> >>> >> > >> > >> On Wed, Jul 21, 2021 at 2:41 PM George McCabe via
RT <
> >>> >> > >> met_help at ucar.edu
> >>> >> > >> > >
> >>> >> > >> > >> wrote:
> >>> >> > >> > >>
> >>> >> > >> > >> > Hi Guang Ping,
> >>> >> > >> > >> >
> >>> >> > >> > >> > I have finished downloading the GFS NetCDF files
from
> >>> Hera so
> >>> >> you
> >>> >> > >> can
> >>> >> > >> > >> > remove them. Thanks again for providing that
data.
> >>> >> > >> > >> >
> >>> >> > >> > >> > - George
> >>> >> > >> > >> >
> >>> >> > >> > >> > On Tue, Jul 20, 2021 at 3:35 PM George McCabe <
> >>> >> mccabe at ucar.edu>
> >>> >> > >> > wrote:
> >>> >> > >> > >> >
> >>> >> > >> > >> > > Hi Guang Ping,
> >>> >> > >> > >> > >
> >>> >> > >> > >> > > I mistyped in my previous email. The variable
to set
> to
> >>> >> "times"
> >>> >> > >> is
> >>> >> > >> > >> > > LOOP_ORDER not LOOP_BY.
> >>> >> > >> > >> > >
> >>> >> > >> > >> > > LOOP_ORDER = times
> >>> >> > >> > >> > > INIT_INCREMENT = 1m
> >>> >> > >> > >> > >
> >>> >> > >> > >> > > Sorry for the confusion.
> >>> >> > >> > >> > >
> >>> >> > >> > >> > > - George
> >>> >> > >> > >> > >
> >>> >> > >> > >> > > On Tue, Jul 20, 2021 at 3:21 PM George McCabe <
> >>> >> mccabe at ucar.edu
> >>> >> > >
> >>> >> > >> > >> wrote:
> >>> >> > >> > >> > >
> >>> >> > >> > >> > >> Hi Guang Ping,
> >>> >> > >> > >> > >>
> >>> >> > >> > >> > >> Thank you for providing the NetCDF files. I am
> >>> currently
> >>> >> > >> > downloading
> >>> >> > >> > >> one
> >>> >> > >> > >> > >> of the files but it is going very slowly.
> >>> >> > >> > >> > >>
> >>> >> > >> > >> > >> In regards to your use case, the logic for
TCPairs
> >>> wrapper
> >>> >> is
> >>> >> > a
> >>> >> > >> > >> little
> >>> >> > >> > >> > >> strange. It behaves differently based on the
value of
> >>> >> LOOP_BY.
> >>> >> > >> If
> >>> >> > >> > >> > LOOP_BY =
> >>> >> > >> > >> > >> processes, then it will run once using the
INIT_BEG
> >>> time as
> >>> >> > the
> >>> >> > >> run
> >>> >> > >> > >> > time to
> >>> >> > >> > >> > >> find input files. If LOOP_BY = times, then it
will
> loop
> >>> >> over
> >>> >> > >> each
> >>> >> > >> > run
> >>> >> > >> > >> > time
> >>> >> > >> > >> > >> specified between INIT_BEG and INIT_END by
> >>> INIT_INCREMENT
> >>> >> and
> >>> >> > >> find
> >>> >> > >> > >> > files to
> >>> >> > >> > >> > >> process for each run time.
> >>> >> > >> > >> > >>
> >>> >> > >> > >> > >> If you change your configuration file to
include
> >>> LOOP_BY =
> >>> >> > >> times,
> >>> >> > >> > >> then
> >>> >> > >> > >> > it
> >>> >> > >> > >> > >> should look over each run time and process for
each.
> It
> >>> >> looks
> >>> >> > >> like
> >>> >> > >> > >> your
> >>> >> > >> > >> > >> INIT_INCREMENT is set to 21600 or 6 hours, so
it will
> >>> run
> >>> >> > every
> >>> >> > >> 6
> >>> >> > >> > >> hours
> >>> >> > >> > >> > >> from 20200201 to 2020331. If you want to run
once per
> >>> >> month,
> >>> >> > you
> >>> >> > >> > will
> >>> >> > >> > >> > need
> >>> >> > >> > >> > >> to set INIT_INCREMENT = 1m so that it will
increment
> >>> by 1
> >>> >> > month
> >>> >> > >> for
> >>> >> > >> > >> each
> >>> >> > >> > >> > >> iteration.
> >>> >> > >> > >> > >>
> >>> >> > >> > >> > >> The caveat is that TCMPRPlotter will only run
if
> >>> LOOP_BY =
> >>> >> > >> > processes
> >>> >> > >> > >> and
> >>> >> > >> > >> > >> will produce an error if LOOP_BY = times. This
is
> >>> because
> >>> >> we
> >>> >> > >> > >> typically
> >>> >> > >> > >> > want
> >>> >> > >> > >> > >> to process all of the data that is available
for all
> >>> run
> >>> >> > times,
> >>> >> > >> > then
> >>> >> > >> > >> run
> >>> >> > >> > >> > >> TCMPRPlotter so it can read all of the data.
For now,
> >>> to
> >>> >> > handle
> >>> >> > >> > this
> >>> >> > >> > >> use
> >>> >> > >> > >> > >> case you will have to call run_metplus.py 2
separate
> >>> times,
> >>> >> > once
> >>> >> > >> > >> calling
> >>> >> > >> > >> > >> TCPairs with LOOP_BY = times and once calling
> >>> TCMPRPlotter
> >>> >> > with
> >>> >> > >> > >> LOOP_BY
> >>> >> > >> > >> > =
> >>> >> > >> > >> > >> processes.
> >>> >> > >> > >> > >>
> >>> >> > >> > >> > >> I am going to add a configuration option to
TCPairs
> >>> >> wrapper so
> >>> >> > >> that
> >>> >> > >> > >> you
> >>> >> > >> > >> > >> can loop over the run times when LOOP_BY =
processes
> is
> >>> >> set if
> >>> >> > >> > >> desired.
> >>> >> > >> > >> > I
> >>> >> > >> > >> > >> will try to get this change into METplus
4.1.0-beta1
> >>> so you
> >>> >> > can
> >>> >> > >> > test
> >>> >> > >> > >> the
> >>> >> > >> > >> > >> change and make sure it works as expected. I
can let
> >>> you
> >>> >> know
> >>> >> > >> when
> >>> >> > >> > >> that
> >>> >> > >> > >> > is
> >>> >> > >> > >> > >> available.
> >>> >> > >> > >> > >>
> >>> >> > >> > >> > >> Thanks,
> >>> >> > >> > >> > >> George
> >>> >> > >> > >> > >>
> >>> >> > >> > >> > >> On Tue, Jul 20, 2021 at 1:27 PM Guang Lou -
NOAA
> >>> Affiliate
> >>> >> via
> >>> >> > >> RT <
> >>> >> > >> > >> > >> met_help at ucar.edu> wrote:
> >>> >> > >> > >> > >>
> >>> >> > >> > >> > >>> Hi John,
> >>> >> > >> > >> > >>>
> >>> >> > >> > >> > >>> Thanks for forwarding George's email answer
to me. I
> >>> >> created
> >>> >> > a
> >>> >> > >> > test
> >>> >> > >> > >> > case
> >>> >> > >> > >> > >>> that does the tcpairs only on Hera:
> >>> >> > >> > >> > >>>
> >>> >> > /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_test3
> >>> >> > >> > >> > >>> Command line:
> >>> >> > >> > >> > >>> run_metplus.py -c
> >>> Plotter_fcstGFS_obsGFS_RPlotting_8.conf
> >>> >> -c
> >>> >> > >> > >> > >>> usecase_8.conf
> >>> >> > >> > >> > >>> where PROCESS_LIST = TCPairs
> >>> >> > >> > >> > >>>
> >>> >> > >> > >> > >>> It generated the month of 202002, that is
because I
> >>> >> changed
> >>> >> > >> > >> > >>> INIT_BEG = 20200101
> >>> >> > >> > >> > >>> INIT_END = 20200331
> >>> >> > >> > >> > >>> to
> >>> >> > >> > >> > >>> INIT_BEG = 20200201
> >>> >> > >> > >> > >>> INIT_END = 20200331
> >>> >> > >> > >> > >>>
> >>> >> > >> > >> > >>> If I use "INIT_BEG = 20200101,INIT_END =
20200331"
> it
> >>> >> will
> >>> >> > >> > generate
> >>> >> > >> > >> > >>> 20200101 only.
> >>> >> > >> > >> > >>>
> >>> >> > >> > >> > >>> The wild card does not work either for the
model's
> >>> place:
> >>> >> > >> > >> > >>> TC_PAIRS_ADECK_TEMPLATE =
> >>> >> > >> > >> > >>>
> >>> >> > >> >
> >>>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> >>> >> > >> > >> > >>> I populated adeck and bdeck on
> >>> >> > >> > >> > >>>
> >>> >> > >> > >> > >>>
> >>> >> > >> > >> >
> >>> >> > >> > >>
> >>> >> > >> >
> >>> >> > >>
> >>> >> >
> >>> >>
> >>>
> /scratch2/NCEPDEV/fv3-cam/noscrub/Guang.Ping.Lou/METplus/METplus-
4.0_sample_data/model_applications/medium_range/tracks_all
> >>> >> > >> > >> > >>> (each subdirectory contains all three months'
> decks.)
> >>> >> > >> > >> > >>>
> >>> >> > >> > >> > >>> I was wondering if you could copy my
directory,
> >>> >> > >> > >> > >>>
> >>> >> > /scratch2/NCEPDEV/fv3-
cam/noscrub/Guang.Ping.Lou/metplus_test3
> >>> >> > >> > >> > >>> and test:
> >>> >> > >> > >> > >>> 1. It can generate multiple months instead of
just
> one
> >>> >> month
> >>> >> > >> and
> >>> >> > >> > one
> >>> >> > >> > >> > >>> model.
> >>> >> > >> > >> > >>> 2. The wild card for models in the
DECK_TEMPATE for
> >>> >> multiple
> >>> >> > >> > models.
> >>> >> > >> > >> > >>>
> >>> >> > >> > >> > >>> George, I'm downloading NetCDF model data to
Hera:
> >>> >> > >> > >> > >>>
> >>> >> > >> >
> >>>
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071300_nc.tar
> >>> >> > >> > >> > >>>
> >>> >> > >> >
> >>>
/scratch2/NCEPDEV/stmp1/Guang.Ping.Lou/tracks/gfs.2021071306_nc.tar
> >>> >> > >> > >> > >>>
> >>> >> > >> > >> > >>> Please let me know after you downloaded the
files
> and
> >>> I
> >>> >> will
> >>> >> > >> have
> >>> >> > >> > to
> >>> >> > >> > >> > >>> remove
> >>> >> > >> > >> > >>> them due to tight disk space.
> >>> >> > >> > >> > >>>
> >>> >> > >> > >> > >>> Thanks,
> >>> >> > >> > >> > >>> Guang Ping
> >>> >> > >> > >> > >>>
> >>> >> > >> > >> > >>> On Mon, Jul 19, 2021 at 5:01 PM John Halley
Gotway
> >>> via RT
> >>> >> <
> >>> >> > >> > >> > >>> met_help at ucar.edu>
> >>> >> > >> > >> > >>> wrote:
> >>> >> > >> > >> > >>>
> >>> >> > >> > >> > >>> > Hi Guang Ping,
> >>> >> > >> > >> > >>> >
> >>> >> > >> > >> > >>> > The command that you ran here:
> >>> >> > >> > >> > >>> >
> >>> >> > >> > >> > >>> >
> >>> >> > >> > >> > >>> >
> >>> >> > >> > >> > >>>
> >>> >> > >> > >> >
> >>> >> > >> > >>
> >>> >> > >> >
> >>> >> > >>
> >>> >> >
> >>> >>
> >>>
>
/home/George.Mccabe/scratch1_George.Mccabe/METplus.feature_926/ush/run_metplus.py
> >>> >> > >> > >> > >>> > -c Plotter_fcstGFS_obsGFS_RPlotting_8.conf
-c
> >>> >> > Plotter_8.conf
> >>> >> > >> > >> > >>> >
> >>> >> > >> > >> > >>> > only runs TCMPRPlotter. This is because
> >>> PROCESS_LIST is
> >>> >> > >> defined
> >>> >> > >> > in
> >>> >> > >> > >> > both
> >>> >> > >> > >> > >>> > Plotter_fcstGFS_obsGFS_RPlotting_8.conf and
> >>> >> Plotter_8.conf.
> >>> >> > >> > Since
> >>> >> > >> > >> > >>> > Plotter_8.conf comes last, then the value
for
> >>> >> PROCESS_LIST
> >>> >> > >> from
> >>> >> > >> > >> that
> >>> >> > >> > >> > >>> file
> >>> >> > >> > >> > >>> > is used, which only has TCMPRPlotter
listed. The
> >>> >> plotter is
> >>> >> > >> > >> reading
> >>> >> > >> > >> > >>> all of
> >>> >> > >> > >> > >>> > the available files generated from TCPairs
and
> since
> >>> >> that
> >>> >> > >> > process
> >>> >> > >> > >> is
> >>> >> > >> > >> > >>> not
> >>> >> > >> > >> > >>> > running again with the new time range, the
data
> >>> that is
> >>> >> > read
> >>> >> > >> > only
> >>> >> > >> > >> > >>> covers
> >>> >> > >> > >> > >>> > the first month.
> >>> >> > >> > >> > >>> >
> >>> >> > >> > >> > >>> > We are planning on releasing METplus 4.1.0-
beta1
> >>> this
> >>> >> week.
> >>> >> > >> > >> However,
> >>> >> > >> > >> > >>> if I
> >>> >> > >> > >> > >>> > understand what you are trying to do here,
it
> >>> shouldn't
> >>> >> > >> require
> >>> >> > >> > >> the
> >>> >> > >> > >> > >>> changes
> >>> >> > >> > >> > >>> > described in Molly's issue. It looks like
you want
> >>> to
> >>> >> run
> >>> >> > >> > TCPairs
> >>> >> > >> > >> > once
> >>> >> > >> > >> > >>> and
> >>> >> > >> > >> > >>> > filter the data based on the init and valid
times.
> >>> While
> >>> >> > this
> >>> >> > >> > >> > >>> configuration
> >>> >> > >> > >> > >>> > should achieve what you are trying to do,
the
> >>> following
> >>> >> > >> > >> configuration
> >>> >> > >> > >> > >>> is
> >>> >> > >> > >> > >>> > more explicit to what you are actually
running:
> >>> >> > >> > >> > >>> >
> >>> >> > >> > >> > >>> > # run once using 20200101 to substitute the
init
> >>> time in
> >>> >> > your
> >>> >> > >> > >> string
> >>> >> > >> > >> > >>> > substitution, i.e. TC_PAIRS_ADECK_TEMPLATE
=
> >>> >> > >> > >> > >>> >
> >>> >> > >> > >>
> >>> >>
{date?fmt=%Y%m}/a{basin?fmt=%s}q{date?fmt=%Y%m}*.*.{cyclone?fmt=%s}
> >>> >> > >> > >> > >>> > INIT_BEG = 20200101
> >>> >> > >> > >> > >>> > INIT_END = 20200101
> >>> >> > >> > >> > >>> >
> >>> >> > >> > >> > >>> > # filter the init times of the lines read
from the
> >>> input
> >>> >> > >> files
> >>> >> > >> > in
> >>> >> > >> > >> the
> >>> >> > >> > >> > >>> > TCPairs configuration file
> >>> >> > >> > >> > >>> > TC_PAIRS_INIT_BEG = 20200101
> >>> >> > >> > >> > >>> > TC_PAIRS_INIT_END = 20200331
> >>> >> > >> > >> > >>> >
> >>> >> > >> > >> > >>> > # filter the valid times of the lines read
from
> the
> >>> >> input
> >>> >> > >> files
> >>> >> > >> > in
> >>> >> > >> > >> > the
> >>> >> > >> > >> > >>> > TCPairs configuration file
> >>> >> > >> > >> > >>> > TC_PAIRS_VALID_BEG = 20200101
> >>> >> > >> > >> > >>> > TC_PAIRS_VALID_END = 20200331
> >>> >> > >> > >> > >>> >
> >>> >> > >> > >> > >>> > If any of TC_PAIRS_[INIT/VALID]_[BEG/END]
are not
> >>> set,
> >>> >> then
> >>> >> > >> the
> >>> >> > >> > >> > >>> > corresponding [INIT/VALID]_[BEG/END] values
are
> >>> used.
> >>> >> This
> >>> >> > is
> >>> >> > >> > >> done to
> >>> >> > >> > >> > >>> > preserve backwards compatibility for old
METplus
> >>> >> > >> configurations.
> >>> >> > >> > >> The
> >>> >> > >> > >> > >>> above
> >>> >> > >> > >> > >>> > settings are more explicit as to what is
actually
> be
> >>> >> > >> configured.
> >>> >> > >> > >> > >>> >
> >>> >> > >> > >> > >>> > You may want to consider either clearing
your
> output
> >>> >> > >> directory
> >>> >> > >> > or
> >>> >> > >> > >> > >>> creating
> >>> >> > >> > >> > >>> > a new output directory for a new run
(especially
> if
> >>> the
> >>> >> > time
> >>> >> > >> > range
> >>> >> > >> > >> > you
> >>> >> > >> > >> > >>> are
> >>> >> > >> > >> > >>> > using has changed) to ensure that the files
you
> are
> >>> >> > generated
> >>> >> > >> > are
> >>> >> > >> > >> > from
> >>> >> > >> > >> > >>> that
> >>> >> > >> > >> > >>> > run and not from a previous run. Doing this
would
> >>> have
> >>> >> > >> alerted
> >>> >> > >> > you
> >>> >> > >> > >> > that
> >>> >> > >> > >> > >>> > TCPairs was not run in this case.
> >>> >> > >> > >> > >>> >
> >>> >> > >> > >> > >>> > Let me know if you have any other issues
getting
> >>> this
> >>> >> > >> running.
> >>> >> > >> > >> > >>> >
> >>> >> > >> > >> > >>> > FYI:
> >>> >> > >> > >> > >>> > We switched from providing support through
this
> >>> email
> >>> >> > >> address to
> >>> >> > >> > >> > >>> providing
> >>> >> > >> > >> > >>> > support through GitHub Discussions. In the
future,
> >>> you
> >>> >> will
> >>> >> > >> need
> >>> >> > >> > >> to
> >>> >> > >> > >> > >>> create
> >>> >> > >> > >> > >>> > a free GitHub account if you don’t have one
> already
> >>> and
> >>> >> > post
> >>> >> > >> > your
> >>> >> > >> > >> > >>> questions
> >>> >> > >> > >> > >>> > to the METplus Components Discussion Forum:
> >>> >> > >> > >> > >>> >
https://github.com/dtcenter/METplus/discussions
> >>> >> > >> > >> > >>> > To post a new question, click on the green
button
> >>> in the
> >>> >> > >> upper
> >>> >> > >> > >> right
> >>> >> > >> > >> > >>> corner
> >>> >> > >> > >> > >>> > of the page labeled "New discussion" and
select
> >>> >> "Incoming"
> >>> >> > as
> >>> >> > >> > the
> >>> >> > >> > >> > >>> category.
> >>> >> > >> > >> > >>> >
> >>> >> > >> > >> > >>> > Thanks,
> >>> >> > >> > >> > >>> > George
> >>> >> > >> > >> > >>> >
> >>> >> > >> > >> > >>> >
> >>> >> > >> > >> > >>>
> >>> >> > >> > >> > >>>
> >>> >> > >> > >> > >>
> >>> >> > >> > >> > >> --
> >>> >> > >> > >> > >> George McCabe - Software Engineer III
> >>> >> > >> > >> > >> National Center for Atmospheric Research
> >>> >> > >> > >> > >> Research Applications Laboratory
> >>> >> > >> > >> > >> 303-497-2768
> >>> >> > >> > >> > >> ---
> >>> >> > >> > >> > >> My working day may not be your working day.
Please do
> >>> not
> >>> >> feel
> >>> >> > >> > >> obliged
> >>> >> > >> > >> > to
> >>> >> > >> > >> > >> reply to this email outside of your normal
working
> >>> hours.
> >>> >> > >> > >> > >>
> >>> >> > >> > >> > >
> >>> >> > >> > >> > >
> >>> >> > >> > >> > > --
> >>> >> > >> > >> > > George McCabe - Software Engineer III
> >>> >> > >> > >> > > National Center for Atmospheric Research
> >>> >> > >> > >> > > Research Applications Laboratory
> >>> >> > >> > >> > > 303-497-2768
> >>> >> > >> > >> > > ---
> >>> >> > >> > >> > > My working day may not be your working day.
Please do
> >>> not
> >>> >> feel
> >>> >> > >> > >> obliged to
> >>> >> > >> > >> > > reply to this email outside of your normal
working
> >>> hours.
> >>> >> > >> > >> > >
> >>> >> > >> > >> >
> >>> >> > >> > >> >
> >>> >> > >> > >> > --
> >>> >> > >> > >> > George McCabe - Software Engineer III
> >>> >> > >> > >> > National Center for Atmospheric Research
> >>> >> > >> > >> > Research Applications Laboratory
> >>> >> > >> > >> > 303-497-2768
> >>> >> > >> > >> > ---
> >>> >> > >> > >> > My working day may not be your working day.
Please do
> not
> >>> feel
> >>> >> > >> obliged
> >>> >> > >> > >> to
> >>> >> > >> > >> > reply to this email outside of your normal
working
> hours.
> >>> >> > >> > >> >
> >>> >> > >> > >> >
> >>> >> > >> > >>
> >>> >> > >> > >>
> >>> >> > >> > >
> >>> >> > >> > > --
> >>> >> > >> > > George McCabe - Software Engineer III
> >>> >> > >> > > National Center for Atmospheric Research
> >>> >> > >> > > Research Applications Laboratory
> >>> >> > >> > > 303-497-2768
> >>> >> > >> > > ---
> >>> >> > >> > > My working day may not be your working day. Please
do not
> >>> feel
> >>> >> > >> obliged to
> >>> >> > >> > > reply to this email outside of your normal working
hours.
> >>> >> > >> > >
> >>> >> > >> >
> >>> >> > >> >
> >>> >> > >> > --
> >>> >> > >> > George McCabe - Software Engineer III
> >>> >> > >> > National Center for Atmospheric Research
> >>> >> > >> > Research Applications Laboratory
> >>> >> > >> > 303-497-2768
> >>> >> > >> > ---
> >>> >> > >> > My working day may not be your working day. Please do
not
> feel
> >>> >> obliged
> >>> >> > >> to
> >>> >> > >> > reply to this email outside of your normal working
hours.
> >>> >> > >> >
> >>> >> > >> >
> >>> >> > >>
> >>> >> > >>
> >>> >> > >
> >>> >> > > --
> >>> >> > > George McCabe - Software Engineer III
> >>> >> > > National Center for Atmospheric Research
> >>> >> > > Research Applications Laboratory
> >>> >> > > 303-497-2768
> >>> >> > > ---
> >>> >> > > My working day may not be your working day. Please do not
feel
> >>> >> obliged to
> >>> >> > > reply to this email outside of your normal working hours.
> >>> >> > >
> >>> >> >
> >>> >> >
> >>> >> > --
> >>> >> > George McCabe - Software Engineer III
> >>> >> > National Center for Atmospheric Research
> >>> >> > Research Applications Laboratory
> >>> >> > 303-497-2768
> >>> >> > ---
> >>> >> > My working day may not be your working day. Please do not
feel
> >>> obliged
> >>> >> to
> >>> >> > reply to this email outside of your normal working hours.
> >>> >> >
> >>> >> >
> >>> >>
> >>> >>
> >>> >
> >>> > --
> >>> > George McCabe - Software Engineer III
> >>> > National Center for Atmospheric Research
> >>> > Research Applications Laboratory
> >>> > 303-497-2768
> >>> > ---
> >>> > My working day may not be your working day. Please do not feel
> obliged
> >>> to
> >>> > reply to this email outside of your normal working hours.
> >>> >
> >>>
> >>>
> >>> --
> >>> George McCabe - Software Engineer III
> >>> National Center for Atmospheric Research
> >>> Research Applications Laboratory
> >>> 303-497-2768
> >>> ---
> >>> My working day may not be your working day. Please do not feel
obliged
> to
> >>> reply to this email outside of your normal working hours.
> >>>
> >>
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
More information about the Met_help
mailing list