[Met_help] [rt.rap.ucar.edu #95164] History for PB2NC_OFFSETS/related questions
George McCabe via RT
met_help at ucar.edu
Thu May 14 10:38:05 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Good morning,
I'm trying to configure a file to process prepbufr data to netcdf format.
There are several files that are within the directory that I want to
process as shown by the bolded text from the
/gpfs/dell1/nco/ops/com/hourly/prod/hourly.20200503 directory:
aqm.t12z.airnow.tm00.bufr_d aqm.t12z.anowpm.pb.tm120
aqm.t12z.status.tm024.bufr_d
aqm.t12z.airnow.tm024.bufr_d *aqm.t12z.prepbufr.tm00 *
aqm.t12z.status.tm120.bufr_d
aqm.t12z.airnow.tm120.bufr_d *aqm.t12z.prepbufr.tm024*
obsproc_version_for_t12z_aqm_dump_run
*aqm.t12z.anowpm.pb.tm00* aqm.t12z.prepbufr.tm120
where_t12z_aqm_dump_ran
*aqm.t12z.anowpm.pb.tm024 * aqm.t12z.status.tm00.bufr_d
Right now I'm focusing only on the *anowpm* files (two files). The tm00 is
time centered on 12z for 05/03/2020, with DHR = ranging from -11 to 12,
covering the whole day. Similarly, the tm024 file is centered at 12z, but
for 05/02/2020. I looked through the metplus glossary and examples to help
with the process of building the configure file. If I run this script and
desire the files from 05/03/2020, then I should set the following:
*[config]# time looping - options are INIT, VALID, RETRO, and
REALTIMELOOP_BY = VALID# Format of VALID_BEG and VALID_ENDVALID_TIME_FMT =
%Y%m%d%H# Start time, end time, and time increment for METplus runVALID_BEG
= {now?fmt=%Y%m%d%H?shift=-2d}VALID_END = {now?fmt=%Y%m%d%H?shift=-2d}
#-21600}VALID_INCREMENT = 24HLEAD_SEQ = 0,24 (((NOT SURE I NEED THIS,
BUT NOT SURE HOW THE LOOP WILL WORK WITHOUT IT?)))*
*LOOP_ORDER = times# List of applications to runPROCESS_LIST =
PB2NCCONFIG_DIR =
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/met_configCONFIG_FILE
= {CONFIG_DIR}/PB2NCConfig_wrappedPB2NC_OBS_BUFR_VAR_LIST = DHR, RPT, SID,
YOB, XOB, TYPO, TPHR, QCIND, COPO, COPOPM*
I've listed the variables I would like to extract from each file as well.
It's not clear if I need the config file, but I listed it just in case.
Here's what I have for the file names from the input directory and what I
would name it once converted to netcdf and placed in the output directory:
*INPUT_BASE=/gpfs/dell1/nco/ops/com/hourly/prod/hourly.{now?fmt=%Y%m%d?shift=-2d}
# -86400}PB2NC_INPUT_DIR = {INPUT_BASE}OUTPUT_BASE =
/gpfs/dell2/emc/modeling/noscrub/Edward.StrobachPB2NC_OUTPUT_DIR =
{OUTPUT_BASE}/AirNow/pb2nc/hourly.{now?fmt=%Y%m%d?shift=-2d} #
-86400}[filename_templates]PB2NC_INPUT_TEMPLATE =
aqm.t12z.anowpm.pb.tm0{LEAD_SEQ} (((RELATED TO LOOP
CONFUSION)))PB2NC_OUTPUT_TEMPLATE = aqm.t12z.anowpm.pb.tm0{LEAD_SEQ}.nc*
Let me know if more information is needed. My main concerns here are 1)
being able to process all the list files correctly, 2) making sure the
looping is correct, and 3) making sure that I've included everything that I
would need to process this step
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: PB2NC_OFFSETS/related questions
From: Edward Strobach - NOAA Affiliate
Time: Tue May 05 11:46:00 2020
Update:
Trying to follow the log files that are generated for guidance. I set
the
verbosity to 7, but still can't understand why I'm receiving this
message:
05/05 17:41:59.861 metplus INFO: Running METplus v3.0 called with
command:
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/master_metplus.py
-c
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/use_cases/model_applications/air_quality/CMAQ_obsAirnow.conf
05/05 17:41:59.882 metplus (met_util.py:776) ERROR:
[INIT/VALID]_TIME_FMT
(%Y%m%d%H) does not match [INIT/VALID]_[BEG/END] (2020050200
#2020050317)
05/05 17:41:59.886 metplus.PB2NC (met_util.py:192) ERROR: Fatal error
occurred
I tried to start basic. If starting basic worked, then I would have
made
it more complex:
# time looping - options are INIT, VALID, RETRO, and REALTIME
LOG_MET_VERBOSITY = 7
LOOP_BY = INIT
# Format of VALID_BEG and VALID_END
INIT_TIME_FMT = %Y%m%d%H
# Start time, end time, and time increment for METplus run
INIT_BEG = 2020050200 #{now?fmt=%Y%m%d%H?shift=-172800}
INIT_END = 2020050200 #{now?fmt=%Y%m%d%H?shift=-172800} #-21600}
INIT_INCREMENT = 0
LEAD_SEQ = 0
With how this looks, I would have thought that only one file would be
processed. But it complains about a mismatch between the
INIT_TIME_FMT and
INIT_BEG/INIT_END
On Tue, May 5, 2020 at 12:01 PM met_help at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> Greetings,
>
> This message has been automatically generated in response to the
creation
> of a trouble ticket regarding:
> "PB2NC_OFFSETS/related questions",
> a summary of which appears below.
>
> There is no need to reply to this message right now. Your ticket
has been
> assigned an ID of [rt.rap.ucar.edu #95164].
>
> Please include the string:
>
> [rt.rap.ucar.edu #95164]
>
> in the subject line of all future correspondence about this issue.
To do
> so, you may reply to this message.
>
> For more information, please see:
>
> MET Online Tutorial:
>
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
>
> MET Users Guide:
> https://www.dtcenter.org/met/users/docs/overview.php
>
> MET FAQs:
> https://www.dtcenter.org/met/users/support/faqs/index.php
>
> MET-Help Email Archive:
> http://mailman.ucar.edu/pipermail/met_help
>
> Thank you,
> met_help at ucar.edu
>
>
-------------------------------------------------------------------------
> Good morning,
>
> I'm trying to configure a file to process prepbufr data to netcdf
format.
> There are several files that are within the directory that I want to
> process as shown by the bolded text from the
> /gpfs/dell1/nco/ops/com/hourly/prod/hourly.20200503 directory:
>
> aqm.t12z.airnow.tm00.bufr_d aqm.t12z.anowpm.pb.tm120
> aqm.t12z.status.tm024.bufr_d
> aqm.t12z.airnow.tm024.bufr_d *aqm.t12z.prepbufr.tm00 *
> aqm.t12z.status.tm120.bufr_d
> aqm.t12z.airnow.tm120.bufr_d *aqm.t12z.prepbufr.tm024*
> obsproc_version_for_t12z_aqm_dump_run
> *aqm.t12z.anowpm.pb.tm00* aqm.t12z.prepbufr.tm120
> where_t12z_aqm_dump_ran
> *aqm.t12z.anowpm.pb.tm024 * aqm.t12z.status.tm00.bufr_d
>
> Right now I'm focusing only on the *anowpm* files (two files). The
tm00 is
> time centered on 12z for 05/03/2020, with DHR = ranging from -11 to
12,
> covering the whole day. Similarly, the tm024 file is centered at
12z, but
> for 05/02/2020. I looked through the metplus glossary and examples
to help
> with the process of building the configure file. If I run this
script and
> desire the files from 05/03/2020, then I should set the following:
>
>
>
>
>
>
>
>
>
>
>
>
> *[config]# time looping - options are INIT, VALID, RETRO, and
> REALTIMELOOP_BY = VALID# Format of VALID_BEG and
VALID_ENDVALID_TIME_FMT =
> %Y%m%d%H# Start time, end time, and time increment for METplus
runVALID_BEG
> = {now?fmt=%Y%m%d%H?shift=-2d}VALID_END = {now?fmt=%Y%m%d%H?shift=-
2d}
> #-21600}VALID_INCREMENT = 24HLEAD_SEQ = 0,24 (((NOT SURE I NEED
THIS,
> BUT NOT SURE HOW THE LOOP WILL WORK WITHOUT IT?)))*
>
>
>
>
>
>
>
>
> *LOOP_ORDER = times# List of applications to runPROCESS_LIST =
> PB2NCCONFIG_DIR =
>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/met_configCONFIG_FILE
> = {CONFIG_DIR}/PB2NCConfig_wrappedPB2NC_OBS_BUFR_VAR_LIST = DHR,
RPT, SID,
> YOB, XOB, TYPO, TPHR, QCIND, COPO, COPOPM*
>
> I've listed the variables I would like to extract from each file as
well.
> It's not clear if I need the config file, but I listed it just in
case.
> Here's what I have for the file names from the input directory and
what I
> would name it once converted to netcdf and placed in the output
directory:
>
>
>
>
>
>
>
>
>
>
>
*INPUT_BASE=/gpfs/dell1/nco/ops/com/hourly/prod/hourly.{now?fmt=%Y%m%d?shift=-
2d}
> # -86400}PB2NC_INPUT_DIR = {INPUT_BASE}OUTPUT_BASE =
> /gpfs/dell2/emc/modeling/noscrub/Edward.StrobachPB2NC_OUTPUT_DIR =
> {OUTPUT_BASE}/AirNow/pb2nc/hourly.{now?fmt=%Y%m%d?shift=-2d} #
> -86400}[filename_templates]PB2NC_INPUT_TEMPLATE =
> aqm.t12z.anowpm.pb.tm0{LEAD_SEQ} (((RELATED TO LOOP
> CONFUSION)))PB2NC_OUTPUT_TEMPLATE =
aqm.t12z.anowpm.pb.tm0{LEAD_SEQ}.nc*
>
> Let me know if more information is needed. My main concerns here
are 1)
> being able to process all the list files correctly, 2) making sure
the
> looping is correct, and 3) making sure that I've included everything
that I
> would need to process this step
>
>
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: PB2NC_OFFSETS/related questions
From: Julie Prestopnik
Time: Tue May 05 12:17:07 2020
Hi Ed.
I have assigned this ticket to George. Please allow a few business
days
for a response. Thank you!
Julie
On Tue, May 5, 2020 at 11:46 AM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
>
> Update:
>
> Trying to follow the log files that are generated for guidance. I
set the
> verbosity to 7, but still can't understand why I'm receiving this
message:
>
> 05/05 17:41:59.861 metplus INFO: Running METplus v3.0 called with
command:
>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/master_metplus.py
> -c
>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/use_cases/model_applications/air_quality/CMAQ_obsAirnow.conf
> 05/05 17:41:59.882 metplus (met_util.py:776) ERROR:
[INIT/VALID]_TIME_FMT
> (%Y%m%d%H) does not match [INIT/VALID]_[BEG/END] (2020050200
#2020050317)
> 05/05 17:41:59.886 metplus.PB2NC (met_util.py:192) ERROR: Fatal
error
> occurred
>
> I tried to start basic. If starting basic worked, then I would have
made
> it more complex:
>
> # time looping - options are INIT, VALID, RETRO, and REALTIME
> LOG_MET_VERBOSITY = 7
>
> LOOP_BY = INIT
>
> # Format of VALID_BEG and VALID_END
> INIT_TIME_FMT = %Y%m%d%H
>
> # Start time, end time, and time increment for METplus run
> INIT_BEG = 2020050200 #{now?fmt=%Y%m%d%H?shift=-172800}
> INIT_END = 2020050200 #{now?fmt=%Y%m%d%H?shift=-172800} #-21600}
> INIT_INCREMENT = 0
> LEAD_SEQ = 0
>
> With how this looks, I would have thought that only one file would
be
> processed. But it complains about a mismatch between the
INIT_TIME_FMT and
> INIT_BEG/INIT_END
>
> On Tue, May 5, 2020 at 12:01 PM met_help at ucar.edu via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > Greetings,
> >
> > This message has been automatically generated in response to the
creation
> > of a trouble ticket regarding:
> > "PB2NC_OFFSETS/related questions",
> > a summary of which appears below.
> >
> > There is no need to reply to this message right now. Your ticket
has
> been
> > assigned an ID of [rt.rap.ucar.edu #95164].
> >
> > Please include the string:
> >
> > [rt.rap.ucar.edu #95164]
> >
> > in the subject line of all future correspondence about this issue.
To do
> > so, you may reply to this message.
> >
> > For more information, please see:
> >
> > MET Online Tutorial:
> >
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> >
> > MET Users Guide:
> > https://www.dtcenter.org/met/users/docs/overview.php
> >
> > MET FAQs:
> > https://www.dtcenter.org/met/users/support/faqs/index.php
> >
> > MET-Help Email Archive:
> > http://mailman.ucar.edu/pipermail/met_help
> >
> > Thank you,
> > met_help at ucar.edu
> >
> >
-------------------------------------------------------------------------
> > Good morning,
> >
> > I'm trying to configure a file to process prepbufr data to netcdf
format.
> > There are several files that are within the directory that I want
to
> > process as shown by the bolded text from the
> > /gpfs/dell1/nco/ops/com/hourly/prod/hourly.20200503 directory:
> >
> > aqm.t12z.airnow.tm00.bufr_d aqm.t12z.anowpm.pb.tm120
> > aqm.t12z.status.tm024.bufr_d
> > aqm.t12z.airnow.tm024.bufr_d *aqm.t12z.prepbufr.tm00 *
> > aqm.t12z.status.tm120.bufr_d
> > aqm.t12z.airnow.tm120.bufr_d *aqm.t12z.prepbufr.tm024*
> > obsproc_version_for_t12z_aqm_dump_run
> > *aqm.t12z.anowpm.pb.tm00* aqm.t12z.prepbufr.tm120
> > where_t12z_aqm_dump_ran
> > *aqm.t12z.anowpm.pb.tm024 * aqm.t12z.status.tm00.bufr_d
> >
> > Right now I'm focusing only on the *anowpm* files (two files).
The tm00
> is
> > time centered on 12z for 05/03/2020, with DHR = ranging from -11
to 12,
> > covering the whole day. Similarly, the tm024 file is centered at
12z,
> but
> > for 05/02/2020. I looked through the metplus glossary and
examples to
> help
> > with the process of building the configure file. If I run this
script
> and
> > desire the files from 05/03/2020, then I should set the following:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *[config]# time looping - options are INIT, VALID, RETRO, and
> > REALTIMELOOP_BY = VALID# Format of VALID_BEG and
VALID_ENDVALID_TIME_FMT
> =
> > %Y%m%d%H# Start time, end time, and time increment for METplus
> runVALID_BEG
> > = {now?fmt=%Y%m%d%H?shift=-2d}VALID_END =
{now?fmt=%Y%m%d%H?shift=-2d}
> > #-21600}VALID_INCREMENT = 24HLEAD_SEQ = 0,24 (((NOT SURE I NEED
THIS,
> > BUT NOT SURE HOW THE LOOP WILL WORK WITHOUT IT?)))*
> >
> >
> >
> >
> >
> >
> >
> >
> > *LOOP_ORDER = times# List of applications to runPROCESS_LIST =
> > PB2NCCONFIG_DIR =
> >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/met_configCONFIG_FILE
> > = {CONFIG_DIR}/PB2NCConfig_wrappedPB2NC_OBS_BUFR_VAR_LIST = DHR,
RPT,
> SID,
> > YOB, XOB, TYPO, TPHR, QCIND, COPO, COPOPM*
> >
> > I've listed the variables I would like to extract from each file
as well.
> > It's not clear if I need the config file, but I listed it just in
case.
> > Here's what I have for the file names from the input directory and
what I
> > would name it once converted to netcdf and placed in the output
> directory:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
*INPUT_BASE=/gpfs/dell1/nco/ops/com/hourly/prod/hourly.{now?fmt=%Y%m%d?shift=-
2d}
> > # -86400}PB2NC_INPUT_DIR = {INPUT_BASE}OUTPUT_BASE =
> > /gpfs/dell2/emc/modeling/noscrub/Edward.StrobachPB2NC_OUTPUT_DIR =
> > {OUTPUT_BASE}/AirNow/pb2nc/hourly.{now?fmt=%Y%m%d?shift=-2d} #
> > -86400}[filename_templates]PB2NC_INPUT_TEMPLATE =
> > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ} (((RELATED TO LOOP
> > CONFUSION)))PB2NC_OUTPUT_TEMPLATE =
aqm.t12z.anowpm.pb.tm0{LEAD_SEQ}.nc*
> >
> > Let me know if more information is needed. My main concerns here
are 1)
> > being able to process all the list files correctly, 2) making sure
the
> > looping is correct, and 3) making sure that I've included
everything
> that I
> > would need to process this step
> >
> >
> >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
--
Julie Prestopnik
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: PB2NC_OFFSETS/related questions
From: George McCabe
Time: Tue May 05 12:36:58 2020
Hi Edward,
1) The LOG_MET_VERBOSITY variable sets the verbosity level when a MET
tool
is run. To change the METplus logging level, set LOG_LEVEL. I would
recommend changing it to DEBUG to get more information.
2) There is a bug in using the {now} tags for [INIT/VALID]_[BEG/END]
that
causes that crash. This has been fixed but we have not yet released a
bugfix release. I will let the engineers know that you need this
functionality and we will push the METplus 3.0.1 bugfix release soon.
You
should be able to set the time explicitly, i.e. 2020050500.
3) The {now} tag can be used in the VALID_BEG and VALID_END config
variables, but you will want to refer to init, valid, lead, etc. in
your
INPUT_TEMPLATE and OUTPUT_TEMPLATE variables so that it uses the time
for
each time loop. See this section for more information:
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#directory-
and-filename-template-info
4) The LEAD_SEQ is a list of forecast leads that will be processed for
each
run time. If you set this to 0 or unset it, then it will just process
once
for each run time.
5) There is documentation describing the time looping variables here:
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#common-
config-variables
This page should answer some of your questions on how this works.
6) I would recommend starting with the Example use case found in
parm/use_cases/met_tool_wrapper/Example/Example.conf. This use case
loops
over times, fills in the filename templates based on each run time,
and
prints information to the logfile. You can copy this use case and
modify it
to see which filenames are searched for each run time. This is a good
way
to understand how to configure METplus to find the files you need.
Once
this makes sense, you can look at the PB2NC.conf use case to see how
the
config variables control that wrapper.
7) METplus sets environment variables based on its configuration
files.
These environment variables are referenced in the MET config file to
set
configurations in that file. If you turn on DEBUG logging, you can see
what
values are set for these environment variables to get a better idea of
how
your METplus configuration will affect the MET configuration.
Please let me know if you have any questions regarding any of this.
On Tue, May 5, 2020 at 11:46 AM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
>
> Update:
>
> Trying to follow the log files that are generated for guidance. I
set the
> verbosity to 7, but still can't understand why I'm receiving this
message:
>
> 05/05 17:41:59.861 metplus INFO: Running METplus v3.0 called with
command:
>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/master_metplus.py
> -c
>
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/use_cases/model_applications/air_quality/CMAQ_obsAirnow.conf
> 05/05 17:41:59.882 metplus (met_util.py:776) ERROR:
[INIT/VALID]_TIME_FMT
> (%Y%m%d%H) does not match [INIT/VALID]_[BEG/END] (2020050200
#2020050317)
> 05/05 17:41:59.886 metplus.PB2NC (met_util.py:192) ERROR: Fatal
error
> occurred
>
> I tried to start basic. If starting basic worked, then I would have
made
> it more complex:
>
> # time looping - options are INIT, VALID, RETRO, and REALTIME
> LOG_MET_VERBOSITY = 7
>
> LOOP_BY = INIT
>
> # Format of VALID_BEG and VALID_END
> INIT_TIME_FMT = %Y%m%d%H
>
> # Start time, end time, and time increment for METplus run
> INIT_BEG = 2020050200 #{now?fmt=%Y%m%d%H?shift=-172800}
> INIT_END = 2020050200 #{now?fmt=%Y%m%d%H?shift=-172800} #-21600}
> INIT_INCREMENT = 0
> LEAD_SEQ = 0
>
> With how this looks, I would have thought that only one file would
be
> processed. But it complains about a mismatch between the
INIT_TIME_FMT and
> INIT_BEG/INIT_END
>
> On Tue, May 5, 2020 at 12:01 PM met_help at ucar.edu via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > Greetings,
> >
> > This message has been automatically generated in response to the
creation
> > of a trouble ticket regarding:
> > "PB2NC_OFFSETS/related questions",
> > a summary of which appears below.
> >
> > There is no need to reply to this message right now. Your ticket
has
> been
> > assigned an ID of [rt.rap.ucar.edu #95164].
> >
> > Please include the string:
> >
> > [rt.rap.ucar.edu #95164]
> >
> > in the subject line of all future correspondence about this issue.
To do
> > so, you may reply to this message.
> >
> > For more information, please see:
> >
> > MET Online Tutorial:
> >
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> >
> > MET Users Guide:
> > https://www.dtcenter.org/met/users/docs/overview.php
> >
> > MET FAQs:
> > https://www.dtcenter.org/met/users/support/faqs/index.php
> >
> > MET-Help Email Archive:
> > http://mailman.ucar.edu/pipermail/met_help
> >
> > Thank you,
> > met_help at ucar.edu
> >
> >
-------------------------------------------------------------------------
> > Good morning,
> >
> > I'm trying to configure a file to process prepbufr data to netcdf
format.
> > There are several files that are within the directory that I want
to
> > process as shown by the bolded text from the
> > /gpfs/dell1/nco/ops/com/hourly/prod/hourly.20200503 directory:
> >
> > aqm.t12z.airnow.tm00.bufr_d aqm.t12z.anowpm.pb.tm120
> > aqm.t12z.status.tm024.bufr_d
> > aqm.t12z.airnow.tm024.bufr_d *aqm.t12z.prepbufr.tm00 *
> > aqm.t12z.status.tm120.bufr_d
> > aqm.t12z.airnow.tm120.bufr_d *aqm.t12z.prepbufr.tm024*
> > obsproc_version_for_t12z_aqm_dump_run
> > *aqm.t12z.anowpm.pb.tm00* aqm.t12z.prepbufr.tm120
> > where_t12z_aqm_dump_ran
> > *aqm.t12z.anowpm.pb.tm024 * aqm.t12z.status.tm00.bufr_d
> >
> > Right now I'm focusing only on the *anowpm* files (two files).
The tm00
> is
> > time centered on 12z for 05/03/2020, with DHR = ranging from -11
to 12,
> > covering the whole day. Similarly, the tm024 file is centered at
12z,
> but
> > for 05/02/2020. I looked through the metplus glossary and
examples to
> help
> > with the process of building the configure file. If I run this
script
> and
> > desire the files from 05/03/2020, then I should set the following:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *[config]# time looping - options are INIT, VALID, RETRO, and
> > REALTIMELOOP_BY = VALID# Format of VALID_BEG and
VALID_ENDVALID_TIME_FMT
> =
> > %Y%m%d%H# Start time, end time, and time increment for METplus
> runVALID_BEG
> > = {now?fmt=%Y%m%d%H?shift=-2d}VALID_END =
{now?fmt=%Y%m%d%H?shift=-2d}
> > #-21600}VALID_INCREMENT = 24HLEAD_SEQ = 0,24 (((NOT SURE I NEED
THIS,
> > BUT NOT SURE HOW THE LOOP WILL WORK WITHOUT IT?)))*
> >
> >
> >
> >
> >
> >
> >
> >
> > *LOOP_ORDER = times# List of applications to runPROCESS_LIST =
> > PB2NCCONFIG_DIR =
> >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/met_configCONFIG_FILE
> > = {CONFIG_DIR}/PB2NCConfig_wrappedPB2NC_OBS_BUFR_VAR_LIST = DHR,
RPT,
> SID,
> > YOB, XOB, TYPO, TPHR, QCIND, COPO, COPOPM*
> >
> > I've listed the variables I would like to extract from each file
as well.
> > It's not clear if I need the config file, but I listed it just in
case.
> > Here's what I have for the file names from the input directory and
what I
> > would name it once converted to netcdf and placed in the output
> directory:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
*INPUT_BASE=/gpfs/dell1/nco/ops/com/hourly/prod/hourly.{now?fmt=%Y%m%d?shift=-
2d}
> > # -86400}PB2NC_INPUT_DIR = {INPUT_BASE}OUTPUT_BASE =
> > /gpfs/dell2/emc/modeling/noscrub/Edward.StrobachPB2NC_OUTPUT_DIR =
> > {OUTPUT_BASE}/AirNow/pb2nc/hourly.{now?fmt=%Y%m%d?shift=-2d} #
> > -86400}[filename_templates]PB2NC_INPUT_TEMPLATE =
> > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ} (((RELATED TO LOOP
> > CONFUSION)))PB2NC_OUTPUT_TEMPLATE =
aqm.t12z.anowpm.pb.tm0{LEAD_SEQ}.nc*
> >
> > Let me know if more information is needed. My main concerns here
are 1)
> > being able to process all the list files correctly, 2) making sure
the
> > looping is correct, and 3) making sure that I've included
everything
> that I
> > would need to process this step
> >
> >
> >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: PB2NC_OFFSETS/related questions
From: Edward Strobach - NOAA Affiliate
Time: Wed May 06 12:43:21 2020
Thanks George. I've been busy trying to figure things out and forgot
to
respond. I talked with Perry about some of these things and he was
surprised that I was receiving this message when running his script:
05/06 17:34:09.517 metplus.PointStat INFO: Processing forecast lead 0
hours
05/06 17:34:09.525 metplus.PointStat (command_builder.py:407) DEBUG:
Looking for FCST file
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
05/06 17:34:09.831 metplus.PointStat (command_builder.py:100) ERROR:
(command_builder.py:437) Could not find FCST file
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/
*aqm.t00z.awpozcon.f00.148.grib2* using template
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
05/06 17:34:09.832 metplus.PointStat (met_util.py:192) ERROR: Fatal
error
occurred
The bolded item does not exist. When Perry runs his script it skips
this
file. When I run the script I get a fatal error. I cloned metplus
just
the other day. I'm not sure if this is related to item 2 from your
email
On Tue, May 5, 2020 at 2:37 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Edward,
>
> 1) The LOG_MET_VERBOSITY variable sets the verbosity level when a
MET tool
> is run. To change the METplus logging level, set LOG_LEVEL. I would
> recommend changing it to DEBUG to get more information.
>
> 2) There is a bug in using the {now} tags for [INIT/VALID]_[BEG/END]
that
> causes that crash. This has been fixed but we have not yet released
a
> bugfix release. I will let the engineers know that you need this
> functionality and we will push the METplus 3.0.1 bugfix release
soon. You
> should be able to set the time explicitly, i.e. 2020050500.
>
> 3) The {now} tag can be used in the VALID_BEG and VALID_END config
> variables, but you will want to refer to init, valid, lead, etc. in
your
> INPUT_TEMPLATE and OUTPUT_TEMPLATE variables so that it uses the
time for
> each time loop. See this section for more information:
>
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#directory-
and-filename-template-info
>
> 4) The LEAD_SEQ is a list of forecast leads that will be processed
for each
> run time. If you set this to 0 or unset it, then it will just
process once
> for each run time.
>
> 5) There is documentation describing the time looping variables
here:
>
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#common-
config-variables
>
> This page should answer some of your questions on how this works.
>
> 6) I would recommend starting with the Example use case found in
> parm/use_cases/met_tool_wrapper/Example/Example.conf. This use case
loops
> over times, fills in the filename templates based on each run time,
and
> prints information to the logfile. You can copy this use case and
modify it
> to see which filenames are searched for each run time. This is a
good way
> to understand how to configure METplus to find the files you need.
Once
> this makes sense, you can look at the PB2NC.conf use case to see how
the
> config variables control that wrapper.
>
> 7) METplus sets environment variables based on its configuration
files.
> These environment variables are referenced in the MET config file to
set
> configurations in that file. If you turn on DEBUG logging, you can
see what
> values are set for these environment variables to get a better idea
of how
> your METplus configuration will affect the MET configuration.
>
> Please let me know if you have any questions regarding any of this.
>
> On Tue, May 5, 2020 at 11:46 AM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> >
> > Update:
> >
> > Trying to follow the log files that are generated for guidance. I
set
> the
> > verbosity to 7, but still can't understand why I'm receiving this
> message:
> >
> > 05/05 17:41:59.861 metplus INFO: Running METplus v3.0 called with
> command:
> >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/master_metplus.py
> > -c
> >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/use_cases/model_applications/air_quality/CMAQ_obsAirnow.conf
> > 05/05 17:41:59.882 metplus (met_util.py:776) ERROR:
[INIT/VALID]_TIME_FMT
> > (%Y%m%d%H) does not match [INIT/VALID]_[BEG/END] (2020050200
> #2020050317)
> > 05/05 17:41:59.886 metplus.PB2NC (met_util.py:192) ERROR: Fatal
error
> > occurred
> >
> > I tried to start basic. If starting basic worked, then I would
have made
> > it more complex:
> >
> > # time looping - options are INIT, VALID, RETRO, and REALTIME
> > LOG_MET_VERBOSITY = 7
> >
> > LOOP_BY = INIT
> >
> > # Format of VALID_BEG and VALID_END
> > INIT_TIME_FMT = %Y%m%d%H
> >
> > # Start time, end time, and time increment for METplus run
> > INIT_BEG = 2020050200 #{now?fmt=%Y%m%d%H?shift=-172800}
> > INIT_END = 2020050200 #{now?fmt=%Y%m%d%H?shift=-172800} #-21600}
> > INIT_INCREMENT = 0
> > LEAD_SEQ = 0
> >
> > With how this looks, I would have thought that only one file would
be
> > processed. But it complains about a mismatch between the
INIT_TIME_FMT
> and
> > INIT_BEG/INIT_END
> >
> > On Tue, May 5, 2020 at 12:01 PM met_help at ucar.edu via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > >
> > > Greetings,
> > >
> > > This message has been automatically generated in response to the
> creation
> > > of a trouble ticket regarding:
> > > "PB2NC_OFFSETS/related questions",
> > > a summary of which appears below.
> > >
> > > There is no need to reply to this message right now. Your
ticket has
> > been
> > > assigned an ID of [rt.rap.ucar.edu #95164].
> > >
> > > Please include the string:
> > >
> > > [rt.rap.ucar.edu #95164]
> > >
> > > in the subject line of all future correspondence about this
issue. To
> do
> > > so, you may reply to this message.
> > >
> > > For more information, please see:
> > >
> > > MET Online Tutorial:
> > >
> https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > >
> > > MET Users Guide:
> > > https://www.dtcenter.org/met/users/docs/overview.php
> > >
> > > MET FAQs:
> > > https://www.dtcenter.org/met/users/support/faqs/index.php
> > >
> > > MET-Help Email Archive:
> > > http://mailman.ucar.edu/pipermail/met_help
> > >
> > > Thank you,
> > > met_help at ucar.edu
> > >
> > >
>
-------------------------------------------------------------------------
> > > Good morning,
> > >
> > > I'm trying to configure a file to process prepbufr data to
netcdf
> format.
> > > There are several files that are within the directory that I
want to
> > > process as shown by the bolded text from the
> > > /gpfs/dell1/nco/ops/com/hourly/prod/hourly.20200503 directory:
> > >
> > > aqm.t12z.airnow.tm00.bufr_d aqm.t12z.anowpm.pb.tm120
> > > aqm.t12z.status.tm024.bufr_d
> > > aqm.t12z.airnow.tm024.bufr_d *aqm.t12z.prepbufr.tm00 *
> > > aqm.t12z.status.tm120.bufr_d
> > > aqm.t12z.airnow.tm120.bufr_d *aqm.t12z.prepbufr.tm024*
> > > obsproc_version_for_t12z_aqm_dump_run
> > > *aqm.t12z.anowpm.pb.tm00* aqm.t12z.prepbufr.tm120
> > > where_t12z_aqm_dump_ran
> > > *aqm.t12z.anowpm.pb.tm024 * aqm.t12z.status.tm00.bufr_d
> > >
> > > Right now I'm focusing only on the *anowpm* files (two files).
The
> tm00
> > is
> > > time centered on 12z for 05/03/2020, with DHR = ranging from -11
to 12,
> > > covering the whole day. Similarly, the tm024 file is centered
at 12z,
> > but
> > > for 05/02/2020. I looked through the metplus glossary and
examples to
> > help
> > > with the process of building the configure file. If I run this
script
> > and
> > > desire the files from 05/03/2020, then I should set the
following:
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *[config]# time looping - options are INIT, VALID, RETRO, and
> > > REALTIMELOOP_BY = VALID# Format of VALID_BEG and
> VALID_ENDVALID_TIME_FMT
> > =
> > > %Y%m%d%H# Start time, end time, and time increment for METplus
> > runVALID_BEG
> > > = {now?fmt=%Y%m%d%H?shift=-2d}VALID_END =
{now?fmt=%Y%m%d%H?shift=-2d}
> > > #-21600}VALID_INCREMENT = 24HLEAD_SEQ = 0,24 (((NOT SURE I
NEED
> THIS,
> > > BUT NOT SURE HOW THE LOOP WILL WORK WITHOUT IT?)))*
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *LOOP_ORDER = times# List of applications to runPROCESS_LIST =
> > > PB2NCCONFIG_DIR =
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/met_configCONFIG_FILE
> > > = {CONFIG_DIR}/PB2NCConfig_wrappedPB2NC_OBS_BUFR_VAR_LIST = DHR,
RPT,
> > SID,
> > > YOB, XOB, TYPO, TPHR, QCIND, COPO, COPOPM*
> > >
> > > I've listed the variables I would like to extract from each file
as
> well.
> > > It's not clear if I need the config file, but I listed it just
in case.
> > > Here's what I have for the file names from the input directory
and
> what I
> > > would name it once converted to netcdf and placed in the output
> > directory:
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
>
*INPUT_BASE=/gpfs/dell1/nco/ops/com/hourly/prod/hourly.{now?fmt=%Y%m%d?shift=-
2d}
> > > # -86400}PB2NC_INPUT_DIR = {INPUT_BASE}OUTPUT_BASE =
> > > /gpfs/dell2/emc/modeling/noscrub/Edward.StrobachPB2NC_OUTPUT_DIR
=
> > > {OUTPUT_BASE}/AirNow/pb2nc/hourly.{now?fmt=%Y%m%d?shift=-2d} #
> > > -86400}[filename_templates]PB2NC_INPUT_TEMPLATE =
> > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ} (((RELATED TO LOOP
> > > CONFUSION)))PB2NC_OUTPUT_TEMPLATE =
> aqm.t12z.anowpm.pb.tm0{LEAD_SEQ}.nc*
> > >
> > > Let me know if more information is needed. My main concerns
here are
> 1)
> > > being able to process all the list files correctly, 2) making
sure the
> > > looping is correct, and 3) making sure that I've included
everything
> > that I
> > > would need to process this step
> > >
> > >
> > >
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> > >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: PB2NC_OFFSETS/related questions
From: George McCabe
Time: Wed May 06 12:55:01 2020
Hi Edward,
When you cloned the repo yourself, are you using the develop branch to
run?
There may have been a bug introduced that is causing that fatal error.
Could you send me the stack trace that comes after the fatal error? I
agree
that it shouldn't crash if it can't find a file.
On Wed, May 6, 2020 at 12:43 PM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
>
> Thanks George. I've been busy trying to figure things out and
forgot to
> respond. I talked with Perry about some of these things and he was
> surprised that I was receiving this message when running his script:
>
> 05/06 17:34:09.517 metplus.PointStat INFO: Processing forecast lead
0 hours
> 05/06 17:34:09.525 metplus.PointStat (command_builder.py:407) DEBUG:
> Looking for FCST file
>
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
> 05/06 17:34:09.831 metplus.PointStat (command_builder.py:100) ERROR:
> (command_builder.py:437) Could not find FCST file
> /gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/
> *aqm.t00z.awpozcon.f00.148.grib2* using template
>
>
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
> 05/06 17:34:09.832 metplus.PointStat (met_util.py:192) ERROR: Fatal
error
> occurred
>
> The bolded item does not exist. When Perry runs his script it skips
this
> file. When I run the script I get a fatal error. I cloned metplus
just
> the other day. I'm not sure if this is related to item 2 from your
email
>
> On Tue, May 5, 2020 at 2:37 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Edward,
> >
> > 1) The LOG_MET_VERBOSITY variable sets the verbosity level when a
MET
> tool
> > is run. To change the METplus logging level, set LOG_LEVEL. I
would
> > recommend changing it to DEBUG to get more information.
> >
> > 2) There is a bug in using the {now} tags for
[INIT/VALID]_[BEG/END] that
> > causes that crash. This has been fixed but we have not yet
released a
> > bugfix release. I will let the engineers know that you need this
> > functionality and we will push the METplus 3.0.1 bugfix release
soon. You
> > should be able to set the time explicitly, i.e. 2020050500.
> >
> > 3) The {now} tag can be used in the VALID_BEG and VALID_END config
> > variables, but you will want to refer to init, valid, lead, etc.
in your
> > INPUT_TEMPLATE and OUTPUT_TEMPLATE variables so that it uses the
time for
> > each time loop. See this section for more information:
> >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#directory-
and-filename-template-info
> >
> > 4) The LEAD_SEQ is a list of forecast leads that will be processed
for
> each
> > run time. If you set this to 0 or unset it, then it will just
process
> once
> > for each run time.
> >
> > 5) There is documentation describing the time looping variables
here:
> >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#common-
config-variables
> >
> > This page should answer some of your questions on how this works.
> >
> > 6) I would recommend starting with the Example use case found in
> > parm/use_cases/met_tool_wrapper/Example/Example.conf. This use
case loops
> > over times, fills in the filename templates based on each run
time, and
> > prints information to the logfile. You can copy this use case and
modify
> it
> > to see which filenames are searched for each run time. This is a
good way
> > to understand how to configure METplus to find the files you need.
Once
> > this makes sense, you can look at the PB2NC.conf use case to see
how the
> > config variables control that wrapper.
> >
> > 7) METplus sets environment variables based on its configuration
files.
> > These environment variables are referenced in the MET config file
to set
> > configurations in that file. If you turn on DEBUG logging, you can
see
> what
> > values are set for these environment variables to get a better
idea of
> how
> > your METplus configuration will affect the MET configuration.
> >
> > Please let me know if you have any questions regarding any of
this.
> >
> > On Tue, May 5, 2020 at 11:46 AM Edward Strobach - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> > >
> > > Update:
> > >
> > > Trying to follow the log files that are generated for guidance.
I set
> > the
> > > verbosity to 7, but still can't understand why I'm receiving
this
> > message:
> > >
> > > 05/05 17:41:59.861 metplus INFO: Running METplus v3.0 called
with
> > command:
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/master_metplus.py
> > > -c
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/use_cases/model_applications/air_quality/CMAQ_obsAirnow.conf
> > > 05/05 17:41:59.882 metplus (met_util.py:776) ERROR:
> [INIT/VALID]_TIME_FMT
> > > (%Y%m%d%H) does not match [INIT/VALID]_[BEG/END] (2020050200
> > #2020050317)
> > > 05/05 17:41:59.886 metplus.PB2NC (met_util.py:192) ERROR: Fatal
error
> > > occurred
> > >
> > > I tried to start basic. If starting basic worked, then I would
have
> made
> > > it more complex:
> > >
> > > # time looping - options are INIT, VALID, RETRO, and REALTIME
> > > LOG_MET_VERBOSITY = 7
> > >
> > > LOOP_BY = INIT
> > >
> > > # Format of VALID_BEG and VALID_END
> > > INIT_TIME_FMT = %Y%m%d%H
> > >
> > > # Start time, end time, and time increment for METplus run
> > > INIT_BEG = 2020050200 #{now?fmt=%Y%m%d%H?shift=-172800}
> > > INIT_END = 2020050200 #{now?fmt=%Y%m%d%H?shift=-172800} #-
21600}
> > > INIT_INCREMENT = 0
> > > LEAD_SEQ = 0
> > >
> > > With how this looks, I would have thought that only one file
would be
> > > processed. But it complains about a mismatch between the
INIT_TIME_FMT
> > and
> > > INIT_BEG/INIT_END
> > >
> > > On Tue, May 5, 2020 at 12:01 PM met_help at ucar.edu via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > Greetings,
> > > >
> > > > This message has been automatically generated in response to
the
> > creation
> > > > of a trouble ticket regarding:
> > > > "PB2NC_OFFSETS/related questions",
> > > > a summary of which appears below.
> > > >
> > > > There is no need to reply to this message right now. Your
ticket has
> > > been
> > > > assigned an ID of [rt.rap.ucar.edu #95164].
> > > >
> > > > Please include the string:
> > > >
> > > > [rt.rap.ucar.edu #95164]
> > > >
> > > > in the subject line of all future correspondence about this
issue. To
> > do
> > > > so, you may reply to this message.
> > > >
> > > > For more information, please see:
> > > >
> > > > MET Online Tutorial:
> > > >
> >
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > > >
> > > > MET Users Guide:
> > > > https://www.dtcenter.org/met/users/docs/overview.php
> > > >
> > > > MET FAQs:
> > > > https://www.dtcenter.org/met/users/support/faqs/index.php
> > > >
> > > > MET-Help Email Archive:
> > > > http://mailman.ucar.edu/pipermail/met_help
> > > >
> > > > Thank you,
> > > > met_help at ucar.edu
> > > >
> > > >
> >
-------------------------------------------------------------------------
> > > > Good morning,
> > > >
> > > > I'm trying to configure a file to process prepbufr data to
netcdf
> > format.
> > > > There are several files that are within the directory that I
want to
> > > > process as shown by the bolded text from the
> > > > /gpfs/dell1/nco/ops/com/hourly/prod/hourly.20200503 directory:
> > > >
> > > > aqm.t12z.airnow.tm00.bufr_d aqm.t12z.anowpm.pb.tm120
> > > > aqm.t12z.status.tm024.bufr_d
> > > > aqm.t12z.airnow.tm024.bufr_d *aqm.t12z.prepbufr.tm00 *
> > > > aqm.t12z.status.tm120.bufr_d
> > > > aqm.t12z.airnow.tm120.bufr_d *aqm.t12z.prepbufr.tm024*
> > > > obsproc_version_for_t12z_aqm_dump_run
> > > > *aqm.t12z.anowpm.pb.tm00* aqm.t12z.prepbufr.tm120
> > > > where_t12z_aqm_dump_ran
> > > > *aqm.t12z.anowpm.pb.tm024 * aqm.t12z.status.tm00.bufr_d
> > > >
> > > > Right now I'm focusing only on the *anowpm* files (two files).
The
> > tm00
> > > is
> > > > time centered on 12z for 05/03/2020, with DHR = ranging from
-11 to
> 12,
> > > > covering the whole day. Similarly, the tm024 file is centered
at
> 12z,
> > > but
> > > > for 05/02/2020. I looked through the metplus glossary and
examples
> to
> > > help
> > > > with the process of building the configure file. If I run
this
> script
> > > and
> > > > desire the files from 05/03/2020, then I should set the
following:
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *[config]# time looping - options are INIT, VALID, RETRO, and
> > > > REALTIMELOOP_BY = VALID# Format of VALID_BEG and
> > VALID_ENDVALID_TIME_FMT
> > > =
> > > > %Y%m%d%H# Start time, end time, and time increment for METplus
> > > runVALID_BEG
> > > > = {now?fmt=%Y%m%d%H?shift=-2d}VALID_END =
> {now?fmt=%Y%m%d%H?shift=-2d}
> > > > #-21600}VALID_INCREMENT = 24HLEAD_SEQ = 0,24 (((NOT SURE I
NEED
> > THIS,
> > > > BUT NOT SURE HOW THE LOOP WILL WORK WITHOUT IT?)))*
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *LOOP_ORDER = times# List of applications to runPROCESS_LIST =
> > > > PB2NCCONFIG_DIR =
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/met_configCONFIG_FILE
> > > > = {CONFIG_DIR}/PB2NCConfig_wrappedPB2NC_OBS_BUFR_VAR_LIST =
DHR, RPT,
> > > SID,
> > > > YOB, XOB, TYPO, TPHR, QCIND, COPO, COPOPM*
> > > >
> > > > I've listed the variables I would like to extract from each
file as
> > well.
> > > > It's not clear if I need the config file, but I listed it just
in
> case.
> > > > Here's what I have for the file names from the input directory
and
> > what I
> > > > would name it once converted to netcdf and placed in the
output
> > > directory:
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
*INPUT_BASE=/gpfs/dell1/nco/ops/com/hourly/prod/hourly.{now?fmt=%Y%m%d?shift=-
2d}
> > > > # -86400}PB2NC_INPUT_DIR = {INPUT_BASE}OUTPUT_BASE =
> > > >
/gpfs/dell2/emc/modeling/noscrub/Edward.StrobachPB2NC_OUTPUT_DIR =
> > > > {OUTPUT_BASE}/AirNow/pb2nc/hourly.{now?fmt=%Y%m%d?shift=-2d} #
> > > > -86400}[filename_templates]PB2NC_INPUT_TEMPLATE =
> > > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ} (((RELATED TO LOOP
> > > > CONFUSION)))PB2NC_OUTPUT_TEMPLATE =
> > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ}.nc*
> > > >
> > > > Let me know if more information is needed. My main concerns
here are
> > 1)
> > > > being able to process all the list files correctly, 2) making
sure
> the
> > > > looping is correct, and 3) making sure that I've included
everything
> > > that I
> > > > would need to process this step
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Edward Strobach
> > > > EMC/NCEP/NWS/
> > > > IMSG Contractor
> > > > Cubicle#: 2029
> > > > 301-683-3717
> > > >
> > > >
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: PB2NC_OFFSETS/related questions
From: Edward Strobach - NOAA Affiliate
Time: Wed May 06 13:02:54 2020
Sure thing. I think you are referring to this?
05/06 17:34:09.831 metplus.PointStat (command_builder.py:100) ERROR:
(command_builder.py:437) Could not find FCST file
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
using template
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
05/06 17:34:09.832 metplus.PointStat (met_util.py:192) ERROR: Fatal
error
occurred
*Traceback (most recent call last):*
File
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
line 169, in run_metplus
process.run_all_times()
File
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
line 751, in run_all_times
util.loop_over_times_and_call(self.config, self)
File
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
line 834, in loop_over_times_and_call
process.run_at_time(input_dict)
File
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/compare_gridded_wrapper.py",
line 183, in run_at_time
self.run_at_time_once(time_info)
File
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/point_stat_wrapper.py",
line 146, in run_at_time_once
model_path = self.find_model(time_info, var_list[0])
File
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
line 319, in find_model
return_list=return_list)
File
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
line 373, in find_data
return self.find_exact_file(**arg_dict)
File
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
line 437, in find_exact_file
self.log_error(msg)
File
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
line 100, in log_error
self.logger.error(f"({os.path.basename(caller.filename)}:{caller.lineno})
{error_string}")
File
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
line 1335, in error
self._log(ERROR, msg, args, **kwargs)
File
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
line 1442, in _log
self.handle(record)
File
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
line 1452, in handle
self.callHandlers(record)
File
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
line 1514, in callHandlers
hdlr.handle(record)
File
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
line 863, in handle
self.emit(record)
File
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/produtil/log.py",
line 227, in emit
with open(self._logger,'at') as f:
On Wed, May 6, 2020 at 2:55 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Edward,
>
> When you cloned the repo yourself, are you using the develop branch
to run?
> There may have been a bug introduced that is causing that fatal
error.
> Could you send me the stack trace that comes after the fatal error?
I agree
> that it shouldn't crash if it can't find a file.
>
> On Wed, May 6, 2020 at 12:43 PM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> >
> > Thanks George. I've been busy trying to figure things out and
forgot to
> > respond. I talked with Perry about some of these things and he
was
> > surprised that I was receiving this message when running his
script:
> >
> > 05/06 17:34:09.517 metplus.PointStat INFO: Processing forecast
lead 0
> hours
> > 05/06 17:34:09.525 metplus.PointStat (command_builder.py:407)
DEBUG:
> > Looking for FCST file
> >
>
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
> > 05/06 17:34:09.831 metplus.PointStat (command_builder.py:100)
ERROR:
> > (command_builder.py:437) Could not find FCST file
> > /gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/
> > *aqm.t00z.awpozcon.f00.148.grib2* using template
> >
> >
>
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
> > 05/06 17:34:09.832 metplus.PointStat (met_util.py:192) ERROR:
Fatal error
> > occurred
> >
> > The bolded item does not exist. When Perry runs his script it
skips this
> > file. When I run the script I get a fatal error. I cloned
metplus just
> > the other day. I'm not sure if this is related to item 2 from
your email
> >
> > On Tue, May 5, 2020 at 2:37 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Edward,
> > >
> > > 1) The LOG_MET_VERBOSITY variable sets the verbosity level when
a MET
> > tool
> > > is run. To change the METplus logging level, set LOG_LEVEL. I
would
> > > recommend changing it to DEBUG to get more information.
> > >
> > > 2) There is a bug in using the {now} tags for
[INIT/VALID]_[BEG/END]
> that
> > > causes that crash. This has been fixed but we have not yet
released a
> > > bugfix release. I will let the engineers know that you need this
> > > functionality and we will push the METplus 3.0.1 bugfix release
soon.
> You
> > > should be able to set the time explicitly, i.e. 2020050500.
> > >
> > > 3) The {now} tag can be used in the VALID_BEG and VALID_END
config
> > > variables, but you will want to refer to init, valid, lead, etc.
in
> your
> > > INPUT_TEMPLATE and OUTPUT_TEMPLATE variables so that it uses the
time
> for
> > > each time loop. See this section for more information:
> > >
> > >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#directory-
and-filename-template-info
> > >
> > > 4) The LEAD_SEQ is a list of forecast leads that will be
processed for
> > each
> > > run time. If you set this to 0 or unset it, then it will just
process
> > once
> > > for each run time.
> > >
> > > 5) There is documentation describing the time looping variables
here:
> > >
> > >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#common-
config-variables
> > >
> > > This page should answer some of your questions on how this
works.
> > >
> > > 6) I would recommend starting with the Example use case found in
> > > parm/use_cases/met_tool_wrapper/Example/Example.conf. This use
case
> loops
> > > over times, fills in the filename templates based on each run
time, and
> > > prints information to the logfile. You can copy this use case
and
> modify
> > it
> > > to see which filenames are searched for each run time. This is a
good
> way
> > > to understand how to configure METplus to find the files you
need. Once
> > > this makes sense, you can look at the PB2NC.conf use case to see
how
> the
> > > config variables control that wrapper.
> > >
> > > 7) METplus sets environment variables based on its configuration
files.
> > > These environment variables are referenced in the MET config
file to
> set
> > > configurations in that file. If you turn on DEBUG logging, you
can see
> > what
> > > values are set for these environment variables to get a better
idea of
> > how
> > > your METplus configuration will affect the MET configuration.
> > >
> > > Please let me know if you have any questions regarding any of
this.
> > >
> > > On Tue, May 5, 2020 at 11:46 AM Edward Strobach - NOAA Affiliate
via
> RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
>
> > > >
> > > > Update:
> > > >
> > > > Trying to follow the log files that are generated for
guidance. I
> set
> > > the
> > > > verbosity to 7, but still can't understand why I'm receiving
this
> > > message:
> > > >
> > > > 05/05 17:41:59.861 metplus INFO: Running METplus v3.0 called
with
> > > command:
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/master_metplus.py
> > > > -c
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/use_cases/model_applications/air_quality/CMAQ_obsAirnow.conf
> > > > 05/05 17:41:59.882 metplus (met_util.py:776) ERROR:
> > [INIT/VALID]_TIME_FMT
> > > > (%Y%m%d%H) does not match [INIT/VALID]_[BEG/END] (2020050200
> > > #2020050317)
> > > > 05/05 17:41:59.886 metplus.PB2NC (met_util.py:192) ERROR:
Fatal error
> > > > occurred
> > > >
> > > > I tried to start basic. If starting basic worked, then I
would have
> > made
> > > > it more complex:
> > > >
> > > > # time looping - options are INIT, VALID, RETRO, and REALTIME
> > > > LOG_MET_VERBOSITY = 7
> > > >
> > > > LOOP_BY = INIT
> > > >
> > > > # Format of VALID_BEG and VALID_END
> > > > INIT_TIME_FMT = %Y%m%d%H
> > > >
> > > > # Start time, end time, and time increment for METplus run
> > > > INIT_BEG = 2020050200 #{now?fmt=%Y%m%d%H?shift=-172800}
> > > > INIT_END = 2020050200 #{now?fmt=%Y%m%d%H?shift=-172800} #-
21600}
> > > > INIT_INCREMENT = 0
> > > > LEAD_SEQ = 0
> > > >
> > > > With how this looks, I would have thought that only one file
would be
> > > > processed. But it complains about a mismatch between the
> INIT_TIME_FMT
> > > and
> > > > INIT_BEG/INIT_END
> > > >
> > > > On Tue, May 5, 2020 at 12:01 PM met_help at ucar.edu via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > Greetings,
> > > > >
> > > > > This message has been automatically generated in response to
the
> > > creation
> > > > > of a trouble ticket regarding:
> > > > > "PB2NC_OFFSETS/related questions",
> > > > > a summary of which appears below.
> > > > >
> > > > > There is no need to reply to this message right now. Your
ticket
> has
> > > > been
> > > > > assigned an ID of [rt.rap.ucar.edu #95164].
> > > > >
> > > > > Please include the string:
> > > > >
> > > > > [rt.rap.ucar.edu #95164]
> > > > >
> > > > > in the subject line of all future correspondence about this
issue.
> To
> > > do
> > > > > so, you may reply to this message.
> > > > >
> > > > > For more information, please see:
> > > > >
> > > > > MET Online Tutorial:
> > > > >
> > >
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > > > >
> > > > > MET Users Guide:
> > > > > https://www.dtcenter.org/met/users/docs/overview.php
> > > > >
> > > > > MET FAQs:
> > > > > https://www.dtcenter.org/met/users/support/faqs/index.php
> > > > >
> > > > > MET-Help Email Archive:
> > > > > http://mailman.ucar.edu/pipermail/met_help
> > > > >
> > > > > Thank you,
> > > > > met_help at ucar.edu
> > > > >
> > > > >
> > >
>
-------------------------------------------------------------------------
> > > > > Good morning,
> > > > >
> > > > > I'm trying to configure a file to process prepbufr data to
netcdf
> > > format.
> > > > > There are several files that are within the directory that I
want
> to
> > > > > process as shown by the bolded text from the
> > > > > /gpfs/dell1/nco/ops/com/hourly/prod/hourly.20200503
directory:
> > > > >
> > > > > aqm.t12z.airnow.tm00.bufr_d aqm.t12z.anowpm.pb.tm120
> > > > > aqm.t12z.status.tm024.bufr_d
> > > > > aqm.t12z.airnow.tm024.bufr_d *aqm.t12z.prepbufr.tm00 *
> > > > > aqm.t12z.status.tm120.bufr_d
> > > > > aqm.t12z.airnow.tm120.bufr_d *aqm.t12z.prepbufr.tm024*
> > > > > obsproc_version_for_t12z_aqm_dump_run
> > > > > *aqm.t12z.anowpm.pb.tm00* aqm.t12z.prepbufr.tm120
> > > > > where_t12z_aqm_dump_ran
> > > > > *aqm.t12z.anowpm.pb.tm024 * aqm.t12z.status.tm00.bufr_d
> > > > >
> > > > > Right now I'm focusing only on the *anowpm* files (two
files). The
> > > tm00
> > > > is
> > > > > time centered on 12z for 05/03/2020, with DHR = ranging from
-11 to
> > 12,
> > > > > covering the whole day. Similarly, the tm024 file is
centered at
> > 12z,
> > > > but
> > > > > for 05/02/2020. I looked through the metplus glossary and
examples
> > to
> > > > help
> > > > > with the process of building the configure file. If I run
this
> > script
> > > > and
> > > > > desire the files from 05/03/2020, then I should set the
following:
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *[config]# time looping - options are INIT, VALID, RETRO,
and
> > > > > REALTIMELOOP_BY = VALID# Format of VALID_BEG and
> > > VALID_ENDVALID_TIME_FMT
> > > > =
> > > > > %Y%m%d%H# Start time, end time, and time increment for
METplus
> > > > runVALID_BEG
> > > > > = {now?fmt=%Y%m%d%H?shift=-2d}VALID_END =
> > {now?fmt=%Y%m%d%H?shift=-2d}
> > > > > #-21600}VALID_INCREMENT = 24HLEAD_SEQ = 0,24 (((NOT SURE
I NEED
> > > THIS,
> > > > > BUT NOT SURE HOW THE LOOP WILL WORK WITHOUT IT?)))*
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *LOOP_ORDER = times# List of applications to runPROCESS_LIST
=
> > > > > PB2NCCONFIG_DIR =
> > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/met_configCONFIG_FILE
> > > > > = {CONFIG_DIR}/PB2NCConfig_wrappedPB2NC_OBS_BUFR_VAR_LIST =
DHR,
> RPT,
> > > > SID,
> > > > > YOB, XOB, TYPO, TPHR, QCIND, COPO, COPOPM*
> > > > >
> > > > > I've listed the variables I would like to extract from each
file as
> > > well.
> > > > > It's not clear if I need the config file, but I listed it
just in
> > case.
> > > > > Here's what I have for the file names from the input
directory and
> > > what I
> > > > > would name it once converted to netcdf and placed in the
output
> > > > directory:
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
*INPUT_BASE=/gpfs/dell1/nco/ops/com/hourly/prod/hourly.{now?fmt=%Y%m%d?shift=-
2d}
> > > > > # -86400}PB2NC_INPUT_DIR = {INPUT_BASE}OUTPUT_BASE =
> > > > >
/gpfs/dell2/emc/modeling/noscrub/Edward.StrobachPB2NC_OUTPUT_DIR =
> > > > > {OUTPUT_BASE}/AirNow/pb2nc/hourly.{now?fmt=%Y%m%d?shift=-2d}
#
> > > > > -86400}[filename_templates]PB2NC_INPUT_TEMPLATE =
> > > > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ} (((RELATED TO LOOP
> > > > > CONFUSION)))PB2NC_OUTPUT_TEMPLATE =
> > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ}.nc*
> > > > >
> > > > > Let me know if more information is needed. My main concerns
here
> are
> > > 1)
> > > > > being able to process all the list files correctly, 2)
making sure
> > the
> > > > > looping is correct, and 3) making sure that I've included
> everything
> > > > that I
> > > > > would need to process this step
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Edward Strobach
> > > > > EMC/NCEP/NWS/
> > > > > IMSG Contractor
> > > > > Cubicle#: 2029
> > > > > 301-683-3717
> > > > >
> > > > >
> > > >
> > > > --
> > > > Edward Strobach
> > > > EMC/NCEP/NWS/
> > > > IMSG Contractor
> > > > Cubicle#: 2029
> > > > 301-683-3717
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: PB2NC_OFFSETS/related questions
From: George McCabe
Time: Wed May 06 13:21:44 2020
That's it, thanks.
Hmm, it looks like there is an error trying to add to the log file?
Did
this run generate a log file? It would be found in the logs directory
under
the value you set for OUTPUT_BASE. If there is one, can you send that
to
me? Also, please send the metplus_final.conf file in your OUTPUT_BASE
as
well.
On Wed, May 6, 2020 at 1:03 PM Edward Strobach - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
>
> Sure thing. I think you are referring to this?
>
> 05/06 17:34:09.831 metplus.PointStat (command_builder.py:100) ERROR:
> (command_builder.py:437) Could not find FCST file
>
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
> using template
>
>
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
> 05/06 17:34:09.832 metplus.PointStat (met_util.py:192) ERROR: Fatal
error
> occurred
> *Traceback (most recent call last):*
> File
>
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> line 169, in run_metplus
> process.run_all_times()
> File
>
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> line 751, in run_all_times
> util.loop_over_times_and_call(self.config, self)
> File
>
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> line 834, in loop_over_times_and_call
> process.run_at_time(input_dict)
> File
>
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/compare_gridded_wrapper.py",
> line 183, in run_at_time
> self.run_at_time_once(time_info)
> File
>
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/point_stat_wrapper.py",
> line 146, in run_at_time_once
> model_path = self.find_model(time_info, var_list[0])
> File
>
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> line 319, in find_model
> return_list=return_list)
> File
>
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> line 373, in find_data
> return self.find_exact_file(**arg_dict)
> File
>
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> line 437, in find_exact_file
> self.log_error(msg)
> File
>
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> line 100, in log_error
>
>
self.logger.error(f"({os.path.basename(caller.filename)}:{caller.lineno})
> {error_string}")
> File
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> line 1335, in error
> self._log(ERROR, msg, args, **kwargs)
> File
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> line 1442, in _log
> self.handle(record)
> File
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> line 1452, in handle
> self.callHandlers(record)
> File
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> line 1514, in callHandlers
> hdlr.handle(record)
> File
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> line 863, in handle
> self.emit(record)
> File
>
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/produtil/log.py",
> line 227, in emit
> with open(self._logger,'at') as f:
>
> On Wed, May 6, 2020 at 2:55 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Edward,
> >
> > When you cloned the repo yourself, are you using the develop
branch to
> run?
> > There may have been a bug introduced that is causing that fatal
error.
> > Could you send me the stack trace that comes after the fatal
error? I
> agree
> > that it shouldn't crash if it can't find a file.
> >
> > On Wed, May 6, 2020 at 12:43 PM Edward Strobach - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> > >
> > > Thanks George. I've been busy trying to figure things out and
forgot
> to
> > > respond. I talked with Perry about some of these things and he
was
> > > surprised that I was receiving this message when running his
script:
> > >
> > > 05/06 17:34:09.517 metplus.PointStat INFO: Processing forecast
lead 0
> > hours
> > > 05/06 17:34:09.525 metplus.PointStat (command_builder.py:407)
DEBUG:
> > > Looking for FCST file
> > >
> >
>
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
> > > 05/06 17:34:09.831 metplus.PointStat (command_builder.py:100)
ERROR:
> > > (command_builder.py:437) Could not find FCST file
> > > /gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/
> > > *aqm.t00z.awpozcon.f00.148.grib2* using template
> > >
> > >
> >
>
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
> > > 05/06 17:34:09.832 metplus.PointStat (met_util.py:192) ERROR:
Fatal
> error
> > > occurred
> > >
> > > The bolded item does not exist. When Perry runs his script it
skips
> this
> > > file. When I run the script I get a fatal error. I cloned
metplus
> just
> > > the other day. I'm not sure if this is related to item 2 from
your
> email
> > >
> > > On Tue, May 5, 2020 at 2:37 PM George McCabe via RT
<met_help at ucar.edu
> >
> > > wrote:
> > >
> > > > Hi Edward,
> > > >
> > > > 1) The LOG_MET_VERBOSITY variable sets the verbosity level
when a MET
> > > tool
> > > > is run. To change the METplus logging level, set LOG_LEVEL. I
would
> > > > recommend changing it to DEBUG to get more information.
> > > >
> > > > 2) There is a bug in using the {now} tags for
[INIT/VALID]_[BEG/END]
> > that
> > > > causes that crash. This has been fixed but we have not yet
released a
> > > > bugfix release. I will let the engineers know that you need
this
> > > > functionality and we will push the METplus 3.0.1 bugfix
release soon.
> > You
> > > > should be able to set the time explicitly, i.e. 2020050500.
> > > >
> > > > 3) The {now} tag can be used in the VALID_BEG and VALID_END
config
> > > > variables, but you will want to refer to init, valid, lead,
etc. in
> > your
> > > > INPUT_TEMPLATE and OUTPUT_TEMPLATE variables so that it uses
the time
> > for
> > > > each time loop. See this section for more information:
> > > >
> > > >
> > >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#directory-
and-filename-template-info
> > > >
> > > > 4) The LEAD_SEQ is a list of forecast leads that will be
processed
> for
> > > each
> > > > run time. If you set this to 0 or unset it, then it will just
process
> > > once
> > > > for each run time.
> > > >
> > > > 5) There is documentation describing the time looping
variables here:
> > > >
> > > >
> > >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#common-
config-variables
> > > >
> > > > This page should answer some of your questions on how this
works.
> > > >
> > > > 6) I would recommend starting with the Example use case found
in
> > > > parm/use_cases/met_tool_wrapper/Example/Example.conf. This use
case
> > loops
> > > > over times, fills in the filename templates based on each run
time,
> and
> > > > prints information to the logfile. You can copy this use case
and
> > modify
> > > it
> > > > to see which filenames are searched for each run time. This is
a good
> > way
> > > > to understand how to configure METplus to find the files you
need.
> Once
> > > > this makes sense, you can look at the PB2NC.conf use case to
see how
> > the
> > > > config variables control that wrapper.
> > > >
> > > > 7) METplus sets environment variables based on its
configuration
> files.
> > > > These environment variables are referenced in the MET config
file to
> > set
> > > > configurations in that file. If you turn on DEBUG logging, you
can
> see
> > > what
> > > > values are set for these environment variables to get a better
idea
> of
> > > how
> > > > your METplus configuration will affect the MET configuration.
> > > >
> > > > Please let me know if you have any questions regarding any of
this.
> > > >
> > > > On Tue, May 5, 2020 at 11:46 AM Edward Strobach - NOAA
Affiliate via
> > RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> > > > >
> > > > > Update:
> > > > >
> > > > > Trying to follow the log files that are generated for
guidance. I
> > set
> > > > the
> > > > > verbosity to 7, but still can't understand why I'm receiving
this
> > > > message:
> > > > >
> > > > > 05/05 17:41:59.861 metplus INFO: Running METplus v3.0 called
with
> > > > command:
> > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/master_metplus.py
> > > > > -c
> > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/use_cases/model_applications/air_quality/CMAQ_obsAirnow.conf
> > > > > 05/05 17:41:59.882 metplus (met_util.py:776) ERROR:
> > > [INIT/VALID]_TIME_FMT
> > > > > (%Y%m%d%H) does not match [INIT/VALID]_[BEG/END] (2020050200
> > > > #2020050317)
> > > > > 05/05 17:41:59.886 metplus.PB2NC (met_util.py:192) ERROR:
Fatal
> error
> > > > > occurred
> > > > >
> > > > > I tried to start basic. If starting basic worked, then I
would
> have
> > > made
> > > > > it more complex:
> > > > >
> > > > > # time looping - options are INIT, VALID, RETRO, and
REALTIME
> > > > > LOG_MET_VERBOSITY = 7
> > > > >
> > > > > LOOP_BY = INIT
> > > > >
> > > > > # Format of VALID_BEG and VALID_END
> > > > > INIT_TIME_FMT = %Y%m%d%H
> > > > >
> > > > > # Start time, end time, and time increment for METplus run
> > > > > INIT_BEG = 2020050200 #{now?fmt=%Y%m%d%H?shift=-172800}
> > > > > INIT_END = 2020050200 #{now?fmt=%Y%m%d%H?shift=-172800} #-
21600}
> > > > > INIT_INCREMENT = 0
> > > > > LEAD_SEQ = 0
> > > > >
> > > > > With how this looks, I would have thought that only one file
would
> be
> > > > > processed. But it complains about a mismatch between the
> > INIT_TIME_FMT
> > > > and
> > > > > INIT_BEG/INIT_END
> > > > >
> > > > > On Tue, May 5, 2020 at 12:01 PM met_help at ucar.edu via RT <
> > > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > Greetings,
> > > > > >
> > > > > > This message has been automatically generated in response
to the
> > > > creation
> > > > > > of a trouble ticket regarding:
> > > > > > "PB2NC_OFFSETS/related questions",
> > > > > > a summary of which appears below.
> > > > > >
> > > > > > There is no need to reply to this message right now. Your
ticket
> > has
> > > > > been
> > > > > > assigned an ID of [rt.rap.ucar.edu #95164].
> > > > > >
> > > > > > Please include the string:
> > > > > >
> > > > > > [rt.rap.ucar.edu #95164]
> > > > > >
> > > > > > in the subject line of all future correspondence about
this
> issue.
> > To
> > > > do
> > > > > > so, you may reply to this message.
> > > > > >
> > > > > > For more information, please see:
> > > > > >
> > > > > > MET Online Tutorial:
> > > > > >
> > > >
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > > > > >
> > > > > > MET Users Guide:
> > > > > > https://www.dtcenter.org/met/users/docs/overview.php
> > > > > >
> > > > > > MET FAQs:
> > > > > >
https://www.dtcenter.org/met/users/support/faqs/index.php
> > > > > >
> > > > > > MET-Help Email Archive:
> > > > > > http://mailman.ucar.edu/pipermail/met_help
> > > > > >
> > > > > > Thank you,
> > > > > > met_help at ucar.edu
> > > > > >
> > > > > >
> > > >
> >
-------------------------------------------------------------------------
> > > > > > Good morning,
> > > > > >
> > > > > > I'm trying to configure a file to process prepbufr data to
netcdf
> > > > format.
> > > > > > There are several files that are within the directory that
I want
> > to
> > > > > > process as shown by the bolded text from the
> > > > > > /gpfs/dell1/nco/ops/com/hourly/prod/hourly.20200503
directory:
> > > > > >
> > > > > > aqm.t12z.airnow.tm00.bufr_d aqm.t12z.anowpm.pb.tm120
> > > > > > aqm.t12z.status.tm024.bufr_d
> > > > > > aqm.t12z.airnow.tm024.bufr_d *aqm.t12z.prepbufr.tm00 *
> > > > > > aqm.t12z.status.tm120.bufr_d
> > > > > > aqm.t12z.airnow.tm120.bufr_d *aqm.t12z.prepbufr.tm024*
> > > > > > obsproc_version_for_t12z_aqm_dump_run
> > > > > > *aqm.t12z.anowpm.pb.tm00* aqm.t12z.prepbufr.tm120
> > > > > > where_t12z_aqm_dump_ran
> > > > > > *aqm.t12z.anowpm.pb.tm024 *
aqm.t12z.status.tm00.bufr_d
> > > > > >
> > > > > > Right now I'm focusing only on the *anowpm* files (two
files).
> The
> > > > tm00
> > > > > is
> > > > > > time centered on 12z for 05/03/2020, with DHR = ranging
from -11
> to
> > > 12,
> > > > > > covering the whole day. Similarly, the tm024 file is
centered at
> > > 12z,
> > > > > but
> > > > > > for 05/02/2020. I looked through the metplus glossary and
> examples
> > > to
> > > > > help
> > > > > > with the process of building the configure file. If I run
this
> > > script
> > > > > and
> > > > > > desire the files from 05/03/2020, then I should set the
> following:
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > *[config]# time looping - options are INIT, VALID, RETRO,
and
> > > > > > REALTIMELOOP_BY = VALID# Format of VALID_BEG and
> > > > VALID_ENDVALID_TIME_FMT
> > > > > =
> > > > > > %Y%m%d%H# Start time, end time, and time increment for
METplus
> > > > > runVALID_BEG
> > > > > > = {now?fmt=%Y%m%d%H?shift=-2d}VALID_END =
> > > {now?fmt=%Y%m%d%H?shift=-2d}
> > > > > > #-21600}VALID_INCREMENT = 24HLEAD_SEQ = 0,24 (((NOT
SURE I
> NEED
> > > > THIS,
> > > > > > BUT NOT SURE HOW THE LOOP WILL WORK WITHOUT IT?)))*
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > *LOOP_ORDER = times# List of applications to
runPROCESS_LIST =
> > > > > > PB2NCCONFIG_DIR =
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/met_configCONFIG_FILE
> > > > > > = {CONFIG_DIR}/PB2NCConfig_wrappedPB2NC_OBS_BUFR_VAR_LIST
= DHR,
> > RPT,
> > > > > SID,
> > > > > > YOB, XOB, TYPO, TPHR, QCIND, COPO, COPOPM*
> > > > > >
> > > > > > I've listed the variables I would like to extract from
each file
> as
> > > > well.
> > > > > > It's not clear if I need the config file, but I listed it
just in
> > > case.
> > > > > > Here's what I have for the file names from the input
directory
> and
> > > > what I
> > > > > > would name it once converted to netcdf and placed in the
output
> > > > > directory:
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
*INPUT_BASE=/gpfs/dell1/nco/ops/com/hourly/prod/hourly.{now?fmt=%Y%m%d?shift=-
2d}
> > > > > > # -86400}PB2NC_INPUT_DIR = {INPUT_BASE}OUTPUT_BASE =
> > > > > >
/gpfs/dell2/emc/modeling/noscrub/Edward.StrobachPB2NC_OUTPUT_DIR
> =
> > > > > > {OUTPUT_BASE}/AirNow/pb2nc/hourly.{now?fmt=%Y%m%d?shift=-
2d} #
> > > > > > -86400}[filename_templates]PB2NC_INPUT_TEMPLATE =
> > > > > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ} (((RELATED TO LOOP
> > > > > > CONFUSION)))PB2NC_OUTPUT_TEMPLATE =
> > > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ}.nc*
> > > > > >
> > > > > > Let me know if more information is needed. My main
concerns here
> > are
> > > > 1)
> > > > > > being able to process all the list files correctly, 2)
making
> sure
> > > the
> > > > > > looping is correct, and 3) making sure that I've included
> > everything
> > > > > that I
> > > > > > would need to process this step
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Edward Strobach
> > > > > > EMC/NCEP/NWS/
> > > > > > IMSG Contractor
> > > > > > Cubicle#: 2029
> > > > > > 301-683-3717
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Edward Strobach
> > > > > EMC/NCEP/NWS/
> > > > > IMSG Contractor
> > > > > Cubicle#: 2029
> > > > > 301-683-3717
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: PB2NC_OFFSETS/related questions
From: Edward Strobach - NOAA Affiliate
Time: Wed May 06 13:43:15 2020
So my OUTPUT_BASE is here:
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq.
There is a conf directory where .conf files are stored in aq
(OUTPUT_BASE/conf/aq) and pm (OUTPUT_BASE/conf/pm). There are some
.conf
files from these runs.
aq:
metplus_final_pb2nc_pointstat.conf
metplus_final_statanalysis_gatherbydate.conf
pm:
metplus_final_pb2nc_pointstat.conf
metplus_final_statanalysis_gatherbydate.conf
Any file in particular that you want to see? I don't have the file
you are
referring to.
On Wed, May 6, 2020 at 3:22 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> That's it, thanks.
>
> Hmm, it looks like there is an error trying to add to the log file?
Did
> this run generate a log file? It would be found in the logs
directory under
> the value you set for OUTPUT_BASE. If there is one, can you send
that to
> me? Also, please send the metplus_final.conf file in your
OUTPUT_BASE as
> well.
>
>
> On Wed, May 6, 2020 at 1:03 PM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> >
> > Sure thing. I think you are referring to this?
> >
> > 05/06 17:34:09.831 metplus.PointStat (command_builder.py:100)
ERROR:
> > (command_builder.py:437) Could not find FCST file
> >
>
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
> > using template
> >
> >
>
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
> > 05/06 17:34:09.832 metplus.PointStat (met_util.py:192) ERROR:
Fatal error
> > occurred
> > *Traceback (most recent call last):*
> > File
> >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > line 169, in run_metplus
> > process.run_all_times()
> > File
> >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > line 751, in run_all_times
> > util.loop_over_times_and_call(self.config, self)
> > File
> >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > line 834, in loop_over_times_and_call
> > process.run_at_time(input_dict)
> > File
> >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/compare_gridded_wrapper.py",
> > line 183, in run_at_time
> > self.run_at_time_once(time_info)
> > File
> >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/point_stat_wrapper.py",
> > line 146, in run_at_time_once
> > model_path = self.find_model(time_info, var_list[0])
> > File
> >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > line 319, in find_model
> > return_list=return_list)
> > File
> >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > line 373, in find_data
> > return self.find_exact_file(**arg_dict)
> > File
> >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > line 437, in find_exact_file
> > self.log_error(msg)
> > File
> >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > line 100, in log_error
> >
> >
self.logger.error(f"({os.path.basename(caller.filename)}:{caller.lineno})
> > {error_string}")
> > File
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > line 1335, in error
> > self._log(ERROR, msg, args, **kwargs)
> > File
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > line 1442, in _log
> > self.handle(record)
> > File
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > line 1452, in handle
> > self.callHandlers(record)
> > File
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > line 1514, in callHandlers
> > hdlr.handle(record)
> > File
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > line 863, in handle
> > self.emit(record)
> > File
> >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/produtil/log.py",
> > line 227, in emit
> > with open(self._logger,'at') as f:
> >
> > On Wed, May 6, 2020 at 2:55 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Edward,
> > >
> > > When you cloned the repo yourself, are you using the develop
branch to
> > run?
> > > There may have been a bug introduced that is causing that fatal
error.
> > > Could you send me the stack trace that comes after the fatal
error? I
> > agree
> > > that it shouldn't crash if it can't find a file.
> > >
> > > On Wed, May 6, 2020 at 12:43 PM Edward Strobach - NOAA Affiliate
via
> RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
>
> > > >
> > > > Thanks George. I've been busy trying to figure things out and
forgot
> > to
> > > > respond. I talked with Perry about some of these things and
he was
> > > > surprised that I was receiving this message when running his
script:
> > > >
> > > > 05/06 17:34:09.517 metplus.PointStat INFO: Processing forecast
lead 0
> > > hours
> > > > 05/06 17:34:09.525 metplus.PointStat (command_builder.py:407)
DEBUG:
> > > > Looking for FCST file
> > > >
> > >
> >
>
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
> > > > 05/06 17:34:09.831 metplus.PointStat (command_builder.py:100)
ERROR:
> > > > (command_builder.py:437) Could not find FCST file
> > > > /gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/
> > > > *aqm.t00z.awpozcon.f00.148.grib2* using template
> > > >
> > > >
> > >
> >
>
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
> > > > 05/06 17:34:09.832 metplus.PointStat (met_util.py:192) ERROR:
Fatal
> > error
> > > > occurred
> > > >
> > > > The bolded item does not exist. When Perry runs his script it
skips
> > this
> > > > file. When I run the script I get a fatal error. I cloned
metplus
> > just
> > > > the other day. I'm not sure if this is related to item 2 from
your
> > email
> > > >
> > > > On Tue, May 5, 2020 at 2:37 PM George McCabe via RT <
> met_help at ucar.edu
> > >
> > > > wrote:
> > > >
> > > > > Hi Edward,
> > > > >
> > > > > 1) The LOG_MET_VERBOSITY variable sets the verbosity level
when a
> MET
> > > > tool
> > > > > is run. To change the METplus logging level, set LOG_LEVEL.
I would
> > > > > recommend changing it to DEBUG to get more information.
> > > > >
> > > > > 2) There is a bug in using the {now} tags for
> [INIT/VALID]_[BEG/END]
> > > that
> > > > > causes that crash. This has been fixed but we have not yet
> released a
> > > > > bugfix release. I will let the engineers know that you need
this
> > > > > functionality and we will push the METplus 3.0.1 bugfix
release
> soon.
> > > You
> > > > > should be able to set the time explicitly, i.e. 2020050500.
> > > > >
> > > > > 3) The {now} tag can be used in the VALID_BEG and VALID_END
config
> > > > > variables, but you will want to refer to init, valid, lead,
etc. in
> > > your
> > > > > INPUT_TEMPLATE and OUTPUT_TEMPLATE variables so that it uses
the
> time
> > > for
> > > > > each time loop. See this section for more information:
> > > > >
> > > > >
> > > >
> > >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#directory-
and-filename-template-info
> > > > >
> > > > > 4) The LEAD_SEQ is a list of forecast leads that will be
processed
> > for
> > > > each
> > > > > run time. If you set this to 0 or unset it, then it will
just
> process
> > > > once
> > > > > for each run time.
> > > > >
> > > > > 5) There is documentation describing the time looping
variables
> here:
> > > > >
> > > > >
> > > >
> > >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#common-
config-variables
> > > > >
> > > > > This page should answer some of your questions on how this
works.
> > > > >
> > > > > 6) I would recommend starting with the Example use case
found in
> > > > > parm/use_cases/met_tool_wrapper/Example/Example.conf. This
use case
> > > loops
> > > > > over times, fills in the filename templates based on each
run time,
> > and
> > > > > prints information to the logfile. You can copy this use
case and
> > > modify
> > > > it
> > > > > to see which filenames are searched for each run time. This
is a
> good
> > > way
> > > > > to understand how to configure METplus to find the files you
need.
> > Once
> > > > > this makes sense, you can look at the PB2NC.conf use case to
see
> how
> > > the
> > > > > config variables control that wrapper.
> > > > >
> > > > > 7) METplus sets environment variables based on its
configuration
> > files.
> > > > > These environment variables are referenced in the MET config
file
> to
> > > set
> > > > > configurations in that file. If you turn on DEBUG logging,
you can
> > see
> > > > what
> > > > > values are set for these environment variables to get a
better idea
> > of
> > > > how
> > > > > your METplus configuration will affect the MET
configuration.
> > > > >
> > > > > Please let me know if you have any questions regarding any
of this.
> > > > >
> > > > > On Tue, May 5, 2020 at 11:46 AM Edward Strobach - NOAA
Affiliate
> via
> > > RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> > > > > >
> > > > > > Update:
> > > > > >
> > > > > > Trying to follow the log files that are generated for
guidance.
> I
> > > set
> > > > > the
> > > > > > verbosity to 7, but still can't understand why I'm
receiving this
> > > > > message:
> > > > > >
> > > > > > 05/05 17:41:59.861 metplus INFO: Running METplus v3.0
called with
> > > > > command:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/master_metplus.py
> > > > > > -c
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/use_cases/model_applications/air_quality/CMAQ_obsAirnow.conf
> > > > > > 05/05 17:41:59.882 metplus (met_util.py:776) ERROR:
> > > > [INIT/VALID]_TIME_FMT
> > > > > > (%Y%m%d%H) does not match [INIT/VALID]_[BEG/END]
(2020050200
> > > > > #2020050317)
> > > > > > 05/05 17:41:59.886 metplus.PB2NC (met_util.py:192) ERROR:
Fatal
> > error
> > > > > > occurred
> > > > > >
> > > > > > I tried to start basic. If starting basic worked, then I
would
> > have
> > > > made
> > > > > > it more complex:
> > > > > >
> > > > > > # time looping - options are INIT, VALID, RETRO, and
REALTIME
> > > > > > LOG_MET_VERBOSITY = 7
> > > > > >
> > > > > > LOOP_BY = INIT
> > > > > >
> > > > > > # Format of VALID_BEG and VALID_END
> > > > > > INIT_TIME_FMT = %Y%m%d%H
> > > > > >
> > > > > > # Start time, end time, and time increment for METplus run
> > > > > > INIT_BEG = 2020050200 #{now?fmt=%Y%m%d%H?shift=-172800}
> > > > > > INIT_END = 2020050200 #{now?fmt=%Y%m%d%H?shift=-172800}
> #-21600}
> > > > > > INIT_INCREMENT = 0
> > > > > > LEAD_SEQ = 0
> > > > > >
> > > > > > With how this looks, I would have thought that only one
file
> would
> > be
> > > > > > processed. But it complains about a mismatch between the
> > > INIT_TIME_FMT
> > > > > and
> > > > > > INIT_BEG/INIT_END
> > > > > >
> > > > > > On Tue, May 5, 2020 at 12:01 PM met_help at ucar.edu via RT <
> > > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > Greetings,
> > > > > > >
> > > > > > > This message has been automatically generated in
response to
> the
> > > > > creation
> > > > > > > of a trouble ticket regarding:
> > > > > > > "PB2NC_OFFSETS/related questions",
> > > > > > > a summary of which appears below.
> > > > > > >
> > > > > > > There is no need to reply to this message right now.
Your
> ticket
> > > has
> > > > > > been
> > > > > > > assigned an ID of [rt.rap.ucar.edu #95164].
> > > > > > >
> > > > > > > Please include the string:
> > > > > > >
> > > > > > > [rt.rap.ucar.edu #95164]
> > > > > > >
> > > > > > > in the subject line of all future correspondence about
this
> > issue.
> > > To
> > > > > do
> > > > > > > so, you may reply to this message.
> > > > > > >
> > > > > > > For more information, please see:
> > > > > > >
> > > > > > > MET Online Tutorial:
> > > > > > >
> > > > >
> https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > > > > > >
> > > > > > > MET Users Guide:
> > > > > > > https://www.dtcenter.org/met/users/docs/overview.php
> > > > > > >
> > > > > > > MET FAQs:
> > > > > > >
https://www.dtcenter.org/met/users/support/faqs/index.php
> > > > > > >
> > > > > > > MET-Help Email Archive:
> > > > > > > http://mailman.ucar.edu/pipermail/met_help
> > > > > > >
> > > > > > > Thank you,
> > > > > > > met_help at ucar.edu
> > > > > > >
> > > > > > >
> > > > >
> > >
>
-------------------------------------------------------------------------
> > > > > > > Good morning,
> > > > > > >
> > > > > > > I'm trying to configure a file to process prepbufr data
to
> netcdf
> > > > > format.
> > > > > > > There are several files that are within the directory
that I
> want
> > > to
> > > > > > > process as shown by the bolded text from the
> > > > > > > /gpfs/dell1/nco/ops/com/hourly/prod/hourly.20200503
directory:
> > > > > > >
> > > > > > > aqm.t12z.airnow.tm00.bufr_d aqm.t12z.anowpm.pb.tm120
> > > > > > > aqm.t12z.status.tm024.bufr_d
> > > > > > > aqm.t12z.airnow.tm024.bufr_d *aqm.t12z.prepbufr.tm00 *
> > > > > > > aqm.t12z.status.tm120.bufr_d
> > > > > > > aqm.t12z.airnow.tm120.bufr_d *aqm.t12z.prepbufr.tm024*
> > > > > > > obsproc_version_for_t12z_aqm_dump_run
> > > > > > > *aqm.t12z.anowpm.pb.tm00* aqm.t12z.prepbufr.tm120
> > > > > > > where_t12z_aqm_dump_ran
> > > > > > > *aqm.t12z.anowpm.pb.tm024 *
aqm.t12z.status.tm00.bufr_d
> > > > > > >
> > > > > > > Right now I'm focusing only on the *anowpm* files (two
files).
> > The
> > > > > tm00
> > > > > > is
> > > > > > > time centered on 12z for 05/03/2020, with DHR = ranging
from
> -11
> > to
> > > > 12,
> > > > > > > covering the whole day. Similarly, the tm024 file is
centered
> at
> > > > 12z,
> > > > > > but
> > > > > > > for 05/02/2020. I looked through the metplus glossary
and
> > examples
> > > > to
> > > > > > help
> > > > > > > with the process of building the configure file. If I
run this
> > > > script
> > > > > > and
> > > > > > > desire the files from 05/03/2020, then I should set the
> > following:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > *[config]# time looping - options are INIT, VALID,
RETRO, and
> > > > > > > REALTIMELOOP_BY = VALID# Format of VALID_BEG and
> > > > > VALID_ENDVALID_TIME_FMT
> > > > > > =
> > > > > > > %Y%m%d%H# Start time, end time, and time increment for
METplus
> > > > > > runVALID_BEG
> > > > > > > = {now?fmt=%Y%m%d%H?shift=-2d}VALID_END =
> > > > {now?fmt=%Y%m%d%H?shift=-2d}
> > > > > > > #-21600}VALID_INCREMENT = 24HLEAD_SEQ = 0,24 (((NOT
SURE I
> > NEED
> > > > > THIS,
> > > > > > > BUT NOT SURE HOW THE LOOP WILL WORK WITHOUT IT?)))*
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > *LOOP_ORDER = times# List of applications to
runPROCESS_LIST =
> > > > > > > PB2NCCONFIG_DIR =
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/met_configCONFIG_FILE
> > > > > > > =
{CONFIG_DIR}/PB2NCConfig_wrappedPB2NC_OBS_BUFR_VAR_LIST =
> DHR,
> > > RPT,
> > > > > > SID,
> > > > > > > YOB, XOB, TYPO, TPHR, QCIND, COPO, COPOPM*
> > > > > > >
> > > > > > > I've listed the variables I would like to extract from
each
> file
> > as
> > > > > well.
> > > > > > > It's not clear if I need the config file, but I listed
it just
> in
> > > > case.
> > > > > > > Here's what I have for the file names from the input
directory
> > and
> > > > > what I
> > > > > > > would name it once converted to netcdf and placed in the
output
> > > > > > directory:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
*INPUT_BASE=/gpfs/dell1/nco/ops/com/hourly/prod/hourly.{now?fmt=%Y%m%d?shift=-
2d}
> > > > > > > # -86400}PB2NC_INPUT_DIR = {INPUT_BASE}OUTPUT_BASE =
> > > > > > >
> /gpfs/dell2/emc/modeling/noscrub/Edward.StrobachPB2NC_OUTPUT_DIR
> > =
> > > > > > >
{OUTPUT_BASE}/AirNow/pb2nc/hourly.{now?fmt=%Y%m%d?shift=-2d} #
> > > > > > > -86400}[filename_templates]PB2NC_INPUT_TEMPLATE =
> > > > > > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ} (((RELATED TO LOOP
> > > > > > > CONFUSION)))PB2NC_OUTPUT_TEMPLATE =
> > > > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ}.nc*
> > > > > > >
> > > > > > > Let me know if more information is needed. My main
concerns
> here
> > > are
> > > > > 1)
> > > > > > > being able to process all the list files correctly, 2)
making
> > sure
> > > > the
> > > > > > > looping is correct, and 3) making sure that I've
included
> > > everything
> > > > > > that I
> > > > > > > would need to process this step
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Edward Strobach
> > > > > > > EMC/NCEP/NWS/
> > > > > > > IMSG Contractor
> > > > > > > Cubicle#: 2029
> > > > > > > 301-683-3717
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Edward Strobach
> > > > > > EMC/NCEP/NWS/
> > > > > > IMSG Contractor
> > > > > > Cubicle#: 2029
> > > > > > 301-683-3717
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > > --
> > > > Edward Strobach
> > > > EMC/NCEP/NWS/
> > > > IMSG Contractor
> > > > Cubicle#: 2029
> > > > 301-683-3717
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: PB2NC_OFFSETS/related questions
From: George McCabe
Time: Wed May 06 16:46:44 2020
Hi Edward,
Julie was able to login and send me some of your log files and info
while I
was in a meeting.
1) I didn't see a metplus_final.conf file in OUTPUT_BASE, but based on
your
last email, I am guessing you changed the value of METPLUS_CONF to
create
those files you listed. Which of the pb2nc_pointstat conf files was
used
for the runs that are producing the error? Could you send me that one?
When
you run a use case, the name of the final conf is output to the screen
towards the beginning of execution, but unfortunately not added to the
logfile currently.
2) It looks like your master_metplus.py command has a lot of extra
conf
files that don't need to be in there. The grid_to_obs conf files are
from
an old version of METplus, so I would think they would cause errors
using
deprecated variables. The corresponding use case in METplus 3.0 for
the
upper_air.conf use case is:
METplus/parm/use_cases/model_applications/medium_range/PointStat_fcstGFS_obsGDAS_UpperAir_MultiField_PrepBufr.conf
I would start with running that conf with a user config and if that
works,
copy the use case conf to another file and start modifying that
instead of
adding new confs onto the command. That will keep things a little more
organized for you.
3) The error message is very odd. It looks like it was logging just
fine up
until that error, then it was trying to write to a log "file" named
/u/Edward.Strobach and crashing because that is a directory. I'm not
understanding why it would change the log file in the middle of
execution
and on top of that still write the offending error to the log. I am
very
baffled by that. I will try to recreate this error locally.
On Wed, May 6, 2020 at 1:43 PM Edward Strobach - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
>
> So my OUTPUT_BASE is here:
> /gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq.
>
> There is a conf directory where .conf files are stored in aq
> (OUTPUT_BASE/conf/aq) and pm (OUTPUT_BASE/conf/pm). There are some
.conf
> files from these runs.
>
> aq:
> metplus_final_pb2nc_pointstat.conf
> metplus_final_statanalysis_gatherbydate.conf
>
> pm:
> metplus_final_pb2nc_pointstat.conf
> metplus_final_statanalysis_gatherbydate.conf
>
> Any file in particular that you want to see? I don't have the file
you are
> referring to.
>
> On Wed, May 6, 2020 at 3:22 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > That's it, thanks.
> >
> > Hmm, it looks like there is an error trying to add to the log
file? Did
> > this run generate a log file? It would be found in the logs
directory
> under
> > the value you set for OUTPUT_BASE. If there is one, can you send
that to
> > me? Also, please send the metplus_final.conf file in your
OUTPUT_BASE as
> > well.
> >
> >
> > On Wed, May 6, 2020 at 1:03 PM Edward Strobach - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> > >
> > > Sure thing. I think you are referring to this?
> > >
> > > 05/06 17:34:09.831 metplus.PointStat (command_builder.py:100)
ERROR:
> > > (command_builder.py:437) Could not find FCST file
> > >
> >
>
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
> > > using template
> > >
> > >
> >
>
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
> > > 05/06 17:34:09.832 metplus.PointStat (met_util.py:192) ERROR:
Fatal
> error
> > > occurred
> > > *Traceback (most recent call last):*
> > > File
> > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > line 169, in run_metplus
> > > process.run_all_times()
> > > File
> > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > line 751, in run_all_times
> > > util.loop_over_times_and_call(self.config, self)
> > > File
> > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > line 834, in loop_over_times_and_call
> > > process.run_at_time(input_dict)
> > > File
> > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/compare_gridded_wrapper.py",
> > > line 183, in run_at_time
> > > self.run_at_time_once(time_info)
> > > File
> > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/point_stat_wrapper.py",
> > > line 146, in run_at_time_once
> > > model_path = self.find_model(time_info, var_list[0])
> > > File
> > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > line 319, in find_model
> > > return_list=return_list)
> > > File
> > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > line 373, in find_data
> > > return self.find_exact_file(**arg_dict)
> > > File
> > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > line 437, in find_exact_file
> > > self.log_error(msg)
> > > File
> > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > line 100, in log_error
> > >
> > >
>
self.logger.error(f"({os.path.basename(caller.filename)}:{caller.lineno})
> > > {error_string}")
> > > File
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > line 1335, in error
> > > self._log(ERROR, msg, args, **kwargs)
> > > File
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > line 1442, in _log
> > > self.handle(record)
> > > File
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > line 1452, in handle
> > > self.callHandlers(record)
> > > File
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > line 1514, in callHandlers
> > > hdlr.handle(record)
> > > File
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > line 863, in handle
> > > self.emit(record)
> > > File
> > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/produtil/log.py",
> > > line 227, in emit
> > > with open(self._logger,'at') as f:
> > >
> > > On Wed, May 6, 2020 at 2:55 PM George McCabe via RT
<met_help at ucar.edu
> >
> > > wrote:
> > >
> > > > Hi Edward,
> > > >
> > > > When you cloned the repo yourself, are you using the develop
branch
> to
> > > run?
> > > > There may have been a bug introduced that is causing that
fatal
> error.
> > > > Could you send me the stack trace that comes after the fatal
error? I
> > > agree
> > > > that it shouldn't crash if it can't find a file.
> > > >
> > > > On Wed, May 6, 2020 at 12:43 PM Edward Strobach - NOAA
Affiliate via
> > RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> > > > >
> > > > > Thanks George. I've been busy trying to figure things out
and
> forgot
> > > to
> > > > > respond. I talked with Perry about some of these things and
he was
> > > > > surprised that I was receiving this message when running his
> script:
> > > > >
> > > > > 05/06 17:34:09.517 metplus.PointStat INFO: Processing
forecast
> lead 0
> > > > hours
> > > > > 05/06 17:34:09.525 metplus.PointStat
(command_builder.py:407)
> DEBUG:
> > > > > Looking for FCST file
> > > > >
> > > >
> > >
> >
>
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
> > > > > 05/06 17:34:09.831 metplus.PointStat
(command_builder.py:100)
> ERROR:
> > > > > (command_builder.py:437) Could not find FCST file
> > > > > /gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/
> > > > > *aqm.t00z.awpozcon.f00.148.grib2* using template
> > > > >
> > > > >
> > > >
> > >
> >
>
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
> > > > > 05/06 17:34:09.832 metplus.PointStat (met_util.py:192)
ERROR: Fatal
> > > error
> > > > > occurred
> > > > >
> > > > > The bolded item does not exist. When Perry runs his script
it
> skips
> > > this
> > > > > file. When I run the script I get a fatal error. I cloned
metplus
> > > just
> > > > > the other day. I'm not sure if this is related to item 2
from your
> > > email
> > > > >
> > > > > On Tue, May 5, 2020 at 2:37 PM George McCabe via RT <
> > met_help at ucar.edu
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Edward,
> > > > > >
> > > > > > 1) The LOG_MET_VERBOSITY variable sets the verbosity level
when a
> > MET
> > > > > tool
> > > > > > is run. To change the METplus logging level, set
LOG_LEVEL. I
> would
> > > > > > recommend changing it to DEBUG to get more information.
> > > > > >
> > > > > > 2) There is a bug in using the {now} tags for
> > [INIT/VALID]_[BEG/END]
> > > > that
> > > > > > causes that crash. This has been fixed but we have not yet
> > released a
> > > > > > bugfix release. I will let the engineers know that you
need this
> > > > > > functionality and we will push the METplus 3.0.1 bugfix
release
> > soon.
> > > > You
> > > > > > should be able to set the time explicitly, i.e.
2020050500.
> > > > > >
> > > > > > 3) The {now} tag can be used in the VALID_BEG and
VALID_END
> config
> > > > > > variables, but you will want to refer to init, valid,
lead, etc.
> in
> > > > your
> > > > > > INPUT_TEMPLATE and OUTPUT_TEMPLATE variables so that it
uses the
> > time
> > > > for
> > > > > > each time loop. See this section for more information:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#directory-
and-filename-template-info
> > > > > >
> > > > > > 4) The LEAD_SEQ is a list of forecast leads that will be
> processed
> > > for
> > > > > each
> > > > > > run time. If you set this to 0 or unset it, then it will
just
> > process
> > > > > once
> > > > > > for each run time.
> > > > > >
> > > > > > 5) There is documentation describing the time looping
variables
> > here:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#common-
config-variables
> > > > > >
> > > > > > This page should answer some of your questions on how this
works.
> > > > > >
> > > > > > 6) I would recommend starting with the Example use case
found in
> > > > > > parm/use_cases/met_tool_wrapper/Example/Example.conf. This
use
> case
> > > > loops
> > > > > > over times, fills in the filename templates based on each
run
> time,
> > > and
> > > > > > prints information to the logfile. You can copy this use
case and
> > > > modify
> > > > > it
> > > > > > to see which filenames are searched for each run time.
This is a
> > good
> > > > way
> > > > > > to understand how to configure METplus to find the files
you
> need.
> > > Once
> > > > > > this makes sense, you can look at the PB2NC.conf use case
to see
> > how
> > > > the
> > > > > > config variables control that wrapper.
> > > > > >
> > > > > > 7) METplus sets environment variables based on its
configuration
> > > files.
> > > > > > These environment variables are referenced in the MET
config file
> > to
> > > > set
> > > > > > configurations in that file. If you turn on DEBUG logging,
you
> can
> > > see
> > > > > what
> > > > > > values are set for these environment variables to get a
better
> idea
> > > of
> > > > > how
> > > > > > your METplus configuration will affect the MET
configuration.
> > > > > >
> > > > > > Please let me know if you have any questions regarding any
of
> this.
> > > > > >
> > > > > > On Tue, May 5, 2020 at 11:46 AM Edward Strobach - NOAA
Affiliate
> > via
> > > > RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
> >
> > > > > > >
> > > > > > > Update:
> > > > > > >
> > > > > > > Trying to follow the log files that are generated for
guidance.
> > I
> > > > set
> > > > > > the
> > > > > > > verbosity to 7, but still can't understand why I'm
receiving
> this
> > > > > > message:
> > > > > > >
> > > > > > > 05/05 17:41:59.861 metplus INFO: Running METplus v3.0
called
> with
> > > > > > command:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/master_metplus.py
> > > > > > > -c
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/use_cases/model_applications/air_quality/CMAQ_obsAirnow.conf
> > > > > > > 05/05 17:41:59.882 metplus (met_util.py:776) ERROR:
> > > > > [INIT/VALID]_TIME_FMT
> > > > > > > (%Y%m%d%H) does not match [INIT/VALID]_[BEG/END]
(2020050200
> > > > > > #2020050317)
> > > > > > > 05/05 17:41:59.886 metplus.PB2NC (met_util.py:192)
ERROR: Fatal
> > > error
> > > > > > > occurred
> > > > > > >
> > > > > > > I tried to start basic. If starting basic worked, then
I would
> > > have
> > > > > made
> > > > > > > it more complex:
> > > > > > >
> > > > > > > # time looping - options are INIT, VALID, RETRO, and
REALTIME
> > > > > > > LOG_MET_VERBOSITY = 7
> > > > > > >
> > > > > > > LOOP_BY = INIT
> > > > > > >
> > > > > > > # Format of VALID_BEG and VALID_END
> > > > > > > INIT_TIME_FMT = %Y%m%d%H
> > > > > > >
> > > > > > > # Start time, end time, and time increment for METplus
run
> > > > > > > INIT_BEG = 2020050200 #{now?fmt=%Y%m%d%H?shift=-172800}
> > > > > > > INIT_END = 2020050200 #{now?fmt=%Y%m%d%H?shift=-172800}
> > #-21600}
> > > > > > > INIT_INCREMENT = 0
> > > > > > > LEAD_SEQ = 0
> > > > > > >
> > > > > > > With how this looks, I would have thought that only one
file
> > would
> > > be
> > > > > > > processed. But it complains about a mismatch between
the
> > > > INIT_TIME_FMT
> > > > > > and
> > > > > > > INIT_BEG/INIT_END
> > > > > > >
> > > > > > > On Tue, May 5, 2020 at 12:01 PM met_help at ucar.edu via RT
<
> > > > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Greetings,
> > > > > > > >
> > > > > > > > This message has been automatically generated in
response to
> > the
> > > > > > creation
> > > > > > > > of a trouble ticket regarding:
> > > > > > > > "PB2NC_OFFSETS/related questions",
> > > > > > > > a summary of which appears below.
> > > > > > > >
> > > > > > > > There is no need to reply to this message right now.
Your
> > ticket
> > > > has
> > > > > > > been
> > > > > > > > assigned an ID of [rt.rap.ucar.edu #95164].
> > > > > > > >
> > > > > > > > Please include the string:
> > > > > > > >
> > > > > > > > [rt.rap.ucar.edu #95164]
> > > > > > > >
> > > > > > > > in the subject line of all future correspondence about
this
> > > issue.
> > > > To
> > > > > > do
> > > > > > > > so, you may reply to this message.
> > > > > > > >
> > > > > > > > For more information, please see:
> > > > > > > >
> > > > > > > > MET Online Tutorial:
> > > > > > > >
> > > > > >
> >
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > > > > > > >
> > > > > > > > MET Users Guide:
> > > > > > > >
https://www.dtcenter.org/met/users/docs/overview.php
> > > > > > > >
> > > > > > > > MET FAQs:
> > > > > > > >
https://www.dtcenter.org/met/users/support/faqs/index.php
> > > > > > > >
> > > > > > > > MET-Help Email Archive:
> > > > > > > > http://mailman.ucar.edu/pipermail/met_help
> > > > > > > >
> > > > > > > > Thank you,
> > > > > > > > met_help at ucar.edu
> > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >
-------------------------------------------------------------------------
> > > > > > > > Good morning,
> > > > > > > >
> > > > > > > > I'm trying to configure a file to process prepbufr
data to
> > netcdf
> > > > > > format.
> > > > > > > > There are several files that are within the directory
that I
> > want
> > > > to
> > > > > > > > process as shown by the bolded text from the
> > > > > > > > /gpfs/dell1/nco/ops/com/hourly/prod/hourly.20200503
> directory:
> > > > > > > >
> > > > > > > > aqm.t12z.airnow.tm00.bufr_d aqm.t12z.anowpm.pb.tm120
> > > > > > > > aqm.t12z.status.tm024.bufr_d
> > > > > > > > aqm.t12z.airnow.tm024.bufr_d *aqm.t12z.prepbufr.tm00
*
> > > > > > > > aqm.t12z.status.tm120.bufr_d
> > > > > > > > aqm.t12z.airnow.tm120.bufr_d
*aqm.t12z.prepbufr.tm024*
> > > > > > > > obsproc_version_for_t12z_aqm_dump_run
> > > > > > > > *aqm.t12z.anowpm.pb.tm00*
aqm.t12z.prepbufr.tm120
> > > > > > > > where_t12z_aqm_dump_ran
> > > > > > > > *aqm.t12z.anowpm.pb.tm024 *
aqm.t12z.status.tm00.bufr_d
> > > > > > > >
> > > > > > > > Right now I'm focusing only on the *anowpm* files (two
> files).
> > > The
> > > > > > tm00
> > > > > > > is
> > > > > > > > time centered on 12z for 05/03/2020, with DHR =
ranging from
> > -11
> > > to
> > > > > 12,
> > > > > > > > covering the whole day. Similarly, the tm024 file is
> centered
> > at
> > > > > 12z,
> > > > > > > but
> > > > > > > > for 05/02/2020. I looked through the metplus glossary
and
> > > examples
> > > > > to
> > > > > > > help
> > > > > > > > with the process of building the configure file. If I
run
> this
> > > > > script
> > > > > > > and
> > > > > > > > desire the files from 05/03/2020, then I should set
the
> > > following:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > *[config]# time looping - options are INIT, VALID,
RETRO, and
> > > > > > > > REALTIMELOOP_BY = VALID# Format of VALID_BEG and
> > > > > > VALID_ENDVALID_TIME_FMT
> > > > > > > =
> > > > > > > > %Y%m%d%H# Start time, end time, and time increment for
> METplus
> > > > > > > runVALID_BEG
> > > > > > > > = {now?fmt=%Y%m%d%H?shift=-2d}VALID_END =
> > > > > {now?fmt=%Y%m%d%H?shift=-2d}
> > > > > > > > #-21600}VALID_INCREMENT = 24HLEAD_SEQ = 0,24 (((NOT
SURE I
> > > NEED
> > > > > > THIS,
> > > > > > > > BUT NOT SURE HOW THE LOOP WILL WORK WITHOUT IT?)))*
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > *LOOP_ORDER = times# List of applications to
runPROCESS_LIST
> =
> > > > > > > > PB2NCCONFIG_DIR =
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/met_configCONFIG_FILE
> > > > > > > > =
{CONFIG_DIR}/PB2NCConfig_wrappedPB2NC_OBS_BUFR_VAR_LIST =
> > DHR,
> > > > RPT,
> > > > > > > SID,
> > > > > > > > YOB, XOB, TYPO, TPHR, QCIND, COPO, COPOPM*
> > > > > > > >
> > > > > > > > I've listed the variables I would like to extract from
each
> > file
> > > as
> > > > > > well.
> > > > > > > > It's not clear if I need the config file, but I listed
it
> just
> > in
> > > > > case.
> > > > > > > > Here's what I have for the file names from the input
> directory
> > > and
> > > > > > what I
> > > > > > > > would name it once converted to netcdf and placed in
the
> output
> > > > > > > directory:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
*INPUT_BASE=/gpfs/dell1/nco/ops/com/hourly/prod/hourly.{now?fmt=%Y%m%d?shift=-
2d}
> > > > > > > > # -86400}PB2NC_INPUT_DIR = {INPUT_BASE}OUTPUT_BASE =
> > > > > > > >
> > /gpfs/dell2/emc/modeling/noscrub/Edward.StrobachPB2NC_OUTPUT_DIR
> > > =
> > > > > > > >
{OUTPUT_BASE}/AirNow/pb2nc/hourly.{now?fmt=%Y%m%d?shift=-2d}
> #
> > > > > > > > -86400}[filename_templates]PB2NC_INPUT_TEMPLATE =
> > > > > > > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ} (((RELATED TO LOOP
> > > > > > > > CONFUSION)))PB2NC_OUTPUT_TEMPLATE =
> > > > > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ}.nc*
> > > > > > > >
> > > > > > > > Let me know if more information is needed. My main
concerns
> > here
> > > > are
> > > > > > 1)
> > > > > > > > being able to process all the list files correctly, 2)
making
> > > sure
> > > > > the
> > > > > > > > looping is correct, and 3) making sure that I've
included
> > > > everything
> > > > > > > that I
> > > > > > > > would need to process this step
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Edward Strobach
> > > > > > > > EMC/NCEP/NWS/
> > > > > > > > IMSG Contractor
> > > > > > > > Cubicle#: 2029
> > > > > > > > 301-683-3717
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Edward Strobach
> > > > > > > EMC/NCEP/NWS/
> > > > > > > IMSG Contractor
> > > > > > > Cubicle#: 2029
> > > > > > > 301-683-3717
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Edward Strobach
> > > > > EMC/NCEP/NWS/
> > > > > IMSG Contractor
> > > > > Cubicle#: 2029
> > > > > 301-683-3717
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: PB2NC_OFFSETS/related questions
From: Edward Strobach - NOAA Affiliate
Time: Thu May 07 07:32:18 2020
1) I didn't see a metplus_final.conf file in OUTPUT_BASE, but based on
your
last email, I am guessing you changed the value of METPLUS_CONF to
create
those files you listed. Which of the pb2nc_pointstat conf files was
used
for the runs that are producing the error? Could you send me that one?
When
you run a use case, the name of the final conf is output to the screen
towards the beginning of execution, but unfortunately not added to the
logfile currently.
They both produce the error. One of them looks for an ozone file which
doesn't exist (aqm.t00z.awpozcon.f00.148.grib2) and the other looks
for a
pm file which doesn't exist (aqm.t00z.pm25.f00.148.grib2). *f00* are
not
saved as output from the CMAQ runs while the others (*fXY* ) are being
saved. I have *.conf files that are created for each process
(shared.conf_aq; aq.conf; pm.conf; and statanalysis.conf). These are
based
on inputs from my runs script. These are located here:
/gpfs/dell2/ptmp/Edward.Strobach/metplus_aq
When running the master_metplus.py command, I use several *.conf
files: *${MET_PLUS}/ush/master_metplus.py
-c ${MET_PLUS}/parm/use_cases/grid_to_obs/grid_to_obs.conf -c
${MET_PLUS}/parm/use_cases/grid_to_obs/examples/conus_surface.conf -c
${MET_PLUS_CONF}/pb2nc_aq.conf -c ${MET_PLUS_CONF}/point_stat_aq.conf
-c
${MET_PLUS_TMP}/${model}.conf -c ${MET_PLUS_CONF}/shared.conf -c
${MET_PLUS_TMP}/shared.conf_aq -c ${MET_PLUS_CONF}/system_aq.conf*
Here, model stands for "aq" or "pm"
Lastly, all the conf files that are stored in the OUTPUT_BASE/conf can
be
found in the aq directory (two of them) and pm directory (2 of them).
There is a log file located here:
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/logs/aq/20200504
and
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/logs/pm/20200504.
Both of them have this error message in them: 05/06 17:36:43.877
metplus.StatAnalysis (stat_analysis_wrapper.py:237) DEBUG: Items in
these
lists will be looped over: FCST_VALID_HOUR_LIST
This, to me, is clearly related to the issue of trying to find that
file
that doesn't exist. I'm not 100%, but I think the conf file that you
want
is this: metplus_final_statanalysis_gatherbydate.conf. I copied over
the
contents and stored them in the attachment below. I'm still trying to
understand what exactly is being processed based on the contents
inside.
2) It looks like your master_metplus.py command has a lot of extra
conf
files that don't need to be in there. The grid_to_obs conf files are
from
an old version of METplus, so I would think they would cause errors
using
deprecated variables. The corresponding use case in METplus 3.0 for
the
upper_air.conf use case is:
METplus/parm/use_cases/model_applications/medium_range/Point
Stat_fcstGFS_obsGDAS_UpperAir_MultiField_PrepBufr.conf
I don't really know, but I would assume that there would be specific
errors
related to deprecated variables that would suggest as much. I didn't
see
that though. I could attach the valid config helper, though, just in
case,
right?
I would start with running that conf with a user config and if that
works,
copy the use case conf to another file I and start modifying that
instead of
adding new confs onto the command. That will keep things a little more
organized for you.
I can look into trying that. I'm not very comfortable yet with metplus
to
make serious changes, so it might take a little bit of time to change
the
set-up and make a test.
3) The error message is very odd. It looks like it was logging just
fine up
until that error, then it was trying to write to a log "file" named
/u/Edward.Strobach and crashing because that is a directory. I'm not
understanding why it would change the log file in the middle of
execution
and on top of that still write the offending error to the log. I am
very
baffled by that. I will try to recreate this error locally.
Could it be how I have things set in my .bashrc? I have these related
settings:
module load python/3.6.3
*# Add METplus to $PATHmodule use
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/modulefiles/module
load met/9.0module load NCO/4.7.0module load grib_util/1.1.1export
PATH=${PATH}:/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/METplus/METplus-
3.0/ushexport
PATH=~/METplus/ush:$PATH# Optional JLOGFILEexport JLOGFILE=~/module
load
prod_util/1.1.3*
On Wed, May 6, 2020 at 6:46 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Edward,
>
> Julie was able to login and send me some of your log files and info
while I
> was in a meeting.
>
> 1) I didn't see a metplus_final.conf file in OUTPUT_BASE, but based
on your
> last email, I am guessing you changed the value of METPLUS_CONF to
create
> those files you listed. Which of the pb2nc_pointstat conf files was
used
> for the runs that are producing the error? Could you send me that
one? When
> you run a use case, the name of the final conf is output to the
screen
> towards the beginning of execution, but unfortunately not added to
the
> logfile currently.
>
>
> 2) It looks like your master_metplus.py command has a lot of extra
conf
> files that don't need to be in there. The grid_to_obs conf files are
from
> an old version of METplus, so I would think they would cause errors
using
> deprecated variables. The corresponding use case in METplus 3.0 for
the
> upper_air.conf use case is:
>
>
>
METplus/parm/use_cases/model_applications/medium_range/PointStat_fcstGFS_obsGDAS_UpperAir_MultiField_PrepBufr.conf
>
> I would start with running that conf with a user config and if that
works,
> copy the use case conf to another file and start modifying that
instead of
> adding new confs onto the command. That will keep things a little
more
> organized for you.
>
>
> 3) The error message is very odd. It looks like it was logging just
fine up
> until that error, then it was trying to write to a log "file" named
> /u/Edward.Strobach and crashing because that is a directory. I'm not
> understanding why it would change the log file in the middle of
execution
> and on top of that still write the offending error to the log. I am
very
> baffled by that. I will try to recreate this error locally.
>
> On Wed, May 6, 2020 at 1:43 PM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> >
> > So my OUTPUT_BASE is here:
> > /gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq.
> >
> > There is a conf directory where .conf files are stored in aq
> > (OUTPUT_BASE/conf/aq) and pm (OUTPUT_BASE/conf/pm). There are
some .conf
> > files from these runs.
> >
> > aq:
> > metplus_final_pb2nc_pointstat.conf
> > metplus_final_statanalysis_gatherbydate.conf
> >
> > pm:
> > metplus_final_pb2nc_pointstat.conf
> > metplus_final_statanalysis_gatherbydate.conf
> >
> > Any file in particular that you want to see? I don't have the
file you
> are
> > referring to.
> >
> > On Wed, May 6, 2020 at 3:22 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > That's it, thanks.
> > >
> > > Hmm, it looks like there is an error trying to add to the log
file? Did
> > > this run generate a log file? It would be found in the logs
directory
> > under
> > > the value you set for OUTPUT_BASE. If there is one, can you send
that
> to
> > > me? Also, please send the metplus_final.conf file in your
OUTPUT_BASE
> as
> > > well.
> > >
> > >
> > > On Wed, May 6, 2020 at 1:03 PM Edward Strobach - NOAA Affiliate
via RT
> <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
>
> > > >
> > > > Sure thing. I think you are referring to this?
> > > >
> > > > 05/06 17:34:09.831 metplus.PointStat (command_builder.py:100)
ERROR:
> > > > (command_builder.py:437) Could not find FCST file
> > > >
> > >
> >
>
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
> > > > using template
> > > >
> > > >
> > >
> >
>
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
> > > > 05/06 17:34:09.832 metplus.PointStat (met_util.py:192) ERROR:
Fatal
> > error
> > > > occurred
> > > > *Traceback (most recent call last):*
> > > > File
> > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > line 169, in run_metplus
> > > > process.run_all_times()
> > > > File
> > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > line 751, in run_all_times
> > > > util.loop_over_times_and_call(self.config, self)
> > > > File
> > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > line 834, in loop_over_times_and_call
> > > > process.run_at_time(input_dict)
> > > > File
> > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/compare_gridded_wrapper.py",
> > > > line 183, in run_at_time
> > > > self.run_at_time_once(time_info)
> > > > File
> > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/point_stat_wrapper.py",
> > > > line 146, in run_at_time_once
> > > > model_path = self.find_model(time_info, var_list[0])
> > > > File
> > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > line 319, in find_model
> > > > return_list=return_list)
> > > > File
> > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > line 373, in find_data
> > > > return self.find_exact_file(**arg_dict)
> > > > File
> > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > line 437, in find_exact_file
> > > > self.log_error(msg)
> > > > File
> > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > line 100, in log_error
> > > >
> > > >
> >
self.logger.error(f"({os.path.basename(caller.filename)}:{caller.lineno})
> > > > {error_string}")
> > > > File
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > line 1335, in error
> > > > self._log(ERROR, msg, args, **kwargs)
> > > > File
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > line 1442, in _log
> > > > self.handle(record)
> > > > File
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > line 1452, in handle
> > > > self.callHandlers(record)
> > > > File
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > line 1514, in callHandlers
> > > > hdlr.handle(record)
> > > > File
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > line 863, in handle
> > > > self.emit(record)
> > > > File
> > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/produtil/log.py",
> > > > line 227, in emit
> > > > with open(self._logger,'at') as f:
> > > >
> > > > On Wed, May 6, 2020 at 2:55 PM George McCabe via RT <
> met_help at ucar.edu
> > >
> > > > wrote:
> > > >
> > > > > Hi Edward,
> > > > >
> > > > > When you cloned the repo yourself, are you using the develop
branch
> > to
> > > > run?
> > > > > There may have been a bug introduced that is causing that
fatal
> > error.
> > > > > Could you send me the stack trace that comes after the fatal
> error? I
> > > > agree
> > > > > that it shouldn't crash if it can't find a file.
> > > > >
> > > > > On Wed, May 6, 2020 at 12:43 PM Edward Strobach - NOAA
Affiliate
> via
> > > RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> > > > > >
> > > > > > Thanks George. I've been busy trying to figure things out
and
> > forgot
> > > > to
> > > > > > respond. I talked with Perry about some of these things
and he
> was
> > > > > > surprised that I was receiving this message when running
his
> > script:
> > > > > >
> > > > > > 05/06 17:34:09.517 metplus.PointStat INFO: Processing
forecast
> > lead 0
> > > > > hours
> > > > > > 05/06 17:34:09.525 metplus.PointStat
(command_builder.py:407)
> > DEBUG:
> > > > > > Looking for FCST file
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
> > > > > > 05/06 17:34:09.831 metplus.PointStat
(command_builder.py:100)
> > ERROR:
> > > > > > (command_builder.py:437) Could not find FCST file
> > > > > > /gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/
> > > > > > *aqm.t00z.awpozcon.f00.148.grib2* using template
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
> > > > > > 05/06 17:34:09.832 metplus.PointStat (met_util.py:192)
ERROR:
> Fatal
> > > > error
> > > > > > occurred
> > > > > >
> > > > > > The bolded item does not exist. When Perry runs his
script it
> > skips
> > > > this
> > > > > > file. When I run the script I get a fatal error. I
cloned
> metplus
> > > > just
> > > > > > the other day. I'm not sure if this is related to item 2
from
> your
> > > > email
> > > > > >
> > > > > > On Tue, May 5, 2020 at 2:37 PM George McCabe via RT <
> > > met_help at ucar.edu
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Edward,
> > > > > > >
> > > > > > > 1) The LOG_MET_VERBOSITY variable sets the verbosity
level
> when a
> > > MET
> > > > > > tool
> > > > > > > is run. To change the METplus logging level, set
LOG_LEVEL. I
> > would
> > > > > > > recommend changing it to DEBUG to get more information.
> > > > > > >
> > > > > > > 2) There is a bug in using the {now} tags for
> > > [INIT/VALID]_[BEG/END]
> > > > > that
> > > > > > > causes that crash. This has been fixed but we have not
yet
> > > released a
> > > > > > > bugfix release. I will let the engineers know that you
need
> this
> > > > > > > functionality and we will push the METplus 3.0.1 bugfix
release
> > > soon.
> > > > > You
> > > > > > > should be able to set the time explicitly, i.e.
2020050500.
> > > > > > >
> > > > > > > 3) The {now} tag can be used in the VALID_BEG and
VALID_END
> > config
> > > > > > > variables, but you will want to refer to init, valid,
lead,
> etc.
> > in
> > > > > your
> > > > > > > INPUT_TEMPLATE and OUTPUT_TEMPLATE variables so that it
uses
> the
> > > time
> > > > > for
> > > > > > > each time loop. See this section for more information:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#directory-
and-filename-template-info
> > > > > > >
> > > > > > > 4) The LEAD_SEQ is a list of forecast leads that will be
> > processed
> > > > for
> > > > > > each
> > > > > > > run time. If you set this to 0 or unset it, then it will
just
> > > process
> > > > > > once
> > > > > > > for each run time.
> > > > > > >
> > > > > > > 5) There is documentation describing the time looping
variables
> > > here:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#common-
config-variables
> > > > > > >
> > > > > > > This page should answer some of your questions on how
this
> works.
> > > > > > >
> > > > > > > 6) I would recommend starting with the Example use case
found
> in
> > > > > > > parm/use_cases/met_tool_wrapper/Example/Example.conf.
This use
> > case
> > > > > loops
> > > > > > > over times, fills in the filename templates based on
each run
> > time,
> > > > and
> > > > > > > prints information to the logfile. You can copy this use
case
> and
> > > > > modify
> > > > > > it
> > > > > > > to see which filenames are searched for each run time.
This is
> a
> > > good
> > > > > way
> > > > > > > to understand how to configure METplus to find the files
you
> > need.
> > > > Once
> > > > > > > this makes sense, you can look at the PB2NC.conf use
case to
> see
> > > how
> > > > > the
> > > > > > > config variables control that wrapper.
> > > > > > >
> > > > > > > 7) METplus sets environment variables based on its
> configuration
> > > > files.
> > > > > > > These environment variables are referenced in the MET
config
> file
> > > to
> > > > > set
> > > > > > > configurations in that file. If you turn on DEBUG
logging, you
> > can
> > > > see
> > > > > > what
> > > > > > > values are set for these environment variables to get a
better
> > idea
> > > > of
> > > > > > how
> > > > > > > your METplus configuration will affect the MET
configuration.
> > > > > > >
> > > > > > > Please let me know if you have any questions regarding
any of
> > this.
> > > > > > >
> > > > > > > On Tue, May 5, 2020 at 11:46 AM Edward Strobach - NOAA
> Affiliate
> > > via
> > > > > RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
> > >
> > > > > > > >
> > > > > > > > Update:
> > > > > > > >
> > > > > > > > Trying to follow the log files that are generated for
> guidance.
> > > I
> > > > > set
> > > > > > > the
> > > > > > > > verbosity to 7, but still can't understand why I'm
receiving
> > this
> > > > > > > message:
> > > > > > > >
> > > > > > > > 05/05 17:41:59.861 metplus INFO: Running METplus v3.0
called
> > with
> > > > > > > command:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/master_metplus.py
> > > > > > > > -c
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/use_cases/model_applications/air_quality/CMAQ_obsAirnow.conf
> > > > > > > > 05/05 17:41:59.882 metplus (met_util.py:776) ERROR:
> > > > > > [INIT/VALID]_TIME_FMT
> > > > > > > > (%Y%m%d%H) does not match [INIT/VALID]_[BEG/END]
(2020050200
> > > > > > > #2020050317)
> > > > > > > > 05/05 17:41:59.886 metplus.PB2NC (met_util.py:192)
ERROR:
> Fatal
> > > > error
> > > > > > > > occurred
> > > > > > > >
> > > > > > > > I tried to start basic. If starting basic worked,
then I
> would
> > > > have
> > > > > > made
> > > > > > > > it more complex:
> > > > > > > >
> > > > > > > > # time looping - options are INIT, VALID, RETRO, and
REALTIME
> > > > > > > > LOG_MET_VERBOSITY = 7
> > > > > > > >
> > > > > > > > LOOP_BY = INIT
> > > > > > > >
> > > > > > > > # Format of VALID_BEG and VALID_END
> > > > > > > > INIT_TIME_FMT = %Y%m%d%H
> > > > > > > >
> > > > > > > > # Start time, end time, and time increment for METplus
run
> > > > > > > > INIT_BEG = 2020050200 #{now?fmt=%Y%m%d%H?shift=-
172800}
> > > > > > > > INIT_END = 2020050200 #{now?fmt=%Y%m%d%H?shift=-
172800}
> > > #-21600}
> > > > > > > > INIT_INCREMENT = 0
> > > > > > > > LEAD_SEQ = 0
> > > > > > > >
> > > > > > > > With how this looks, I would have thought that only
one file
> > > would
> > > > be
> > > > > > > > processed. But it complains about a mismatch between
the
> > > > > INIT_TIME_FMT
> > > > > > > and
> > > > > > > > INIT_BEG/INIT_END
> > > > > > > >
> > > > > > > > On Tue, May 5, 2020 at 12:01 PM met_help at ucar.edu via
RT <
> > > > > > > > met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Greetings,
> > > > > > > > >
> > > > > > > > > This message has been automatically generated in
response
> to
> > > the
> > > > > > > creation
> > > > > > > > > of a trouble ticket regarding:
> > > > > > > > > "PB2NC_OFFSETS/related questions",
> > > > > > > > > a summary of which appears below.
> > > > > > > > >
> > > > > > > > > There is no need to reply to this message right now.
Your
> > > ticket
> > > > > has
> > > > > > > > been
> > > > > > > > > assigned an ID of [rt.rap.ucar.edu #95164].
> > > > > > > > >
> > > > > > > > > Please include the string:
> > > > > > > > >
> > > > > > > > > [rt.rap.ucar.edu #95164]
> > > > > > > > >
> > > > > > > > > in the subject line of all future correspondence
about this
> > > > issue.
> > > > > To
> > > > > > > do
> > > > > > > > > so, you may reply to this message.
> > > > > > > > >
> > > > > > > > > For more information, please see:
> > > > > > > > >
> > > > > > > > > MET Online Tutorial:
> > > > > > > > >
> > > > > > >
> > >
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > > > > > > > >
> > > > > > > > > MET Users Guide:
> > > > > > > > >
https://www.dtcenter.org/met/users/docs/overview.php
> > > > > > > > >
> > > > > > > > > MET FAQs:
> > > > > > > > >
> https://www.dtcenter.org/met/users/support/faqs/index.php
> > > > > > > > >
> > > > > > > > > MET-Help Email Archive:
> > > > > > > > > http://mailman.ucar.edu/pipermail/met_help
> > > > > > > > >
> > > > > > > > > Thank you,
> > > > > > > > > met_help at ucar.edu
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
>
-------------------------------------------------------------------------
> > > > > > > > > Good morning,
> > > > > > > > >
> > > > > > > > > I'm trying to configure a file to process prepbufr
data to
> > > netcdf
> > > > > > > format.
> > > > > > > > > There are several files that are within the
directory that
> I
> > > want
> > > > > to
> > > > > > > > > process as shown by the bolded text from the
> > > > > > > > > /gpfs/dell1/nco/ops/com/hourly/prod/hourly.20200503
> > directory:
> > > > > > > > >
> > > > > > > > > aqm.t12z.airnow.tm00.bufr_d
aqm.t12z.anowpm.pb.tm120
> > > > > > > > > aqm.t12z.status.tm024.bufr_d
> > > > > > > > > aqm.t12z.airnow.tm024.bufr_d
*aqm.t12z.prepbufr.tm00 *
> > > > > > > > > aqm.t12z.status.tm120.bufr_d
> > > > > > > > > aqm.t12z.airnow.tm120.bufr_d
*aqm.t12z.prepbufr.tm024*
> > > > > > > > > obsproc_version_for_t12z_aqm_dump_run
> > > > > > > > > *aqm.t12z.anowpm.pb.tm00*
aqm.t12z.prepbufr.tm120
> > > > > > > > > where_t12z_aqm_dump_ran
> > > > > > > > > *aqm.t12z.anowpm.pb.tm024 *
aqm.t12z.status.tm00.bufr_d
> > > > > > > > >
> > > > > > > > > Right now I'm focusing only on the *anowpm* files
(two
> > files).
> > > > The
> > > > > > > tm00
> > > > > > > > is
> > > > > > > > > time centered on 12z for 05/03/2020, with DHR =
ranging
> from
> > > -11
> > > > to
> > > > > > 12,
> > > > > > > > > covering the whole day. Similarly, the tm024 file
is
> > centered
> > > at
> > > > > > 12z,
> > > > > > > > but
> > > > > > > > > for 05/02/2020. I looked through the metplus
glossary and
> > > > examples
> > > > > > to
> > > > > > > > help
> > > > > > > > > with the process of building the configure file. If
I run
> > this
> > > > > > script
> > > > > > > > and
> > > > > > > > > desire the files from 05/03/2020, then I should set
the
> > > > following:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *[config]# time looping - options are INIT, VALID,
RETRO,
> and
> > > > > > > > > REALTIMELOOP_BY = VALID# Format of VALID_BEG and
> > > > > > > VALID_ENDVALID_TIME_FMT
> > > > > > > > =
> > > > > > > > > %Y%m%d%H# Start time, end time, and time increment
for
> > METplus
> > > > > > > > runVALID_BEG
> > > > > > > > > = {now?fmt=%Y%m%d%H?shift=-2d}VALID_END =
> > > > > > {now?fmt=%Y%m%d%H?shift=-2d}
> > > > > > > > > #-21600}VALID_INCREMENT = 24HLEAD_SEQ = 0,24
(((NOT
> SURE I
> > > > NEED
> > > > > > > THIS,
> > > > > > > > > BUT NOT SURE HOW THE LOOP WILL WORK WITHOUT IT?)))*
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *LOOP_ORDER = times# List of applications to
> runPROCESS_LIST
> > =
> > > > > > > > > PB2NCCONFIG_DIR =
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/met_configCONFIG_FILE
> > > > > > > > > =
{CONFIG_DIR}/PB2NCConfig_wrappedPB2NC_OBS_BUFR_VAR_LIST =
> > > DHR,
> > > > > RPT,
> > > > > > > > SID,
> > > > > > > > > YOB, XOB, TYPO, TPHR, QCIND, COPO, COPOPM*
> > > > > > > > >
> > > > > > > > > I've listed the variables I would like to extract
from each
> > > file
> > > > as
> > > > > > > well.
> > > > > > > > > It's not clear if I need the config file, but I
listed it
> > just
> > > in
> > > > > > case.
> > > > > > > > > Here's what I have for the file names from the input
> > directory
> > > > and
> > > > > > > what I
> > > > > > > > > would name it once converted to netcdf and placed in
the
> > output
> > > > > > > > directory:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
*INPUT_BASE=/gpfs/dell1/nco/ops/com/hourly/prod/hourly.{now?fmt=%Y%m%d?shift=-
2d}
> > > > > > > > > # -86400}PB2NC_INPUT_DIR = {INPUT_BASE}OUTPUT_BASE
=
> > > > > > > > >
> > > /gpfs/dell2/emc/modeling/noscrub/Edward.StrobachPB2NC_OUTPUT_DIR
> > > > =
> > > > > > > > >
> {OUTPUT_BASE}/AirNow/pb2nc/hourly.{now?fmt=%Y%m%d?shift=-2d}
> > #
> > > > > > > > > -86400}[filename_templates]PB2NC_INPUT_TEMPLATE =
> > > > > > > > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ} (((RELATED TO
LOOP
> > > > > > > > > CONFUSION)))PB2NC_OUTPUT_TEMPLATE =
> > > > > > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ}.nc*
> > > > > > > > >
> > > > > > > > > Let me know if more information is needed. My main
> concerns
> > > here
> > > > > are
> > > > > > > 1)
> > > > > > > > > being able to process all the list files correctly,
2)
> making
> > > > sure
> > > > > > the
> > > > > > > > > looping is correct, and 3) making sure that I've
included
> > > > > everything
> > > > > > > > that I
> > > > > > > > > would need to process this step
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Edward Strobach
> > > > > > > > > EMC/NCEP/NWS/
> > > > > > > > > IMSG Contractor
> > > > > > > > > Cubicle#: 2029
> > > > > > > > > 301-683-3717
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Edward Strobach
> > > > > > > > EMC/NCEP/NWS/
> > > > > > > > IMSG Contractor
> > > > > > > > Cubicle#: 2029
> > > > > > > > 301-683-3717
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > George McCabe - Software Engineer III
> > > > > > > National Center for Atmospheric Research
> > > > > > > Research Applications Laboratory
> > > > > > > 303-497-2768
> > > > > > > ---
> > > > > > > My working day may not be your working day. Please do
not feel
> > > > obliged
> > > > > to
> > > > > > > reply to this email outside of your normal working
hours.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Edward Strobach
> > > > > > EMC/NCEP/NWS/
> > > > > > IMSG Contractor
> > > > > > Cubicle#: 2029
> > > > > > 301-683-3717
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > > --
> > > > Edward Strobach
> > > > EMC/NCEP/NWS/
> > > > IMSG Contractor
> > > > Cubicle#: 2029
> > > > 301-683-3717
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: PB2NC_OFFSETS/related questions
From: George McCabe
Time: Thu May 07 09:50:01 2020
Hi Edward,
Thanks for sending all of this information.
I think the fatal error lies in this line in your bashrc:
export JLOGFILE=~/
I would recommend not setting JLOGFILE at all, as it isn't necessary
to run
METplus. I believe it should be set to a filepath to write job logs,
so
setting this to your home directory is causing the fatal error. I am
guessing it only writes to the JLOGFILE when there is an error, and it
can't write to ~/ because it is a directory not a file. Removing that
from
your bashrc should fix the fatal error you are experiencing.
It might be beneficial to set up a meeting with you, me, and John
Halley
Gotway (MET engineer) to share screens and step through your current
setup
to help you figure out how to set up what you want to do. We have done
this
with Perry in the past and it proved to be very productive. Let me
know if
you would like to do that and we can set up a time to meet.
Thanks,
George
On Thu, May 7, 2020 at 7:32 AM Edward Strobach - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
>
> 1) I didn't see a metplus_final.conf file in OUTPUT_BASE, but based
on your
> last email, I am guessing you changed the value of METPLUS_CONF to
create
> those files you listed. Which of the pb2nc_pointstat conf files was
used
> for the runs that are producing the error? Could you send me that
one? When
> you run a use case, the name of the final conf is output to the
screen
> towards the beginning of execution, but unfortunately not added to
the
> logfile currently.
>
> They both produce the error. One of them looks for an ozone file
which
> doesn't exist (aqm.t00z.awpozcon.f00.148.grib2) and the other looks
for a
> pm file which doesn't exist (aqm.t00z.pm25.f00.148.grib2). *f00*
are not
> saved as output from the CMAQ runs while the others (*fXY* ) are
being
> saved. I have *.conf files that are created for each process
> (shared.conf_aq; aq.conf; pm.conf; and statanalysis.conf). These
are based
> on inputs from my runs script. These are located here:
> /gpfs/dell2/ptmp/Edward.Strobach/metplus_aq
>
> When running the master_metplus.py command, I use several *.conf
> files: *${MET_PLUS}/ush/master_metplus.py
> -c ${MET_PLUS}/parm/use_cases/grid_to_obs/grid_to_obs.conf -c
> ${MET_PLUS}/parm/use_cases/grid_to_obs/examples/conus_surface.conf
-c
> ${MET_PLUS_CONF}/pb2nc_aq.conf -c
${MET_PLUS_CONF}/point_stat_aq.conf -c
> ${MET_PLUS_TMP}/${model}.conf -c ${MET_PLUS_CONF}/shared.conf -c
> ${MET_PLUS_TMP}/shared.conf_aq -c ${MET_PLUS_CONF}/system_aq.conf*
>
> Here, model stands for "aq" or "pm"
>
> Lastly, all the conf files that are stored in the OUTPUT_BASE/conf
can be
> found in the aq directory (two of them) and pm directory (2 of
them).
> There is a log file located here:
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/logs/aq/20200504
> and
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/logs/pm/20200504.
> Both of them have this error message in them: 05/06 17:36:43.877
> metplus.StatAnalysis (stat_analysis_wrapper.py:237) DEBUG: Items in
these
> lists will be looped over: FCST_VALID_HOUR_LIST
>
> This, to me, is clearly related to the issue of trying to find that
file
> that doesn't exist. I'm not 100%, but I think the conf file that you
want
> is this: metplus_final_statanalysis_gatherbydate.conf. I copied
over the
> contents and stored them in the attachment below. I'm still trying
to
> understand what exactly is being processed based on the contents
inside.
>
>
> 2) It looks like your master_metplus.py command has a lot of extra
conf
> files that don't need to be in there. The grid_to_obs conf files are
from
> an old version of METplus, so I would think they would cause errors
using
> deprecated variables. The corresponding use case in METplus 3.0 for
the
> upper_air.conf use case is:
>
> METplus/parm/use_cases/model_applications/medium_range/Point
> Stat_fcstGFS_obsGDAS_UpperAir_MultiField_PrepBufr.conf
>
> I don't really know, but I would assume that there would be specific
errors
> related to deprecated variables that would suggest as much. I didn't
see
> that though. I could attach the valid config helper, though, just in
case,
> right?
>
> I would start with running that conf with a user config and if that
works,
> copy the use case conf to another file I and start modifying that
instead
> of
> adding new confs onto the command. That will keep things a little
more
> organized for you.
>
> I can look into trying that. I'm not very comfortable yet with
metplus to
> make serious changes, so it might take a little bit of time to
change the
> set-up and make a test.
>
> 3) The error message is very odd. It looks like it was logging just
fine up
> until that error, then it was trying to write to a log "file" named
> /u/Edward.Strobach and crashing because that is a directory. I'm not
> understanding why it would change the log file in the middle of
execution
> and on top of that still write the offending error to the log. I am
very
> baffled by that. I will try to recreate this error locally.
>
> Could it be how I have things set in my .bashrc? I have these
related
> settings:
> module load python/3.6.3
>
>
>
>
>
>
>
>
>
>
> *# Add METplus to $PATHmodule use
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/modulefiles/module
> load met/9.0module load NCO/4.7.0module load grib_util/1.1.1export
>
>
PATH=${PATH}:/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/METplus/METplus-
3.0/ushexport
> PATH=~/METplus/ush:$PATH# Optional JLOGFILEexport JLOGFILE=~/module
load
> prod_util/1.1.3*
>
>
>
>
> On Wed, May 6, 2020 at 6:46 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Edward,
> >
> > Julie was able to login and send me some of your log files and
info
> while I
> > was in a meeting.
> >
> > 1) I didn't see a metplus_final.conf file in OUTPUT_BASE, but
based on
> your
> > last email, I am guessing you changed the value of METPLUS_CONF to
create
> > those files you listed. Which of the pb2nc_pointstat conf files
was used
> > for the runs that are producing the error? Could you send me that
one?
> When
> > you run a use case, the name of the final conf is output to the
screen
> > towards the beginning of execution, but unfortunately not added to
the
> > logfile currently.
> >
> >
> > 2) It looks like your master_metplus.py command has a lot of extra
conf
> > files that don't need to be in there. The grid_to_obs conf files
are from
> > an old version of METplus, so I would think they would cause
errors using
> > deprecated variables. The corresponding use case in METplus 3.0
for the
> > upper_air.conf use case is:
> >
> >
> >
>
METplus/parm/use_cases/model_applications/medium_range/PointStat_fcstGFS_obsGDAS_UpperAir_MultiField_PrepBufr.conf
> >
> > I would start with running that conf with a user config and if
that
> works,
> > copy the use case conf to another file and start modifying that
instead
> of
> > adding new confs onto the command. That will keep things a little
more
> > organized for you.
> >
> >
> > 3) The error message is very odd. It looks like it was logging
just fine
> up
> > until that error, then it was trying to write to a log "file"
named
> > /u/Edward.Strobach and crashing because that is a directory. I'm
not
> > understanding why it would change the log file in the middle of
execution
> > and on top of that still write the offending error to the log. I
am very
> > baffled by that. I will try to recreate this error locally.
> >
> > On Wed, May 6, 2020 at 1:43 PM Edward Strobach - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> > >
> > > So my OUTPUT_BASE is here:
> > > /gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq.
> > >
> > > There is a conf directory where .conf files are stored in aq
> > > (OUTPUT_BASE/conf/aq) and pm (OUTPUT_BASE/conf/pm). There are
some
> .conf
> > > files from these runs.
> > >
> > > aq:
> > > metplus_final_pb2nc_pointstat.conf
> > > metplus_final_statanalysis_gatherbydate.conf
> > >
> > > pm:
> > > metplus_final_pb2nc_pointstat.conf
> > > metplus_final_statanalysis_gatherbydate.conf
> > >
> > > Any file in particular that you want to see? I don't have the
file you
> > are
> > > referring to.
> > >
> > > On Wed, May 6, 2020 at 3:22 PM George McCabe via RT
<met_help at ucar.edu
> >
> > > wrote:
> > >
> > > > That's it, thanks.
> > > >
> > > > Hmm, it looks like there is an error trying to add to the log
file?
> Did
> > > > this run generate a log file? It would be found in the logs
directory
> > > under
> > > > the value you set for OUTPUT_BASE. If there is one, can you
send that
> > to
> > > > me? Also, please send the metplus_final.conf file in your
OUTPUT_BASE
> > as
> > > > well.
> > > >
> > > >
> > > > On Wed, May 6, 2020 at 1:03 PM Edward Strobach - NOAA
Affiliate via
> RT
> > <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> > > > >
> > > > > Sure thing. I think you are referring to this?
> > > > >
> > > > > 05/06 17:34:09.831 metplus.PointStat
(command_builder.py:100)
> ERROR:
> > > > > (command_builder.py:437) Could not find FCST file
> > > > >
> > > >
> > >
> >
>
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
> > > > > using template
> > > > >
> > > > >
> > > >
> > >
> >
>
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
> > > > > 05/06 17:34:09.832 metplus.PointStat (met_util.py:192)
ERROR: Fatal
> > > error
> > > > > occurred
> > > > > *Traceback (most recent call last):*
> > > > > File
> > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > > line 169, in run_metplus
> > > > > process.run_all_times()
> > > > > File
> > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > line 751, in run_all_times
> > > > > util.loop_over_times_and_call(self.config, self)
> > > > > File
> > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > > line 834, in loop_over_times_and_call
> > > > > process.run_at_time(input_dict)
> > > > > File
> > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/compare_gridded_wrapper.py",
> > > > > line 183, in run_at_time
> > > > > self.run_at_time_once(time_info)
> > > > > File
> > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/point_stat_wrapper.py",
> > > > > line 146, in run_at_time_once
> > > > > model_path = self.find_model(time_info, var_list[0])
> > > > > File
> > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > line 319, in find_model
> > > > > return_list=return_list)
> > > > > File
> > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > line 373, in find_data
> > > > > return self.find_exact_file(**arg_dict)
> > > > > File
> > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > line 437, in find_exact_file
> > > > > self.log_error(msg)
> > > > > File
> > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > line 100, in log_error
> > > > >
> > > > >
> > >
>
self.logger.error(f"({os.path.basename(caller.filename)}:{caller.lineno})
> > > > > {error_string}")
> > > > > File
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > line 1335, in error
> > > > > self._log(ERROR, msg, args, **kwargs)
> > > > > File
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > line 1442, in _log
> > > > > self.handle(record)
> > > > > File
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > line 1452, in handle
> > > > > self.callHandlers(record)
> > > > > File
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > line 1514, in callHandlers
> > > > > hdlr.handle(record)
> > > > > File
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > line 863, in handle
> > > > > self.emit(record)
> > > > > File
> > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/produtil/log.py",
> > > > > line 227, in emit
> > > > > with open(self._logger,'at') as f:
> > > > >
> > > > > On Wed, May 6, 2020 at 2:55 PM George McCabe via RT <
> > met_help at ucar.edu
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Edward,
> > > > > >
> > > > > > When you cloned the repo yourself, are you using the
develop
> branch
> > > to
> > > > > run?
> > > > > > There may have been a bug introduced that is causing that
fatal
> > > error.
> > > > > > Could you send me the stack trace that comes after the
fatal
> > error? I
> > > > > agree
> > > > > > that it shouldn't crash if it can't find a file.
> > > > > >
> > > > > > On Wed, May 6, 2020 at 12:43 PM Edward Strobach - NOAA
Affiliate
> > via
> > > > RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
> >
> > > > > > >
> > > > > > > Thanks George. I've been busy trying to figure things
out and
> > > forgot
> > > > > to
> > > > > > > respond. I talked with Perry about some of these things
and he
> > was
> > > > > > > surprised that I was receiving this message when running
his
> > > script:
> > > > > > >
> > > > > > > 05/06 17:34:09.517 metplus.PointStat INFO: Processing
forecast
> > > lead 0
> > > > > > hours
> > > > > > > 05/06 17:34:09.525 metplus.PointStat
(command_builder.py:407)
> > > DEBUG:
> > > > > > > Looking for FCST file
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
> > > > > > > 05/06 17:34:09.831 metplus.PointStat
(command_builder.py:100)
> > > ERROR:
> > > > > > > (command_builder.py:437) Could not find FCST file
> > > > > > > /gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/
> > > > > > > *aqm.t00z.awpozcon.f00.148.grib2* using template
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
> > > > > > > 05/06 17:34:09.832 metplus.PointStat (met_util.py:192)
ERROR:
> > Fatal
> > > > > error
> > > > > > > occurred
> > > > > > >
> > > > > > > The bolded item does not exist. When Perry runs his
script it
> > > skips
> > > > > this
> > > > > > > file. When I run the script I get a fatal error. I
cloned
> > metplus
> > > > > just
> > > > > > > the other day. I'm not sure if this is related to item
2 from
> > your
> > > > > email
> > > > > > >
> > > > > > > On Tue, May 5, 2020 at 2:37 PM George McCabe via RT <
> > > > met_help at ucar.edu
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Edward,
> > > > > > > >
> > > > > > > > 1) The LOG_MET_VERBOSITY variable sets the verbosity
level
> > when a
> > > > MET
> > > > > > > tool
> > > > > > > > is run. To change the METplus logging level, set
LOG_LEVEL. I
> > > would
> > > > > > > > recommend changing it to DEBUG to get more
information.
> > > > > > > >
> > > > > > > > 2) There is a bug in using the {now} tags for
> > > > [INIT/VALID]_[BEG/END]
> > > > > > that
> > > > > > > > causes that crash. This has been fixed but we have not
yet
> > > > released a
> > > > > > > > bugfix release. I will let the engineers know that you
need
> > this
> > > > > > > > functionality and we will push the METplus 3.0.1
bugfix
> release
> > > > soon.
> > > > > > You
> > > > > > > > should be able to set the time explicitly, i.e.
2020050500.
> > > > > > > >
> > > > > > > > 3) The {now} tag can be used in the VALID_BEG and
VALID_END
> > > config
> > > > > > > > variables, but you will want to refer to init, valid,
lead,
> > etc.
> > > in
> > > > > > your
> > > > > > > > INPUT_TEMPLATE and OUTPUT_TEMPLATE variables so that
it uses
> > the
> > > > time
> > > > > > for
> > > > > > > > each time loop. See this section for more information:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#directory-
and-filename-template-info
> > > > > > > >
> > > > > > > > 4) The LEAD_SEQ is a list of forecast leads that will
be
> > > processed
> > > > > for
> > > > > > > each
> > > > > > > > run time. If you set this to 0 or unset it, then it
will just
> > > > process
> > > > > > > once
> > > > > > > > for each run time.
> > > > > > > >
> > > > > > > > 5) There is documentation describing the time looping
> variables
> > > > here:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#common-
config-variables
> > > > > > > >
> > > > > > > > This page should answer some of your questions on how
this
> > works.
> > > > > > > >
> > > > > > > > 6) I would recommend starting with the Example use
case found
> > in
> > > > > > > > parm/use_cases/met_tool_wrapper/Example/Example.conf.
This
> use
> > > case
> > > > > > loops
> > > > > > > > over times, fills in the filename templates based on
each run
> > > time,
> > > > > and
> > > > > > > > prints information to the logfile. You can copy this
use case
> > and
> > > > > > modify
> > > > > > > it
> > > > > > > > to see which filenames are searched for each run time.
This
> is
> > a
> > > > good
> > > > > > way
> > > > > > > > to understand how to configure METplus to find the
files you
> > > need.
> > > > > Once
> > > > > > > > this makes sense, you can look at the PB2NC.conf use
case to
> > see
> > > > how
> > > > > > the
> > > > > > > > config variables control that wrapper.
> > > > > > > >
> > > > > > > > 7) METplus sets environment variables based on its
> > configuration
> > > > > files.
> > > > > > > > These environment variables are referenced in the MET
config
> > file
> > > > to
> > > > > > set
> > > > > > > > configurations in that file. If you turn on DEBUG
logging,
> you
> > > can
> > > > > see
> > > > > > > what
> > > > > > > > values are set for these environment variables to get
a
> better
> > > idea
> > > > > of
> > > > > > > how
> > > > > > > > your METplus configuration will affect the MET
configuration.
> > > > > > > >
> > > > > > > > Please let me know if you have any questions regarding
any of
> > > this.
> > > > > > > >
> > > > > > > > On Tue, May 5, 2020 at 11:46 AM Edward Strobach - NOAA
> > Affiliate
> > > > via
> > > > > > RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
> > > >
> > > > > > > > >
> > > > > > > > > Update:
> > > > > > > > >
> > > > > > > > > Trying to follow the log files that are generated
for
> > guidance.
> > > > I
> > > > > > set
> > > > > > > > the
> > > > > > > > > verbosity to 7, but still can't understand why I'm
> receiving
> > > this
> > > > > > > > message:
> > > > > > > > >
> > > > > > > > > 05/05 17:41:59.861 metplus INFO: Running METplus
v3.0
> called
> > > with
> > > > > > > > command:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/master_metplus.py
> > > > > > > > > -c
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/use_cases/model_applications/air_quality/CMAQ_obsAirnow.conf
> > > > > > > > > 05/05 17:41:59.882 metplus (met_util.py:776) ERROR:
> > > > > > > [INIT/VALID]_TIME_FMT
> > > > > > > > > (%Y%m%d%H) does not match [INIT/VALID]_[BEG/END]
> (2020050200
> > > > > > > > #2020050317)
> > > > > > > > > 05/05 17:41:59.886 metplus.PB2NC (met_util.py:192)
ERROR:
> > Fatal
> > > > > error
> > > > > > > > > occurred
> > > > > > > > >
> > > > > > > > > I tried to start basic. If starting basic worked,
then I
> > would
> > > > > have
> > > > > > > made
> > > > > > > > > it more complex:
> > > > > > > > >
> > > > > > > > > # time looping - options are INIT, VALID, RETRO, and
> REALTIME
> > > > > > > > > LOG_MET_VERBOSITY = 7
> > > > > > > > >
> > > > > > > > > LOOP_BY = INIT
> > > > > > > > >
> > > > > > > > > # Format of VALID_BEG and VALID_END
> > > > > > > > > INIT_TIME_FMT = %Y%m%d%H
> > > > > > > > >
> > > > > > > > > # Start time, end time, and time increment for
METplus run
> > > > > > > > > INIT_BEG = 2020050200 #{now?fmt=%Y%m%d%H?shift=-
172800}
> > > > > > > > > INIT_END = 2020050200 #{now?fmt=%Y%m%d%H?shift=-
172800}
> > > > #-21600}
> > > > > > > > > INIT_INCREMENT = 0
> > > > > > > > > LEAD_SEQ = 0
> > > > > > > > >
> > > > > > > > > With how this looks, I would have thought that only
one
> file
> > > > would
> > > > > be
> > > > > > > > > processed. But it complains about a mismatch
between the
> > > > > > INIT_TIME_FMT
> > > > > > > > and
> > > > > > > > > INIT_BEG/INIT_END
> > > > > > > > >
> > > > > > > > > On Tue, May 5, 2020 at 12:01 PM met_help at ucar.edu
via RT <
> > > > > > > > > met_help at ucar.edu>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Greetings,
> > > > > > > > > >
> > > > > > > > > > This message has been automatically generated in
response
> > to
> > > > the
> > > > > > > > creation
> > > > > > > > > > of a trouble ticket regarding:
> > > > > > > > > > "PB2NC_OFFSETS/related questions",
> > > > > > > > > > a summary of which appears below.
> > > > > > > > > >
> > > > > > > > > > There is no need to reply to this message right
now.
> Your
> > > > ticket
> > > > > > has
> > > > > > > > > been
> > > > > > > > > > assigned an ID of [rt.rap.ucar.edu #95164].
> > > > > > > > > >
> > > > > > > > > > Please include the string:
> > > > > > > > > >
> > > > > > > > > > [rt.rap.ucar.edu #95164]
> > > > > > > > > >
> > > > > > > > > > in the subject line of all future correspondence
about
> this
> > > > > issue.
> > > > > > To
> > > > > > > > do
> > > > > > > > > > so, you may reply to this message.
> > > > > > > > > >
> > > > > > > > > > For more information, please see:
> > > > > > > > > >
> > > > > > > > > > MET Online Tutorial:
> > > > > > > > > >
> > > > > > > >
> > > >
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > > > > > > > > >
> > > > > > > > > > MET Users Guide:
> > > > > > > > > >
https://www.dtcenter.org/met/users/docs/overview.php
> > > > > > > > > >
> > > > > > > > > > MET FAQs:
> > > > > > > > > >
> > https://www.dtcenter.org/met/users/support/faqs/index.php
> > > > > > > > > >
> > > > > > > > > > MET-Help Email Archive:
> > > > > > > > > > http://mailman.ucar.edu/pipermail/met_help
> > > > > > > > > >
> > > > > > > > > > Thank you,
> > > > > > > > > > met_help at ucar.edu
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >
-------------------------------------------------------------------------
> > > > > > > > > > Good morning,
> > > > > > > > > >
> > > > > > > > > > I'm trying to configure a file to process prepbufr
data
> to
> > > > netcdf
> > > > > > > > format.
> > > > > > > > > > There are several files that are within the
directory
> that
> > I
> > > > want
> > > > > > to
> > > > > > > > > > process as shown by the bolded text from the
> > > > > > > > > >
/gpfs/dell1/nco/ops/com/hourly/prod/hourly.20200503
> > > directory:
> > > > > > > > > >
> > > > > > > > > > aqm.t12z.airnow.tm00.bufr_d
aqm.t12z.anowpm.pb.tm120
> > > > > > > > > > aqm.t12z.status.tm024.bufr_d
> > > > > > > > > > aqm.t12z.airnow.tm024.bufr_d
*aqm.t12z.prepbufr.tm00 *
> > > > > > > > > > aqm.t12z.status.tm120.bufr_d
> > > > > > > > > > aqm.t12z.airnow.tm120.bufr_d
*aqm.t12z.prepbufr.tm024*
> > > > > > > > > > obsproc_version_for_t12z_aqm_dump_run
> > > > > > > > > > *aqm.t12z.anowpm.pb.tm00*
aqm.t12z.prepbufr.tm120
> > > > > > > > > > where_t12z_aqm_dump_ran
> > > > > > > > > > *aqm.t12z.anowpm.pb.tm024 *
> aqm.t12z.status.tm00.bufr_d
> > > > > > > > > >
> > > > > > > > > > Right now I'm focusing only on the *anowpm* files
(two
> > > files).
> > > > > The
> > > > > > > > tm00
> > > > > > > > > is
> > > > > > > > > > time centered on 12z for 05/03/2020, with DHR =
ranging
> > from
> > > > -11
> > > > > to
> > > > > > > 12,
> > > > > > > > > > covering the whole day. Similarly, the tm024 file
is
> > > centered
> > > > at
> > > > > > > 12z,
> > > > > > > > > but
> > > > > > > > > > for 05/02/2020. I looked through the metplus
glossary
> and
> > > > > examples
> > > > > > > to
> > > > > > > > > help
> > > > > > > > > > with the process of building the configure file.
If I
> run
> > > this
> > > > > > > script
> > > > > > > > > and
> > > > > > > > > > desire the files from 05/03/2020, then I should
set the
> > > > > following:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > *[config]# time looping - options are INIT, VALID,
RETRO,
> > and
> > > > > > > > > > REALTIMELOOP_BY = VALID# Format of VALID_BEG and
> > > > > > > > VALID_ENDVALID_TIME_FMT
> > > > > > > > > =
> > > > > > > > > > %Y%m%d%H# Start time, end time, and time increment
for
> > > METplus
> > > > > > > > > runVALID_BEG
> > > > > > > > > > = {now?fmt=%Y%m%d%H?shift=-2d}VALID_END =
> > > > > > > {now?fmt=%Y%m%d%H?shift=-2d}
> > > > > > > > > > #-21600}VALID_INCREMENT = 24HLEAD_SEQ = 0,24
(((NOT
> > SURE I
> > > > > NEED
> > > > > > > > THIS,
> > > > > > > > > > BUT NOT SURE HOW THE LOOP WILL WORK WITHOUT
IT?)))*
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > *LOOP_ORDER = times# List of applications to
> > runPROCESS_LIST
> > > =
> > > > > > > > > > PB2NCCONFIG_DIR =
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/met_configCONFIG_FILE
> > > > > > > > > > =
> {CONFIG_DIR}/PB2NCConfig_wrappedPB2NC_OBS_BUFR_VAR_LIST =
> > > > DHR,
> > > > > > RPT,
> > > > > > > > > SID,
> > > > > > > > > > YOB, XOB, TYPO, TPHR, QCIND, COPO, COPOPM*
> > > > > > > > > >
> > > > > > > > > > I've listed the variables I would like to extract
from
> each
> > > > file
> > > > > as
> > > > > > > > well.
> > > > > > > > > > It's not clear if I need the config file, but I
listed it
> > > just
> > > > in
> > > > > > > case.
> > > > > > > > > > Here's what I have for the file names from the
input
> > > directory
> > > > > and
> > > > > > > > what I
> > > > > > > > > > would name it once converted to netcdf and placed
in the
> > > output
> > > > > > > > > directory:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
*INPUT_BASE=/gpfs/dell1/nco/ops/com/hourly/prod/hourly.{now?fmt=%Y%m%d?shift=-
2d}
> > > > > > > > > > # -86400}PB2NC_INPUT_DIR =
{INPUT_BASE}OUTPUT_BASE =
> > > > > > > > > >
> > > >
/gpfs/dell2/emc/modeling/noscrub/Edward.StrobachPB2NC_OUTPUT_DIR
> > > > > =
> > > > > > > > > >
> > {OUTPUT_BASE}/AirNow/pb2nc/hourly.{now?fmt=%Y%m%d?shift=-2d}
> > > #
> > > > > > > > > > -86400}[filename_templates]PB2NC_INPUT_TEMPLATE =
> > > > > > > > > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ} (((RELATED TO
LOOP
> > > > > > > > > > CONFUSION)))PB2NC_OUTPUT_TEMPLATE =
> > > > > > > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ}.nc*
> > > > > > > > > >
> > > > > > > > > > Let me know if more information is needed. My
main
> > concerns
> > > > here
> > > > > > are
> > > > > > > > 1)
> > > > > > > > > > being able to process all the list files
correctly, 2)
> > making
> > > > > sure
> > > > > > > the
> > > > > > > > > > looping is correct, and 3) making sure that I've
included
> > > > > > everything
> > > > > > > > > that I
> > > > > > > > > > would need to process this step
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Edward Strobach
> > > > > > > > > > EMC/NCEP/NWS/
> > > > > > > > > > IMSG Contractor
> > > > > > > > > > Cubicle#: 2029
> > > > > > > > > > 301-683-3717
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Edward Strobach
> > > > > > > > > EMC/NCEP/NWS/
> > > > > > > > > IMSG Contractor
> > > > > > > > > Cubicle#: 2029
> > > > > > > > > 301-683-3717
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > George McCabe - Software Engineer III
> > > > > > > > National Center for Atmospheric Research
> > > > > > > > Research Applications Laboratory
> > > > > > > > 303-497-2768
> > > > > > > > ---
> > > > > > > > My working day may not be your working day. Please do
not
> feel
> > > > > obliged
> > > > > > to
> > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Edward Strobach
> > > > > > > EMC/NCEP/NWS/
> > > > > > > IMSG Contractor
> > > > > > > Cubicle#: 2029
> > > > > > > 301-683-3717
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Edward Strobach
> > > > > EMC/NCEP/NWS/
> > > > > IMSG Contractor
> > > > > Cubicle#: 2029
> > > > > 301-683-3717
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: PB2NC_OFFSETS/related questions
From: Edward Strobach - NOAA Affiliate
Time: Thu May 07 10:54:24 2020
Removing the JLOGFILE from .bashrc moved me along nicely. I think
everything processed just fine.
In terms of setting up a meeting, that would be great. Sometimes a
nudge
in the right direction is all that is needed. My goals, both short-
term and
long-term, would be run this script, generate plots (I've seen
plot_data_plane and plot_point_obs) both stats and maps, and set this
up in
a batch system so to process it with a cron job so that I have a time
series of stats for different regions, and maps with site overlays.
Eventually I can expand this to other fields if needed. Right now I'm
not
exactly sure how to incorporate plotting utilities in this process,
but I
assume it's just a matter of setting up another config file with path
information, processes to be used, and information related to time. I
have
a meeting in a few minutes but am free otherwise. We could find a
time
that works for all of us, perhaps?
On Thu, May 7, 2020 at 11:56 AM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Edward,
>
> Thanks for sending all of this information.
>
> I think the fatal error lies in this line in your bashrc:
>
> export JLOGFILE=~/
>
> I would recommend not setting JLOGFILE at all, as it isn't necessary
to run
> METplus. I believe it should be set to a filepath to write job logs,
so
> setting this to your home directory is causing the fatal error. I am
> guessing it only writes to the JLOGFILE when there is an error, and
it
> can't write to ~/ because it is a directory not a file. Removing
that from
> your bashrc should fix the fatal error you are experiencing.
>
> It might be beneficial to set up a meeting with you, me, and John
Halley
> Gotway (MET engineer) to share screens and step through your current
setup
> to help you figure out how to set up what you want to do. We have
done this
> with Perry in the past and it proved to be very productive. Let me
know if
> you would like to do that and we can set up a time to meet.
>
> Thanks,
> George
>
> On Thu, May 7, 2020 at 7:32 AM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> >
> > 1) I didn't see a metplus_final.conf file in OUTPUT_BASE, but
based on
> your
> > last email, I am guessing you changed the value of METPLUS_CONF to
create
> > those files you listed. Which of the pb2nc_pointstat conf files
was used
> > for the runs that are producing the error? Could you send me that
one?
> When
> > you run a use case, the name of the final conf is output to the
screen
> > towards the beginning of execution, but unfortunately not added to
the
> > logfile currently.
> >
> > They both produce the error. One of them looks for an ozone file
which
> > doesn't exist (aqm.t00z.awpozcon.f00.148.grib2) and the other
looks for a
> > pm file which doesn't exist (aqm.t00z.pm25.f00.148.grib2). *f00*
are not
> > saved as output from the CMAQ runs while the others (*fXY* ) are
being
> > saved. I have *.conf files that are created for each process
> > (shared.conf_aq; aq.conf; pm.conf; and statanalysis.conf). These
are
> based
> > on inputs from my runs script. These are located here:
> > /gpfs/dell2/ptmp/Edward.Strobach/metplus_aq
> >
> > When running the master_metplus.py command, I use several *.conf
> > files: *${MET_PLUS}/ush/master_metplus.py
> > -c ${MET_PLUS}/parm/use_cases/grid_to_obs/grid_to_obs.conf -c
> > ${MET_PLUS}/parm/use_cases/grid_to_obs/examples/conus_surface.conf
-c
> > ${MET_PLUS_CONF}/pb2nc_aq.conf -c
${MET_PLUS_CONF}/point_stat_aq.conf -c
> > ${MET_PLUS_TMP}/${model}.conf -c ${MET_PLUS_CONF}/shared.conf -c
> > ${MET_PLUS_TMP}/shared.conf_aq -c ${MET_PLUS_CONF}/system_aq.conf*
> >
> > Here, model stands for "aq" or "pm"
> >
> > Lastly, all the conf files that are stored in the OUTPUT_BASE/conf
can be
> > found in the aq directory (two of them) and pm directory (2 of
them).
> > There is a log file located here:
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/logs/aq/20200504
> > and
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/logs/pm/20200504.
> > Both of them have this error message in them: 05/06 17:36:43.877
> > metplus.StatAnalysis (stat_analysis_wrapper.py:237) DEBUG: Items
in these
> > lists will be looped over: FCST_VALID_HOUR_LIST
> >
> > This, to me, is clearly related to the issue of trying to find
that file
> > that doesn't exist. I'm not 100%, but I think the conf file that
you want
> > is this: metplus_final_statanalysis_gatherbydate.conf. I copied
over the
> > contents and stored them in the attachment below. I'm still trying
to
> > understand what exactly is being processed based on the contents
inside.
> >
> >
> > 2) It looks like your master_metplus.py command has a lot of extra
conf
> > files that don't need to be in there. The grid_to_obs conf files
are from
> > an old version of METplus, so I would think they would cause
errors using
> > deprecated variables. The corresponding use case in METplus 3.0
for the
> > upper_air.conf use case is:
> >
> > METplus/parm/use_cases/model_applications/medium_range/Point
> > Stat_fcstGFS_obsGDAS_UpperAir_MultiField_PrepBufr.conf
> >
> > I don't really know, but I would assume that there would be
specific
> errors
> > related to deprecated variables that would suggest as much. I
didn't see
> > that though. I could attach the valid config helper, though, just
in
> case,
> > right?
> >
> > I would start with running that conf with a user config and if
that
> works,
> > copy the use case conf to another file I and start modifying that
instead
> > of
> > adding new confs onto the command. That will keep things a little
more
> > organized for you.
> >
> > I can look into trying that. I'm not very comfortable yet with
metplus to
> > make serious changes, so it might take a little bit of time to
change the
> > set-up and make a test.
> >
> > 3) The error message is very odd. It looks like it was logging
just fine
> up
> > until that error, then it was trying to write to a log "file"
named
> > /u/Edward.Strobach and crashing because that is a directory. I'm
not
> > understanding why it would change the log file in the middle of
execution
> > and on top of that still write the offending error to the log. I
am very
> > baffled by that. I will try to recreate this error locally.
> >
> > Could it be how I have things set in my .bashrc? I have these
related
> > settings:
> > module load python/3.6.3
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *# Add METplus to $PATHmodule use
> >
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/modulefiles/module
> > load met/9.0module load NCO/4.7.0module load grib_util/1.1.1export
> >
> >
>
PATH=${PATH}:/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/METplus/METplus-
3.0/ushexport
> > PATH=~/METplus/ush:$PATH# Optional JLOGFILEexport
JLOGFILE=~/module load
> > prod_util/1.1.3*
> >
> >
> >
> >
> > On Wed, May 6, 2020 at 6:46 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Edward,
> > >
> > > Julie was able to login and send me some of your log files and
info
> > while I
> > > was in a meeting.
> > >
> > > 1) I didn't see a metplus_final.conf file in OUTPUT_BASE, but
based on
> > your
> > > last email, I am guessing you changed the value of METPLUS_CONF
to
> create
> > > those files you listed. Which of the pb2nc_pointstat conf files
was
> used
> > > for the runs that are producing the error? Could you send me
that one?
> > When
> > > you run a use case, the name of the final conf is output to the
screen
> > > towards the beginning of execution, but unfortunately not added
to the
> > > logfile currently.
> > >
> > >
> > > 2) It looks like your master_metplus.py command has a lot of
extra conf
> > > files that don't need to be in there. The grid_to_obs conf files
are
> from
> > > an old version of METplus, so I would think they would cause
errors
> using
> > > deprecated variables. The corresponding use case in METplus 3.0
for the
> > > upper_air.conf use case is:
> > >
> > >
> > >
> >
>
METplus/parm/use_cases/model_applications/medium_range/PointStat_fcstGFS_obsGDAS_UpperAir_MultiField_PrepBufr.conf
> > >
> > > I would start with running that conf with a user config and if
that
> > works,
> > > copy the use case conf to another file and start modifying that
instead
> > of
> > > adding new confs onto the command. That will keep things a
little more
> > > organized for you.
> > >
> > >
> > > 3) The error message is very odd. It looks like it was logging
just
> fine
> > up
> > > until that error, then it was trying to write to a log "file"
named
> > > /u/Edward.Strobach and crashing because that is a directory. I'm
not
> > > understanding why it would change the log file in the middle of
> execution
> > > and on top of that still write the offending error to the log. I
am
> very
> > > baffled by that. I will try to recreate this error locally.
> > >
> > > On Wed, May 6, 2020 at 1:43 PM Edward Strobach - NOAA Affiliate
via RT
> <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
>
> > > >
> > > > So my OUTPUT_BASE is here:
> > > > /gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq.
> > > >
> > > > There is a conf directory where .conf files are stored in aq
> > > > (OUTPUT_BASE/conf/aq) and pm (OUTPUT_BASE/conf/pm). There are
some
> > .conf
> > > > files from these runs.
> > > >
> > > > aq:
> > > > metplus_final_pb2nc_pointstat.conf
> > > > metplus_final_statanalysis_gatherbydate.conf
> > > >
> > > > pm:
> > > > metplus_final_pb2nc_pointstat.conf
> > > > metplus_final_statanalysis_gatherbydate.conf
> > > >
> > > > Any file in particular that you want to see? I don't have the
file
> you
> > > are
> > > > referring to.
> > > >
> > > > On Wed, May 6, 2020 at 3:22 PM George McCabe via RT <
> met_help at ucar.edu
> > >
> > > > wrote:
> > > >
> > > > > That's it, thanks.
> > > > >
> > > > > Hmm, it looks like there is an error trying to add to the
log file?
> > Did
> > > > > this run generate a log file? It would be found in the logs
> directory
> > > > under
> > > > > the value you set for OUTPUT_BASE. If there is one, can you
send
> that
> > > to
> > > > > me? Also, please send the metplus_final.conf file in your
> OUTPUT_BASE
> > > as
> > > > > well.
> > > > >
> > > > >
> > > > > On Wed, May 6, 2020 at 1:03 PM Edward Strobach - NOAA
Affiliate via
> > RT
> > > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> > > > > >
> > > > > > Sure thing. I think you are referring to this?
> > > > > >
> > > > > > 05/06 17:34:09.831 metplus.PointStat
(command_builder.py:100)
> > ERROR:
> > > > > > (command_builder.py:437) Could not find FCST file
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
> > > > > > using template
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
> > > > > > 05/06 17:34:09.832 metplus.PointStat (met_util.py:192)
ERROR:
> Fatal
> > > > error
> > > > > > occurred
> > > > > > *Traceback (most recent call last):*
> > > > > > File
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > > > line 169, in run_metplus
> > > > > > process.run_all_times()
> > > > > > File
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > line 751, in run_all_times
> > > > > > util.loop_over_times_and_call(self.config, self)
> > > > > > File
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > > > line 834, in loop_over_times_and_call
> > > > > > process.run_at_time(input_dict)
> > > > > > File
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/compare_gridded_wrapper.py",
> > > > > > line 183, in run_at_time
> > > > > > self.run_at_time_once(time_info)
> > > > > > File
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/point_stat_wrapper.py",
> > > > > > line 146, in run_at_time_once
> > > > > > model_path = self.find_model(time_info, var_list[0])
> > > > > > File
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > line 319, in find_model
> > > > > > return_list=return_list)
> > > > > > File
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > line 373, in find_data
> > > > > > return self.find_exact_file(**arg_dict)
> > > > > > File
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > line 437, in find_exact_file
> > > > > > self.log_error(msg)
> > > > > > File
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > line 100, in log_error
> > > > > >
> > > > > >
> > > >
> >
self.logger.error(f"({os.path.basename(caller.filename)}:{caller.lineno})
> > > > > > {error_string}")
> > > > > > File
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > line 1335, in error
> > > > > > self._log(ERROR, msg, args, **kwargs)
> > > > > > File
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > line 1442, in _log
> > > > > > self.handle(record)
> > > > > > File
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > line 1452, in handle
> > > > > > self.callHandlers(record)
> > > > > > File
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > line 1514, in callHandlers
> > > > > > hdlr.handle(record)
> > > > > > File
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > line 863, in handle
> > > > > > self.emit(record)
> > > > > > File
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/produtil/log.py",
> > > > > > line 227, in emit
> > > > > > with open(self._logger,'at') as f:
> > > > > >
> > > > > > On Wed, May 6, 2020 at 2:55 PM George McCabe via RT <
> > > met_help at ucar.edu
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Edward,
> > > > > > >
> > > > > > > When you cloned the repo yourself, are you using the
develop
> > branch
> > > > to
> > > > > > run?
> > > > > > > There may have been a bug introduced that is causing
that fatal
> > > > error.
> > > > > > > Could you send me the stack trace that comes after the
fatal
> > > error? I
> > > > > > agree
> > > > > > > that it shouldn't crash if it can't find a file.
> > > > > > >
> > > > > > > On Wed, May 6, 2020 at 12:43 PM Edward Strobach - NOAA
> Affiliate
> > > via
> > > > > RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
> > >
> > > > > > > >
> > > > > > > > Thanks George. I've been busy trying to figure things
out
> and
> > > > forgot
> > > > > > to
> > > > > > > > respond. I talked with Perry about some of these
things and
> he
> > > was
> > > > > > > > surprised that I was receiving this message when
running his
> > > > script:
> > > > > > > >
> > > > > > > > 05/06 17:34:09.517 metplus.PointStat INFO: Processing
> forecast
> > > > lead 0
> > > > > > > hours
> > > > > > > > 05/06 17:34:09.525 metplus.PointStat
(command_builder.py:407)
> > > > DEBUG:
> > > > > > > > Looking for FCST file
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
> > > > > > > > 05/06 17:34:09.831 metplus.PointStat
(command_builder.py:100)
> > > > ERROR:
> > > > > > > > (command_builder.py:437) Could not find FCST file
> > > > > > > > /gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/
> > > > > > > > *aqm.t00z.awpozcon.f00.148.grib2* using template
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
> > > > > > > > 05/06 17:34:09.832 metplus.PointStat (met_util.py:192)
ERROR:
> > > Fatal
> > > > > > error
> > > > > > > > occurred
> > > > > > > >
> > > > > > > > The bolded item does not exist. When Perry runs his
script
> it
> > > > skips
> > > > > > this
> > > > > > > > file. When I run the script I get a fatal error. I
cloned
> > > metplus
> > > > > > just
> > > > > > > > the other day. I'm not sure if this is related to
item 2
> from
> > > your
> > > > > > email
> > > > > > > >
> > > > > > > > On Tue, May 5, 2020 at 2:37 PM George McCabe via RT <
> > > > > met_help at ucar.edu
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Edward,
> > > > > > > > >
> > > > > > > > > 1) The LOG_MET_VERBOSITY variable sets the verbosity
level
> > > when a
> > > > > MET
> > > > > > > > tool
> > > > > > > > > is run. To change the METplus logging level, set
> LOG_LEVEL. I
> > > > would
> > > > > > > > > recommend changing it to DEBUG to get more
information.
> > > > > > > > >
> > > > > > > > > 2) There is a bug in using the {now} tags for
> > > > > [INIT/VALID]_[BEG/END]
> > > > > > > that
> > > > > > > > > causes that crash. This has been fixed but we have
not yet
> > > > > released a
> > > > > > > > > bugfix release. I will let the engineers know that
you need
> > > this
> > > > > > > > > functionality and we will push the METplus 3.0.1
bugfix
> > release
> > > > > soon.
> > > > > > > You
> > > > > > > > > should be able to set the time explicitly, i.e.
2020050500.
> > > > > > > > >
> > > > > > > > > 3) The {now} tag can be used in the VALID_BEG and
VALID_END
> > > > config
> > > > > > > > > variables, but you will want to refer to init,
valid, lead,
> > > etc.
> > > > in
> > > > > > > your
> > > > > > > > > INPUT_TEMPLATE and OUTPUT_TEMPLATE variables so that
it
> uses
> > > the
> > > > > time
> > > > > > > for
> > > > > > > > > each time loop. See this section for more
information:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#directory-
and-filename-template-info
> > > > > > > > >
> > > > > > > > > 4) The LEAD_SEQ is a list of forecast leads that
will be
> > > > processed
> > > > > > for
> > > > > > > > each
> > > > > > > > > run time. If you set this to 0 or unset it, then it
will
> just
> > > > > process
> > > > > > > > once
> > > > > > > > > for each run time.
> > > > > > > > >
> > > > > > > > > 5) There is documentation describing the time
looping
> > variables
> > > > > here:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#common-
config-variables
> > > > > > > > >
> > > > > > > > > This page should answer some of your questions on
how this
> > > works.
> > > > > > > > >
> > > > > > > > > 6) I would recommend starting with the Example use
case
> found
> > > in
> > > > > > > > >
parm/use_cases/met_tool_wrapper/Example/Example.conf. This
> > use
> > > > case
> > > > > > > loops
> > > > > > > > > over times, fills in the filename templates based on
each
> run
> > > > time,
> > > > > > and
> > > > > > > > > prints information to the logfile. You can copy this
use
> case
> > > and
> > > > > > > modify
> > > > > > > > it
> > > > > > > > > to see which filenames are searched for each run
time. This
> > is
> > > a
> > > > > good
> > > > > > > way
> > > > > > > > > to understand how to configure METplus to find the
files
> you
> > > > need.
> > > > > > Once
> > > > > > > > > this makes sense, you can look at the PB2NC.conf use
case
> to
> > > see
> > > > > how
> > > > > > > the
> > > > > > > > > config variables control that wrapper.
> > > > > > > > >
> > > > > > > > > 7) METplus sets environment variables based on its
> > > configuration
> > > > > > files.
> > > > > > > > > These environment variables are referenced in the
MET
> config
> > > file
> > > > > to
> > > > > > > set
> > > > > > > > > configurations in that file. If you turn on DEBUG
logging,
> > you
> > > > can
> > > > > > see
> > > > > > > > what
> > > > > > > > > values are set for these environment variables to
get a
> > better
> > > > idea
> > > > > > of
> > > > > > > > how
> > > > > > > > > your METplus configuration will affect the MET
> configuration.
> > > > > > > > >
> > > > > > > > > Please let me know if you have any questions
regarding any
> of
> > > > this.
> > > > > > > > >
> > > > > > > > > On Tue, May 5, 2020 at 11:46 AM Edward Strobach -
NOAA
> > > Affiliate
> > > > > via
> > > > > > > RT <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
> > > > >
> > > > > > > > > >
> > > > > > > > > > Update:
> > > > > > > > > >
> > > > > > > > > > Trying to follow the log files that are generated
for
> > > guidance.
> > > > > I
> > > > > > > set
> > > > > > > > > the
> > > > > > > > > > verbosity to 7, but still can't understand why I'm
> > receiving
> > > > this
> > > > > > > > > message:
> > > > > > > > > >
> > > > > > > > > > 05/05 17:41:59.861 metplus INFO: Running METplus
v3.0
> > called
> > > > with
> > > > > > > > > command:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/master_metplus.py
> > > > > > > > > > -c
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/use_cases/model_applications/air_quality/CMAQ_obsAirnow.conf
> > > > > > > > > > 05/05 17:41:59.882 metplus (met_util.py:776)
ERROR:
> > > > > > > > [INIT/VALID]_TIME_FMT
> > > > > > > > > > (%Y%m%d%H) does not match [INIT/VALID]_[BEG/END]
> > (2020050200
> > > > > > > > > #2020050317)
> > > > > > > > > > 05/05 17:41:59.886 metplus.PB2NC (met_util.py:192)
ERROR:
> > > Fatal
> > > > > > error
> > > > > > > > > > occurred
> > > > > > > > > >
> > > > > > > > > > I tried to start basic. If starting basic worked,
then I
> > > would
> > > > > > have
> > > > > > > > made
> > > > > > > > > > it more complex:
> > > > > > > > > >
> > > > > > > > > > # time looping - options are INIT, VALID, RETRO,
and
> > REALTIME
> > > > > > > > > > LOG_MET_VERBOSITY = 7
> > > > > > > > > >
> > > > > > > > > > LOOP_BY = INIT
> > > > > > > > > >
> > > > > > > > > > # Format of VALID_BEG and VALID_END
> > > > > > > > > > INIT_TIME_FMT = %Y%m%d%H
> > > > > > > > > >
> > > > > > > > > > # Start time, end time, and time increment for
METplus
> run
> > > > > > > > > > INIT_BEG = 2020050200 #{now?fmt=%Y%m%d%H?shift=-
172800}
> > > > > > > > > > INIT_END = 2020050200 #{now?fmt=%Y%m%d%H?shift=-
172800}
> > > > > #-21600}
> > > > > > > > > > INIT_INCREMENT = 0
> > > > > > > > > > LEAD_SEQ = 0
> > > > > > > > > >
> > > > > > > > > > With how this looks, I would have thought that
only one
> > file
> > > > > would
> > > > > > be
> > > > > > > > > > processed. But it complains about a mismatch
between the
> > > > > > > INIT_TIME_FMT
> > > > > > > > > and
> > > > > > > > > > INIT_BEG/INIT_END
> > > > > > > > > >
> > > > > > > > > > On Tue, May 5, 2020 at 12:01 PM met_help at ucar.edu
via
> RT <
> > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Greetings,
> > > > > > > > > > >
> > > > > > > > > > > This message has been automatically generated in
> response
> > > to
> > > > > the
> > > > > > > > > creation
> > > > > > > > > > > of a trouble ticket regarding:
> > > > > > > > > > > "PB2NC_OFFSETS/related questions",
> > > > > > > > > > > a summary of which appears below.
> > > > > > > > > > >
> > > > > > > > > > > There is no need to reply to this message right
now.
> > Your
> > > > > ticket
> > > > > > > has
> > > > > > > > > > been
> > > > > > > > > > > assigned an ID of [rt.rap.ucar.edu #95164].
> > > > > > > > > > >
> > > > > > > > > > > Please include the string:
> > > > > > > > > > >
> > > > > > > > > > > [rt.rap.ucar.edu #95164]
> > > > > > > > > > >
> > > > > > > > > > > in the subject line of all future correspondence
about
> > this
> > > > > > issue.
> > > > > > > To
> > > > > > > > > do
> > > > > > > > > > > so, you may reply to this message.
> > > > > > > > > > >
> > > > > > > > > > > For more information, please see:
> > > > > > > > > > >
> > > > > > > > > > > MET Online Tutorial:
> > > > > > > > > > >
> > > > > > > > >
> > > > >
> https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > > > > > > > > > >
> > > > > > > > > > > MET Users Guide:
> > > > > > > > > > >
> https://www.dtcenter.org/met/users/docs/overview.php
> > > > > > > > > > >
> > > > > > > > > > > MET FAQs:
> > > > > > > > > > >
> > > https://www.dtcenter.org/met/users/support/faqs/index.php
> > > > > > > > > > >
> > > > > > > > > > > MET-Help Email Archive:
> > > > > > > > > > > http://mailman.ucar.edu/pipermail/met_help
> > > > > > > > > > >
> > > > > > > > > > > Thank you,
> > > > > > > > > > > met_help at ucar.edu
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
>
-------------------------------------------------------------------------
> > > > > > > > > > > Good morning,
> > > > > > > > > > >
> > > > > > > > > > > I'm trying to configure a file to process
prepbufr data
> > to
> > > > > netcdf
> > > > > > > > > format.
> > > > > > > > > > > There are several files that are within the
directory
> > that
> > > I
> > > > > want
> > > > > > > to
> > > > > > > > > > > process as shown by the bolded text from the
> > > > > > > > > > >
/gpfs/dell1/nco/ops/com/hourly/prod/hourly.20200503
> > > > directory:
> > > > > > > > > > >
> > > > > > > > > > > aqm.t12z.airnow.tm00.bufr_d
aqm.t12z.anowpm.pb.tm120
> > > > > > > > > > > aqm.t12z.status.tm024.bufr_d
> > > > > > > > > > > aqm.t12z.airnow.tm024.bufr_d
*aqm.t12z.prepbufr.tm00 *
> > > > > > > > > > > aqm.t12z.status.tm120.bufr_d
> > > > > > > > > > > aqm.t12z.airnow.tm120.bufr_d
*aqm.t12z.prepbufr.tm024*
> > > > > > > > > > > obsproc_version_for_t12z_aqm_dump_run
> > > > > > > > > > > *aqm.t12z.anowpm.pb.tm00*
aqm.t12z.prepbufr.tm120
> > > > > > > > > > > where_t12z_aqm_dump_ran
> > > > > > > > > > > *aqm.t12z.anowpm.pb.tm024 *
> > aqm.t12z.status.tm00.bufr_d
> > > > > > > > > > >
> > > > > > > > > > > Right now I'm focusing only on the *anowpm*
files (two
> > > > files).
> > > > > > The
> > > > > > > > > tm00
> > > > > > > > > > is
> > > > > > > > > > > time centered on 12z for 05/03/2020, with DHR =
ranging
> > > from
> > > > > -11
> > > > > > to
> > > > > > > > 12,
> > > > > > > > > > > covering the whole day. Similarly, the tm024
file is
> > > > centered
> > > > > at
> > > > > > > > 12z,
> > > > > > > > > > but
> > > > > > > > > > > for 05/02/2020. I looked through the metplus
glossary
> > and
> > > > > > examples
> > > > > > > > to
> > > > > > > > > > help
> > > > > > > > > > > with the process of building the configure file.
If I
> > run
> > > > this
> > > > > > > > script
> > > > > > > > > > and
> > > > > > > > > > > desire the files from 05/03/2020, then I should
set the
> > > > > > following:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > *[config]# time looping - options are INIT,
VALID,
> RETRO,
> > > and
> > > > > > > > > > > REALTIMELOOP_BY = VALID# Format of VALID_BEG and
> > > > > > > > > VALID_ENDVALID_TIME_FMT
> > > > > > > > > > =
> > > > > > > > > > > %Y%m%d%H# Start time, end time, and time
increment for
> > > > METplus
> > > > > > > > > > runVALID_BEG
> > > > > > > > > > > = {now?fmt=%Y%m%d%H?shift=-2d}VALID_END =
> > > > > > > > {now?fmt=%Y%m%d%H?shift=-2d}
> > > > > > > > > > > #-21600}VALID_INCREMENT = 24HLEAD_SEQ = 0,24
(((NOT
> > > SURE I
> > > > > > NEED
> > > > > > > > > THIS,
> > > > > > > > > > > BUT NOT SURE HOW THE LOOP WILL WORK WITHOUT
IT?)))*
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > *LOOP_ORDER = times# List of applications to
> > > runPROCESS_LIST
> > > > =
> > > > > > > > > > > PB2NCCONFIG_DIR =
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/met_configCONFIG_FILE
> > > > > > > > > > > =
> > {CONFIG_DIR}/PB2NCConfig_wrappedPB2NC_OBS_BUFR_VAR_LIST =
> > > > > DHR,
> > > > > > > RPT,
> > > > > > > > > > SID,
> > > > > > > > > > > YOB, XOB, TYPO, TPHR, QCIND, COPO, COPOPM*
> > > > > > > > > > >
> > > > > > > > > > > I've listed the variables I would like to
extract from
> > each
> > > > > file
> > > > > > as
> > > > > > > > > well.
> > > > > > > > > > > It's not clear if I need the config file, but I
listed
> it
> > > > just
> > > > > in
> > > > > > > > case.
> > > > > > > > > > > Here's what I have for the file names from the
input
> > > > directory
> > > > > > and
> > > > > > > > > what I
> > > > > > > > > > > would name it once converted to netcdf and
placed in
> the
> > > > output
> > > > > > > > > > directory:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
*INPUT_BASE=/gpfs/dell1/nco/ops/com/hourly/prod/hourly.{now?fmt=%Y%m%d?shift=-
2d}
> > > > > > > > > > > # -86400}PB2NC_INPUT_DIR =
{INPUT_BASE}OUTPUT_BASE =
> > > > > > > > > > >
> > > > >
/gpfs/dell2/emc/modeling/noscrub/Edward.StrobachPB2NC_OUTPUT_DIR
> > > > > > =
> > > > > > > > > > >
> > > {OUTPUT_BASE}/AirNow/pb2nc/hourly.{now?fmt=%Y%m%d?shift=-2d}
> > > > #
> > > > > > > > > > > -86400}[filename_templates]PB2NC_INPUT_TEMPLATE
=
> > > > > > > > > > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ} (((RELATED TO
LOOP
> > > > > > > > > > > CONFUSION)))PB2NC_OUTPUT_TEMPLATE =
> > > > > > > > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ}.nc*
> > > > > > > > > > >
> > > > > > > > > > > Let me know if more information is needed. My
main
> > > concerns
> > > > > here
> > > > > > > are
> > > > > > > > > 1)
> > > > > > > > > > > being able to process all the list files
correctly, 2)
> > > making
> > > > > > sure
> > > > > > > > the
> > > > > > > > > > > looping is correct, and 3) making sure that I've
> included
> > > > > > > everything
> > > > > > > > > > that I
> > > > > > > > > > > would need to process this step
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Edward Strobach
> > > > > > > > > > > EMC/NCEP/NWS/
> > > > > > > > > > > IMSG Contractor
> > > > > > > > > > > Cubicle#: 2029
> > > > > > > > > > > 301-683-3717
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Edward Strobach
> > > > > > > > > > EMC/NCEP/NWS/
> > > > > > > > > > IMSG Contractor
> > > > > > > > > > Cubicle#: 2029
> > > > > > > > > > 301-683-3717
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > Research Applications Laboratory
> > > > > > > > > 303-497-2768
> > > > > > > > > ---
> > > > > > > > > My working day may not be your working day. Please
do not
> > feel
> > > > > > obliged
> > > > > > > to
> > > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Edward Strobach
> > > > > > > > EMC/NCEP/NWS/
> > > > > > > > IMSG Contractor
> > > > > > > > Cubicle#: 2029
> > > > > > > > 301-683-3717
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > George McCabe - Software Engineer III
> > > > > > > National Center for Atmospheric Research
> > > > > > > Research Applications Laboratory
> > > > > > > 303-497-2768
> > > > > > > ---
> > > > > > > My working day may not be your working day. Please do
not feel
> > > > obliged
> > > > > to
> > > > > > > reply to this email outside of your normal working
hours.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Edward Strobach
> > > > > > EMC/NCEP/NWS/
> > > > > > IMSG Contractor
> > > > > > Cubicle#: 2029
> > > > > > 301-683-3717
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > > --
> > > > Edward Strobach
> > > > EMC/NCEP/NWS/
> > > > IMSG Contractor
> > > > Cubicle#: 2029
> > > > 301-683-3717
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: PB2NC_OFFSETS/related questions
From: George McCabe
Time: Thu May 07 17:21:22 2020
Hi Edward,
John and I have had a full day today and I am taking the day off on
Friday.
Let's schedule a time to meet next week. Do any days/time work better
for
you?
On Thu, May 7, 2020 at 10:54 AM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
>
> Removing the JLOGFILE from .bashrc moved me along nicely. I think
> everything processed just fine.
>
> In terms of setting up a meeting, that would be great. Sometimes a
nudge
> in the right direction is all that is needed. My goals, both short-
term and
> long-term, would be run this script, generate plots (I've seen
> plot_data_plane and plot_point_obs) both stats and maps, and set
this up in
> a batch system so to process it with a cron job so that I have a
time
> series of stats for different regions, and maps with site overlays.
> Eventually I can expand this to other fields if needed. Right now
I'm not
> exactly sure how to incorporate plotting utilities in this process,
but I
> assume it's just a matter of setting up another config file with
path
> information, processes to be used, and information related to time.
I have
> a meeting in a few minutes but am free otherwise. We could find a
time
> that works for all of us, perhaps?
>
> On Thu, May 7, 2020 at 11:56 AM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Edward,
> >
> > Thanks for sending all of this information.
> >
> > I think the fatal error lies in this line in your bashrc:
> >
> > export JLOGFILE=~/
> >
> > I would recommend not setting JLOGFILE at all, as it isn't
necessary to
> run
> > METplus. I believe it should be set to a filepath to write job
logs, so
> > setting this to your home directory is causing the fatal error. I
am
> > guessing it only writes to the JLOGFILE when there is an error,
and it
> > can't write to ~/ because it is a directory not a file. Removing
that
> from
> > your bashrc should fix the fatal error you are experiencing.
> >
> > It might be beneficial to set up a meeting with you, me, and John
Halley
> > Gotway (MET engineer) to share screens and step through your
current
> setup
> > to help you figure out how to set up what you want to do. We have
done
> this
> > with Perry in the past and it proved to be very productive. Let me
know
> if
> > you would like to do that and we can set up a time to meet.
> >
> > Thanks,
> > George
> >
> > On Thu, May 7, 2020 at 7:32 AM Edward Strobach - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> > >
> > > 1) I didn't see a metplus_final.conf file in OUTPUT_BASE, but
based on
> > your
> > > last email, I am guessing you changed the value of METPLUS_CONF
to
> create
> > > those files you listed. Which of the pb2nc_pointstat conf files
was
> used
> > > for the runs that are producing the error? Could you send me
that one?
> > When
> > > you run a use case, the name of the final conf is output to the
screen
> > > towards the beginning of execution, but unfortunately not added
to the
> > > logfile currently.
> > >
> > > They both produce the error. One of them looks for an ozone file
which
> > > doesn't exist (aqm.t00z.awpozcon.f00.148.grib2) and the other
looks
> for a
> > > pm file which doesn't exist (aqm.t00z.pm25.f00.148.grib2).
*f00* are
> not
> > > saved as output from the CMAQ runs while the others (*fXY* ) are
being
> > > saved. I have *.conf files that are created for each process
> > > (shared.conf_aq; aq.conf; pm.conf; and statanalysis.conf).
These are
> > based
> > > on inputs from my runs script. These are located here:
> > > /gpfs/dell2/ptmp/Edward.Strobach/metplus_aq
> > >
> > > When running the master_metplus.py command, I use several *.conf
> > > files: *${MET_PLUS}/ush/master_metplus.py
> > > -c ${MET_PLUS}/parm/use_cases/grid_to_obs/grid_to_obs.conf -c
> > >
${MET_PLUS}/parm/use_cases/grid_to_obs/examples/conus_surface.conf -c
> > > ${MET_PLUS_CONF}/pb2nc_aq.conf -c
${MET_PLUS_CONF}/point_stat_aq.conf
> -c
> > > ${MET_PLUS_TMP}/${model}.conf -c ${MET_PLUS_CONF}/shared.conf -c
> > > ${MET_PLUS_TMP}/shared.conf_aq -c
${MET_PLUS_CONF}/system_aq.conf*
> > >
> > > Here, model stands for "aq" or "pm"
> > >
> > > Lastly, all the conf files that are stored in the
OUTPUT_BASE/conf can
> be
> > > found in the aq directory (two of them) and pm directory (2 of
them).
> > > There is a log file located here:
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/logs/aq/20200504
> > > and
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/logs/pm/20200504.
> > > Both of them have this error message in them: 05/06
17:36:43.877
> > > metplus.StatAnalysis (stat_analysis_wrapper.py:237) DEBUG: Items
in
> these
> > > lists will be looped over: FCST_VALID_HOUR_LIST
> > >
> > > This, to me, is clearly related to the issue of trying to find
that
> file
> > > that doesn't exist. I'm not 100%, but I think the conf file that
you
> want
> > > is this: metplus_final_statanalysis_gatherbydate.conf. I copied
over
> the
> > > contents and stored them in the attachment below. I'm still
trying to
> > > understand what exactly is being processed based on the contents
> inside.
> > >
> > >
> > > 2) It looks like your master_metplus.py command has a lot of
extra conf
> > > files that don't need to be in there. The grid_to_obs conf files
are
> from
> > > an old version of METplus, so I would think they would cause
errors
> using
> > > deprecated variables. The corresponding use case in METplus 3.0
for the
> > > upper_air.conf use case is:
> > >
> > > METplus/parm/use_cases/model_applications/medium_range/Point
> > > Stat_fcstGFS_obsGDAS_UpperAir_MultiField_PrepBufr.conf
> > >
> > > I don't really know, but I would assume that there would be
specific
> > errors
> > > related to deprecated variables that would suggest as much. I
didn't
> see
> > > that though. I could attach the valid config helper, though,
just in
> > case,
> > > right?
> > >
> > > I would start with running that conf with a user config and if
that
> > works,
> > > copy the use case conf to another file I and start modifying
that
> instead
> > > of
> > > adding new confs onto the command. That will keep things a
little more
> > > organized for you.
> > >
> > > I can look into trying that. I'm not very comfortable yet with
metplus
> to
> > > make serious changes, so it might take a little bit of time to
change
> the
> > > set-up and make a test.
> > >
> > > 3) The error message is very odd. It looks like it was logging
just
> fine
> > up
> > > until that error, then it was trying to write to a log "file"
named
> > > /u/Edward.Strobach and crashing because that is a directory. I'm
not
> > > understanding why it would change the log file in the middle of
> execution
> > > and on top of that still write the offending error to the log. I
am
> very
> > > baffled by that. I will try to recreate this error locally.
> > >
> > > Could it be how I have things set in my .bashrc? I have these
related
> > > settings:
> > > module load python/3.6.3
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *# Add METplus to $PATHmodule use
> > >
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/modulefiles/module
> > > load met/9.0module load NCO/4.7.0module load
grib_util/1.1.1export
> > >
> > >
> >
>
PATH=${PATH}:/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/METplus/METplus-
3.0/ushexport
> > > PATH=~/METplus/ush:$PATH# Optional JLOGFILEexport
JLOGFILE=~/module
> load
> > > prod_util/1.1.3*
> > >
> > >
> > >
> > >
> > > On Wed, May 6, 2020 at 6:46 PM George McCabe via RT
<met_help at ucar.edu
> >
> > > wrote:
> > >
> > > > Hi Edward,
> > > >
> > > > Julie was able to login and send me some of your log files and
info
> > > while I
> > > > was in a meeting.
> > > >
> > > > 1) I didn't see a metplus_final.conf file in OUTPUT_BASE, but
based
> on
> > > your
> > > > last email, I am guessing you changed the value of
METPLUS_CONF to
> > create
> > > > those files you listed. Which of the pb2nc_pointstat conf
files was
> > used
> > > > for the runs that are producing the error? Could you send me
that
> one?
> > > When
> > > > you run a use case, the name of the final conf is output to
the
> screen
> > > > towards the beginning of execution, but unfortunately not
added to
> the
> > > > logfile currently.
> > > >
> > > >
> > > > 2) It looks like your master_metplus.py command has a lot of
extra
> conf
> > > > files that don't need to be in there. The grid_to_obs conf
files are
> > from
> > > > an old version of METplus, so I would think they would cause
errors
> > using
> > > > deprecated variables. The corresponding use case in METplus
3.0 for
> the
> > > > upper_air.conf use case is:
> > > >
> > > >
> > > >
> > >
> >
>
METplus/parm/use_cases/model_applications/medium_range/PointStat_fcstGFS_obsGDAS_UpperAir_MultiField_PrepBufr.conf
> > > >
> > > > I would start with running that conf with a user config and if
that
> > > works,
> > > > copy the use case conf to another file and start modifying
that
> instead
> > > of
> > > > adding new confs onto the command. That will keep things a
little
> more
> > > > organized for you.
> > > >
> > > >
> > > > 3) The error message is very odd. It looks like it was logging
just
> > fine
> > > up
> > > > until that error, then it was trying to write to a log "file"
named
> > > > /u/Edward.Strobach and crashing because that is a directory.
I'm not
> > > > understanding why it would change the log file in the middle
of
> > execution
> > > > and on top of that still write the offending error to the log.
I am
> > very
> > > > baffled by that. I will try to recreate this error locally.
> > > >
> > > > On Wed, May 6, 2020 at 1:43 PM Edward Strobach - NOAA
Affiliate via
> RT
> > <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> > > > >
> > > > > So my OUTPUT_BASE is here:
> > > > > /gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq.
> > > > >
> > > > > There is a conf directory where .conf files are stored in aq
> > > > > (OUTPUT_BASE/conf/aq) and pm (OUTPUT_BASE/conf/pm). There
are some
> > > .conf
> > > > > files from these runs.
> > > > >
> > > > > aq:
> > > > > metplus_final_pb2nc_pointstat.conf
> > > > > metplus_final_statanalysis_gatherbydate.conf
> > > > >
> > > > > pm:
> > > > > metplus_final_pb2nc_pointstat.conf
> > > > > metplus_final_statanalysis_gatherbydate.conf
> > > > >
> > > > > Any file in particular that you want to see? I don't have
the file
> > you
> > > > are
> > > > > referring to.
> > > > >
> > > > > On Wed, May 6, 2020 at 3:22 PM George McCabe via RT <
> > met_help at ucar.edu
> > > >
> > > > > wrote:
> > > > >
> > > > > > That's it, thanks.
> > > > > >
> > > > > > Hmm, it looks like there is an error trying to add to the
log
> file?
> > > Did
> > > > > > this run generate a log file? It would be found in the
logs
> > directory
> > > > > under
> > > > > > the value you set for OUTPUT_BASE. If there is one, can
you send
> > that
> > > > to
> > > > > > me? Also, please send the metplus_final.conf file in your
> > OUTPUT_BASE
> > > > as
> > > > > > well.
> > > > > >
> > > > > >
> > > > > > On Wed, May 6, 2020 at 1:03 PM Edward Strobach - NOAA
Affiliate
> via
> > > RT
> > > > <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
> >
> > > > > > >
> > > > > > > Sure thing. I think you are referring to this?
> > > > > > >
> > > > > > > 05/06 17:34:09.831 metplus.PointStat
(command_builder.py:100)
> > > ERROR:
> > > > > > > (command_builder.py:437) Could not find FCST file
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
> > > > > > > using template
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
> > > > > > > 05/06 17:34:09.832 metplus.PointStat (met_util.py:192)
ERROR:
> > Fatal
> > > > > error
> > > > > > > occurred
> > > > > > > *Traceback (most recent call last):*
> > > > > > > File
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > > > > line 169, in run_metplus
> > > > > > > process.run_all_times()
> > > > > > > File
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > > line 751, in run_all_times
> > > > > > > util.loop_over_times_and_call(self.config, self)
> > > > > > > File
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > > > > line 834, in loop_over_times_and_call
> > > > > > > process.run_at_time(input_dict)
> > > > > > > File
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/compare_gridded_wrapper.py",
> > > > > > > line 183, in run_at_time
> > > > > > > self.run_at_time_once(time_info)
> > > > > > > File
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/point_stat_wrapper.py",
> > > > > > > line 146, in run_at_time_once
> > > > > > > model_path = self.find_model(time_info, var_list[0])
> > > > > > > File
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > > line 319, in find_model
> > > > > > > return_list=return_list)
> > > > > > > File
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > > line 373, in find_data
> > > > > > > return self.find_exact_file(**arg_dict)
> > > > > > > File
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > > line 437, in find_exact_file
> > > > > > > self.log_error(msg)
> > > > > > > File
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > > line 100, in log_error
> > > > > > >
> > > > > > >
> > > > >
> > >
>
self.logger.error(f"({os.path.basename(caller.filename)}:{caller.lineno})
> > > > > > > {error_string}")
> > > > > > > File
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > > line 1335, in error
> > > > > > > self._log(ERROR, msg, args, **kwargs)
> > > > > > > File
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > > line 1442, in _log
> > > > > > > self.handle(record)
> > > > > > > File
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > > line 1452, in handle
> > > > > > > self.callHandlers(record)
> > > > > > > File
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > > line 1514, in callHandlers
> > > > > > > hdlr.handle(record)
> > > > > > > File
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > > line 863, in handle
> > > > > > > self.emit(record)
> > > > > > > File
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/produtil/log.py",
> > > > > > > line 227, in emit
> > > > > > > with open(self._logger,'at') as f:
> > > > > > >
> > > > > > > On Wed, May 6, 2020 at 2:55 PM George McCabe via RT <
> > > > met_help at ucar.edu
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Edward,
> > > > > > > >
> > > > > > > > When you cloned the repo yourself, are you using the
develop
> > > branch
> > > > > to
> > > > > > > run?
> > > > > > > > There may have been a bug introduced that is causing
that
> fatal
> > > > > error.
> > > > > > > > Could you send me the stack trace that comes after the
fatal
> > > > error? I
> > > > > > > agree
> > > > > > > > that it shouldn't crash if it can't find a file.
> > > > > > > >
> > > > > > > > On Wed, May 6, 2020 at 12:43 PM Edward Strobach - NOAA
> > Affiliate
> > > > via
> > > > > > RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
> > > >
> > > > > > > > >
> > > > > > > > > Thanks George. I've been busy trying to figure
things out
> > and
> > > > > forgot
> > > > > > > to
> > > > > > > > > respond. I talked with Perry about some of these
things
> and
> > he
> > > > was
> > > > > > > > > surprised that I was receiving this message when
running
> his
> > > > > script:
> > > > > > > > >
> > > > > > > > > 05/06 17:34:09.517 metplus.PointStat INFO:
Processing
> > forecast
> > > > > lead 0
> > > > > > > > hours
> > > > > > > > > 05/06 17:34:09.525 metplus.PointStat
> (command_builder.py:407)
> > > > > DEBUG:
> > > > > > > > > Looking for FCST file
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
> > > > > > > > > 05/06 17:34:09.831 metplus.PointStat
> (command_builder.py:100)
> > > > > ERROR:
> > > > > > > > > (command_builder.py:437) Could not find FCST file
> > > > > > > > > /gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/
> > > > > > > > > *aqm.t00z.awpozcon.f00.148.grib2* using template
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
> > > > > > > > > 05/06 17:34:09.832 metplus.PointStat
(met_util.py:192)
> ERROR:
> > > > Fatal
> > > > > > > error
> > > > > > > > > occurred
> > > > > > > > >
> > > > > > > > > The bolded item does not exist. When Perry runs his
script
> > it
> > > > > skips
> > > > > > > this
> > > > > > > > > file. When I run the script I get a fatal error. I
cloned
> > > > metplus
> > > > > > > just
> > > > > > > > > the other day. I'm not sure if this is related to
item 2
> > from
> > > > your
> > > > > > > email
> > > > > > > > >
> > > > > > > > > On Tue, May 5, 2020 at 2:37 PM George McCabe via RT
<
> > > > > > met_help at ucar.edu
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Edward,
> > > > > > > > > >
> > > > > > > > > > 1) The LOG_MET_VERBOSITY variable sets the
verbosity
> level
> > > > when a
> > > > > > MET
> > > > > > > > > tool
> > > > > > > > > > is run. To change the METplus logging level, set
> > LOG_LEVEL. I
> > > > > would
> > > > > > > > > > recommend changing it to DEBUG to get more
information.
> > > > > > > > > >
> > > > > > > > > > 2) There is a bug in using the {now} tags for
> > > > > > [INIT/VALID]_[BEG/END]
> > > > > > > > that
> > > > > > > > > > causes that crash. This has been fixed but we have
not
> yet
> > > > > > released a
> > > > > > > > > > bugfix release. I will let the engineers know that
you
> need
> > > > this
> > > > > > > > > > functionality and we will push the METplus 3.0.1
bugfix
> > > release
> > > > > > soon.
> > > > > > > > You
> > > > > > > > > > should be able to set the time explicitly, i.e.
> 2020050500.
> > > > > > > > > >
> > > > > > > > > > 3) The {now} tag can be used in the VALID_BEG and
> VALID_END
> > > > > config
> > > > > > > > > > variables, but you will want to refer to init,
valid,
> lead,
> > > > etc.
> > > > > in
> > > > > > > > your
> > > > > > > > > > INPUT_TEMPLATE and OUTPUT_TEMPLATE variables so
that it
> > uses
> > > > the
> > > > > > time
> > > > > > > > for
> > > > > > > > > > each time loop. See this section for more
information:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#directory-
and-filename-template-info
> > > > > > > > > >
> > > > > > > > > > 4) The LEAD_SEQ is a list of forecast leads that
will be
> > > > > processed
> > > > > > > for
> > > > > > > > > each
> > > > > > > > > > run time. If you set this to 0 or unset it, then
it will
> > just
> > > > > > process
> > > > > > > > > once
> > > > > > > > > > for each run time.
> > > > > > > > > >
> > > > > > > > > > 5) There is documentation describing the time
looping
> > > variables
> > > > > > here:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#common-
config-variables
> > > > > > > > > >
> > > > > > > > > > This page should answer some of your questions on
how
> this
> > > > works.
> > > > > > > > > >
> > > > > > > > > > 6) I would recommend starting with the Example use
case
> > found
> > > > in
> > > > > > > > > >
parm/use_cases/met_tool_wrapper/Example/Example.conf.
> This
> > > use
> > > > > case
> > > > > > > > loops
> > > > > > > > > > over times, fills in the filename templates based
on each
> > run
> > > > > time,
> > > > > > > and
> > > > > > > > > > prints information to the logfile. You can copy
this use
> > case
> > > > and
> > > > > > > > modify
> > > > > > > > > it
> > > > > > > > > > to see which filenames are searched for each run
time.
> This
> > > is
> > > > a
> > > > > > good
> > > > > > > > way
> > > > > > > > > > to understand how to configure METplus to find the
files
> > you
> > > > > need.
> > > > > > > Once
> > > > > > > > > > this makes sense, you can look at the PB2NC.conf
use case
> > to
> > > > see
> > > > > > how
> > > > > > > > the
> > > > > > > > > > config variables control that wrapper.
> > > > > > > > > >
> > > > > > > > > > 7) METplus sets environment variables based on its
> > > > configuration
> > > > > > > files.
> > > > > > > > > > These environment variables are referenced in the
MET
> > config
> > > > file
> > > > > > to
> > > > > > > > set
> > > > > > > > > > configurations in that file. If you turn on DEBUG
> logging,
> > > you
> > > > > can
> > > > > > > see
> > > > > > > > > what
> > > > > > > > > > values are set for these environment variables to
get a
> > > better
> > > > > idea
> > > > > > > of
> > > > > > > > > how
> > > > > > > > > > your METplus configuration will affect the MET
> > configuration.
> > > > > > > > > >
> > > > > > > > > > Please let me know if you have any questions
regarding
> any
> > of
> > > > > this.
> > > > > > > > > >
> > > > > > > > > > On Tue, May 5, 2020 at 11:46 AM Edward Strobach -
NOAA
> > > > Affiliate
> > > > > > via
> > > > > > > > RT <
> > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
> > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Update:
> > > > > > > > > > >
> > > > > > > > > > > Trying to follow the log files that are
generated for
> > > > guidance.
> > > > > > I
> > > > > > > > set
> > > > > > > > > > the
> > > > > > > > > > > verbosity to 7, but still can't understand why
I'm
> > > receiving
> > > > > this
> > > > > > > > > > message:
> > > > > > > > > > >
> > > > > > > > > > > 05/05 17:41:59.861 metplus INFO: Running METplus
v3.0
> > > called
> > > > > with
> > > > > > > > > > command:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/master_metplus.py
> > > > > > > > > > > -c
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/use_cases/model_applications/air_quality/CMAQ_obsAirnow.conf
> > > > > > > > > > > 05/05 17:41:59.882 metplus (met_util.py:776)
ERROR:
> > > > > > > > > [INIT/VALID]_TIME_FMT
> > > > > > > > > > > (%Y%m%d%H) does not match [INIT/VALID]_[BEG/END]
> > > (2020050200
> > > > > > > > > > #2020050317)
> > > > > > > > > > > 05/05 17:41:59.886 metplus.PB2NC
(met_util.py:192)
> ERROR:
> > > > Fatal
> > > > > > > error
> > > > > > > > > > > occurred
> > > > > > > > > > >
> > > > > > > > > > > I tried to start basic. If starting basic
worked,
> then I
> > > > would
> > > > > > > have
> > > > > > > > > made
> > > > > > > > > > > it more complex:
> > > > > > > > > > >
> > > > > > > > > > > # time looping - options are INIT, VALID, RETRO,
and
> > > REALTIME
> > > > > > > > > > > LOG_MET_VERBOSITY = 7
> > > > > > > > > > >
> > > > > > > > > > > LOOP_BY = INIT
> > > > > > > > > > >
> > > > > > > > > > > # Format of VALID_BEG and VALID_END
> > > > > > > > > > > INIT_TIME_FMT = %Y%m%d%H
> > > > > > > > > > >
> > > > > > > > > > > # Start time, end time, and time increment for
METplus
> > run
> > > > > > > > > > > INIT_BEG = 2020050200
> #{now?fmt=%Y%m%d%H?shift=-172800}
> > > > > > > > > > > INIT_END = 2020050200 #{now?fmt=%Y%m%d%H?shift=-
172800}
> > > > > > #-21600}
> > > > > > > > > > > INIT_INCREMENT = 0
> > > > > > > > > > > LEAD_SEQ = 0
> > > > > > > > > > >
> > > > > > > > > > > With how this looks, I would have thought that
only one
> > > file
> > > > > > would
> > > > > > > be
> > > > > > > > > > > processed. But it complains about a mismatch
between
> the
> > > > > > > > INIT_TIME_FMT
> > > > > > > > > > and
> > > > > > > > > > > INIT_BEG/INIT_END
> > > > > > > > > > >
> > > > > > > > > > > On Tue, May 5, 2020 at 12:01 PM
met_help at ucar.edu via
> > RT <
> > > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Greetings,
> > > > > > > > > > > >
> > > > > > > > > > > > This message has been automatically generated
in
> > response
> > > > to
> > > > > > the
> > > > > > > > > > creation
> > > > > > > > > > > > of a trouble ticket regarding:
> > > > > > > > > > > > "PB2NC_OFFSETS/related questions",
> > > > > > > > > > > > a summary of which appears below.
> > > > > > > > > > > >
> > > > > > > > > > > > There is no need to reply to this message
right now.
> > > Your
> > > > > > ticket
> > > > > > > > has
> > > > > > > > > > > been
> > > > > > > > > > > > assigned an ID of [rt.rap.ucar.edu #95164].
> > > > > > > > > > > >
> > > > > > > > > > > > Please include the string:
> > > > > > > > > > > >
> > > > > > > > > > > > [rt.rap.ucar.edu #95164]
> > > > > > > > > > > >
> > > > > > > > > > > > in the subject line of all future
correspondence
> about
> > > this
> > > > > > > issue.
> > > > > > > > To
> > > > > > > > > > do
> > > > > > > > > > > > so, you may reply to this message.
> > > > > > > > > > > >
> > > > > > > > > > > > For more information, please see:
> > > > > > > > > > > >
> > > > > > > > > > > > MET Online Tutorial:
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > >
> >
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > > > > > > > > > > >
> > > > > > > > > > > > MET Users Guide:
> > > > > > > > > > > >
> > https://www.dtcenter.org/met/users/docs/overview.php
> > > > > > > > > > > >
> > > > > > > > > > > > MET FAQs:
> > > > > > > > > > > >
> > > > https://www.dtcenter.org/met/users/support/faqs/index.php
> > > > > > > > > > > >
> > > > > > > > > > > > MET-Help Email Archive:
> > > > > > > > > > > > http://mailman.ucar.edu/pipermail/met_help
> > > > > > > > > > > >
> > > > > > > > > > > > Thank you,
> > > > > > > > > > > > met_help at ucar.edu
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >
-------------------------------------------------------------------------
> > > > > > > > > > > > Good morning,
> > > > > > > > > > > >
> > > > > > > > > > > > I'm trying to configure a file to process
prepbufr
> data
> > > to
> > > > > > netcdf
> > > > > > > > > > format.
> > > > > > > > > > > > There are several files that are within the
directory
> > > that
> > > > I
> > > > > > want
> > > > > > > > to
> > > > > > > > > > > > process as shown by the bolded text from the
> > > > > > > > > > > >
/gpfs/dell1/nco/ops/com/hourly/prod/hourly.20200503
> > > > > directory:
> > > > > > > > > > > >
> > > > > > > > > > > > aqm.t12z.airnow.tm00.bufr_d
> aqm.t12z.anowpm.pb.tm120
> > > > > > > > > > > > aqm.t12z.status.tm024.bufr_d
> > > > > > > > > > > > aqm.t12z.airnow.tm024.bufr_d
> *aqm.t12z.prepbufr.tm00 *
> > > > > > > > > > > > aqm.t12z.status.tm120.bufr_d
> > > > > > > > > > > > aqm.t12z.airnow.tm120.bufr_d
> *aqm.t12z.prepbufr.tm024*
> > > > > > > > > > > > obsproc_version_for_t12z_aqm_dump_run
> > > > > > > > > > > > *aqm.t12z.anowpm.pb.tm00*
> aqm.t12z.prepbufr.tm120
> > > > > > > > > > > > where_t12z_aqm_dump_ran
> > > > > > > > > > > > *aqm.t12z.anowpm.pb.tm024 *
> > > aqm.t12z.status.tm00.bufr_d
> > > > > > > > > > > >
> > > > > > > > > > > > Right now I'm focusing only on the *anowpm*
files
> (two
> > > > > files).
> > > > > > > The
> > > > > > > > > > tm00
> > > > > > > > > > > is
> > > > > > > > > > > > time centered on 12z for 05/03/2020, with DHR
=
> ranging
> > > > from
> > > > > > -11
> > > > > > > to
> > > > > > > > > 12,
> > > > > > > > > > > > covering the whole day. Similarly, the tm024
file is
> > > > > centered
> > > > > > at
> > > > > > > > > 12z,
> > > > > > > > > > > but
> > > > > > > > > > > > for 05/02/2020. I looked through the metplus
> glossary
> > > and
> > > > > > > examples
> > > > > > > > > to
> > > > > > > > > > > help
> > > > > > > > > > > > with the process of building the configure
file. If
> I
> > > run
> > > > > this
> > > > > > > > > script
> > > > > > > > > > > and
> > > > > > > > > > > > desire the files from 05/03/2020, then I
should set
> the
> > > > > > > following:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > *[config]# time looping - options are INIT,
VALID,
> > RETRO,
> > > > and
> > > > > > > > > > > > REALTIMELOOP_BY = VALID# Format of VALID_BEG
and
> > > > > > > > > > VALID_ENDVALID_TIME_FMT
> > > > > > > > > > > =
> > > > > > > > > > > > %Y%m%d%H# Start time, end time, and time
increment
> for
> > > > > METplus
> > > > > > > > > > > runVALID_BEG
> > > > > > > > > > > > = {now?fmt=%Y%m%d%H?shift=-2d}VALID_END =
> > > > > > > > > {now?fmt=%Y%m%d%H?shift=-2d}
> > > > > > > > > > > > #-21600}VALID_INCREMENT = 24HLEAD_SEQ = 0,24
> (((NOT
> > > > SURE I
> > > > > > > NEED
> > > > > > > > > > THIS,
> > > > > > > > > > > > BUT NOT SURE HOW THE LOOP WILL WORK WITHOUT
IT?)))*
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > *LOOP_ORDER = times# List of applications to
> > > > runPROCESS_LIST
> > > > > =
> > > > > > > > > > > > PB2NCCONFIG_DIR =
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/met_configCONFIG_FILE
> > > > > > > > > > > > =
> > > {CONFIG_DIR}/PB2NCConfig_wrappedPB2NC_OBS_BUFR_VAR_LIST =
> > > > > > DHR,
> > > > > > > > RPT,
> > > > > > > > > > > SID,
> > > > > > > > > > > > YOB, XOB, TYPO, TPHR, QCIND, COPO, COPOPM*
> > > > > > > > > > > >
> > > > > > > > > > > > I've listed the variables I would like to
extract
> from
> > > each
> > > > > > file
> > > > > > > as
> > > > > > > > > > well.
> > > > > > > > > > > > It's not clear if I need the config file, but
I
> listed
> > it
> > > > > just
> > > > > > in
> > > > > > > > > case.
> > > > > > > > > > > > Here's what I have for the file names from the
input
> > > > > directory
> > > > > > > and
> > > > > > > > > > what I
> > > > > > > > > > > > would name it once converted to netcdf and
placed in
> > the
> > > > > output
> > > > > > > > > > > directory:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
*INPUT_BASE=/gpfs/dell1/nco/ops/com/hourly/prod/hourly.{now?fmt=%Y%m%d?shift=-
2d}
> > > > > > > > > > > > # -86400}PB2NC_INPUT_DIR =
{INPUT_BASE}OUTPUT_BASE =
> > > > > > > > > > > >
> > > > > >
/gpfs/dell2/emc/modeling/noscrub/Edward.StrobachPB2NC_OUTPUT_DIR
> > > > > > > =
> > > > > > > > > > > >
> > > > {OUTPUT_BASE}/AirNow/pb2nc/hourly.{now?fmt=%Y%m%d?shift=-2d}
> > > > > #
> > > > > > > > > > > >
-86400}[filename_templates]PB2NC_INPUT_TEMPLATE =
> > > > > > > > > > > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ} (((RELATED
TO LOOP
> > > > > > > > > > > > CONFUSION)))PB2NC_OUTPUT_TEMPLATE =
> > > > > > > > > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ}.nc*
> > > > > > > > > > > >
> > > > > > > > > > > > Let me know if more information is needed. My
main
> > > > concerns
> > > > > > here
> > > > > > > > are
> > > > > > > > > > 1)
> > > > > > > > > > > > being able to process all the list files
correctly,
> 2)
> > > > making
> > > > > > > sure
> > > > > > > > > the
> > > > > > > > > > > > looping is correct, and 3) making sure that
I've
> > included
> > > > > > > > everything
> > > > > > > > > > > that I
> > > > > > > > > > > > would need to process this step
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Edward Strobach
> > > > > > > > > > > > EMC/NCEP/NWS/
> > > > > > > > > > > > IMSG Contractor
> > > > > > > > > > > > Cubicle#: 2029
> > > > > > > > > > > > 301-683-3717
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Edward Strobach
> > > > > > > > > > > EMC/NCEP/NWS/
> > > > > > > > > > > IMSG Contractor
> > > > > > > > > > > Cubicle#: 2029
> > > > > > > > > > > 301-683-3717
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > 303-497-2768
> > > > > > > > > > ---
> > > > > > > > > > My working day may not be your working day. Please
do not
> > > feel
> > > > > > > obliged
> > > > > > > > to
> > > > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Edward Strobach
> > > > > > > > > EMC/NCEP/NWS/
> > > > > > > > > IMSG Contractor
> > > > > > > > > Cubicle#: 2029
> > > > > > > > > 301-683-3717
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > George McCabe - Software Engineer III
> > > > > > > > National Center for Atmospheric Research
> > > > > > > > Research Applications Laboratory
> > > > > > > > 303-497-2768
> > > > > > > > ---
> > > > > > > > My working day may not be your working day. Please do
not
> feel
> > > > > obliged
> > > > > > to
> > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Edward Strobach
> > > > > > > EMC/NCEP/NWS/
> > > > > > > IMSG Contractor
> > > > > > > Cubicle#: 2029
> > > > > > > 301-683-3717
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Edward Strobach
> > > > > EMC/NCEP/NWS/
> > > > > IMSG Contractor
> > > > > Cubicle#: 2029
> > > > > 301-683-3717
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: PB2NC_OFFSETS/related questions
From: Edward Strobach - NOAA Affiliate
Time: Thu May 07 17:37:49 2020
I understand. Are you good some time Tues. or Wed? No meetings for
me
those days so any time works
On Thu, May 7, 2020 at 7:21 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Edward,
>
> John and I have had a full day today and I am taking the day off on
Friday.
> Let's schedule a time to meet next week. Do any days/time work
better for
> you?
>
> On Thu, May 7, 2020 at 10:54 AM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> >
> > Removing the JLOGFILE from .bashrc moved me along nicely. I think
> > everything processed just fine.
> >
> > In terms of setting up a meeting, that would be great. Sometimes
a nudge
> > in the right direction is all that is needed. My goals, both
short-term
> and
> > long-term, would be run this script, generate plots (I've seen
> > plot_data_plane and plot_point_obs) both stats and maps, and set
this up
> in
> > a batch system so to process it with a cron job so that I have a
time
> > series of stats for different regions, and maps with site
overlays.
> > Eventually I can expand this to other fields if needed. Right now
I'm not
> > exactly sure how to incorporate plotting utilities in this
process, but I
> > assume it's just a matter of setting up another config file with
path
> > information, processes to be used, and information related to
time. I
> have
> > a meeting in a few minutes but am free otherwise. We could find a
time
> > that works for all of us, perhaps?
> >
> > On Thu, May 7, 2020 at 11:56 AM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Edward,
> > >
> > > Thanks for sending all of this information.
> > >
> > > I think the fatal error lies in this line in your bashrc:
> > >
> > > export JLOGFILE=~/
> > >
> > > I would recommend not setting JLOGFILE at all, as it isn't
necessary to
> > run
> > > METplus. I believe it should be set to a filepath to write job
logs, so
> > > setting this to your home directory is causing the fatal error.
I am
> > > guessing it only writes to the JLOGFILE when there is an error,
and it
> > > can't write to ~/ because it is a directory not a file. Removing
that
> > from
> > > your bashrc should fix the fatal error you are experiencing.
> > >
> > > It might be beneficial to set up a meeting with you, me, and
John
> Halley
> > > Gotway (MET engineer) to share screens and step through your
current
> > setup
> > > to help you figure out how to set up what you want to do. We
have done
> > this
> > > with Perry in the past and it proved to be very productive. Let
me know
> > if
> > > you would like to do that and we can set up a time to meet.
> > >
> > > Thanks,
> > > George
> > >
> > > On Thu, May 7, 2020 at 7:32 AM Edward Strobach - NOAA Affiliate
via RT
> <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
>
> > > >
> > > > 1) I didn't see a metplus_final.conf file in OUTPUT_BASE, but
based
> on
> > > your
> > > > last email, I am guessing you changed the value of
METPLUS_CONF to
> > create
> > > > those files you listed. Which of the pb2nc_pointstat conf
files was
> > used
> > > > for the runs that are producing the error? Could you send me
that
> one?
> > > When
> > > > you run a use case, the name of the final conf is output to
the
> screen
> > > > towards the beginning of execution, but unfortunately not
added to
> the
> > > > logfile currently.
> > > >
> > > > They both produce the error. One of them looks for an ozone
file
> which
> > > > doesn't exist (aqm.t00z.awpozcon.f00.148.grib2) and the other
looks
> > for a
> > > > pm file which doesn't exist (aqm.t00z.pm25.f00.148.grib2).
*f00* are
> > not
> > > > saved as output from the CMAQ runs while the others (*fXY* )
are
> being
> > > > saved. I have *.conf files that are created for each process
> > > > (shared.conf_aq; aq.conf; pm.conf; and statanalysis.conf).
These are
> > > based
> > > > on inputs from my runs script. These are located here:
> > > > /gpfs/dell2/ptmp/Edward.Strobach/metplus_aq
> > > >
> > > > When running the master_metplus.py command, I use several
*.conf
> > > > files: *${MET_PLUS}/ush/master_metplus.py
> > > > -c ${MET_PLUS}/parm/use_cases/grid_to_obs/grid_to_obs.conf -c
> > > >
${MET_PLUS}/parm/use_cases/grid_to_obs/examples/conus_surface.conf -c
> > > > ${MET_PLUS_CONF}/pb2nc_aq.conf -c
${MET_PLUS_CONF}/point_stat_aq.conf
> > -c
> > > > ${MET_PLUS_TMP}/${model}.conf -c ${MET_PLUS_CONF}/shared.conf
-c
> > > > ${MET_PLUS_TMP}/shared.conf_aq -c
${MET_PLUS_CONF}/system_aq.conf*
> > > >
> > > > Here, model stands for "aq" or "pm"
> > > >
> > > > Lastly, all the conf files that are stored in the
OUTPUT_BASE/conf
> can
> > be
> > > > found in the aq directory (two of them) and pm directory (2 of
them).
> > > > There is a log file located here:
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/logs/aq/20200504
> > > > and
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/logs/pm/20200504.
> > > > Both of them have this error message in them: 05/06
17:36:43.877
> > > > metplus.StatAnalysis (stat_analysis_wrapper.py:237) DEBUG:
Items in
> > these
> > > > lists will be looped over: FCST_VALID_HOUR_LIST
> > > >
> > > > This, to me, is clearly related to the issue of trying to find
that
> > file
> > > > that doesn't exist. I'm not 100%, but I think the conf file
that you
> > want
> > > > is this: metplus_final_statanalysis_gatherbydate.conf. I
copied over
> > the
> > > > contents and stored them in the attachment below. I'm still
trying to
> > > > understand what exactly is being processed based on the
contents
> > inside.
> > > >
> > > >
> > > > 2) It looks like your master_metplus.py command has a lot of
extra
> conf
> > > > files that don't need to be in there. The grid_to_obs conf
files are
> > from
> > > > an old version of METplus, so I would think they would cause
errors
> > using
> > > > deprecated variables. The corresponding use case in METplus
3.0 for
> the
> > > > upper_air.conf use case is:
> > > >
> > > > METplus/parm/use_cases/model_applications/medium_range/Point
> > > > Stat_fcstGFS_obsGDAS_UpperAir_MultiField_PrepBufr.conf
> > > >
> > > > I don't really know, but I would assume that there would be
specific
> > > errors
> > > > related to deprecated variables that would suggest as much. I
didn't
> > see
> > > > that though. I could attach the valid config helper, though,
just in
> > > case,
> > > > right?
> > > >
> > > > I would start with running that conf with a user config and if
that
> > > works,
> > > > copy the use case conf to another file I and start modifying
that
> > instead
> > > > of
> > > > adding new confs onto the command. That will keep things a
little
> more
> > > > organized for you.
> > > >
> > > > I can look into trying that. I'm not very comfortable yet with
> metplus
> > to
> > > > make serious changes, so it might take a little bit of time to
change
> > the
> > > > set-up and make a test.
> > > >
> > > > 3) The error message is very odd. It looks like it was logging
just
> > fine
> > > up
> > > > until that error, then it was trying to write to a log "file"
named
> > > > /u/Edward.Strobach and crashing because that is a directory.
I'm not
> > > > understanding why it would change the log file in the middle
of
> > execution
> > > > and on top of that still write the offending error to the log.
I am
> > very
> > > > baffled by that. I will try to recreate this error locally.
> > > >
> > > > Could it be how I have things set in my .bashrc? I have these
> related
> > > > settings:
> > > > module load python/3.6.3
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *# Add METplus to $PATHmodule use
> > > >
> >
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/modulefiles/module
> > > > load met/9.0module load NCO/4.7.0module load
grib_util/1.1.1export
> > > >
> > > >
> > >
> >
>
PATH=${PATH}:/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/METplus/METplus-
3.0/ushexport
> > > > PATH=~/METplus/ush:$PATH# Optional JLOGFILEexport
JLOGFILE=~/module
> > load
> > > > prod_util/1.1.3*
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, May 6, 2020 at 6:46 PM George McCabe via RT <
> met_help at ucar.edu
> > >
> > > > wrote:
> > > >
> > > > > Hi Edward,
> > > > >
> > > > > Julie was able to login and send me some of your log files
and info
> > > > while I
> > > > > was in a meeting.
> > > > >
> > > > > 1) I didn't see a metplus_final.conf file in OUTPUT_BASE,
but based
> > on
> > > > your
> > > > > last email, I am guessing you changed the value of
METPLUS_CONF to
> > > create
> > > > > those files you listed. Which of the pb2nc_pointstat conf
files was
> > > used
> > > > > for the runs that are producing the error? Could you send me
that
> > one?
> > > > When
> > > > > you run a use case, the name of the final conf is output to
the
> > screen
> > > > > towards the beginning of execution, but unfortunately not
added to
> > the
> > > > > logfile currently.
> > > > >
> > > > >
> > > > > 2) It looks like your master_metplus.py command has a lot of
extra
> > conf
> > > > > files that don't need to be in there. The grid_to_obs conf
files
> are
> > > from
> > > > > an old version of METplus, so I would think they would cause
errors
> > > using
> > > > > deprecated variables. The corresponding use case in METplus
3.0 for
> > the
> > > > > upper_air.conf use case is:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
METplus/parm/use_cases/model_applications/medium_range/PointStat_fcstGFS_obsGDAS_UpperAir_MultiField_PrepBufr.conf
> > > > >
> > > > > I would start with running that conf with a user config and
if that
> > > > works,
> > > > > copy the use case conf to another file and start modifying
that
> > instead
> > > > of
> > > > > adding new confs onto the command. That will keep things a
little
> > more
> > > > > organized for you.
> > > > >
> > > > >
> > > > > 3) The error message is very odd. It looks like it was
logging just
> > > fine
> > > > up
> > > > > until that error, then it was trying to write to a log
"file" named
> > > > > /u/Edward.Strobach and crashing because that is a directory.
I'm
> not
> > > > > understanding why it would change the log file in the middle
of
> > > execution
> > > > > and on top of that still write the offending error to the
log. I am
> > > very
> > > > > baffled by that. I will try to recreate this error locally.
> > > > >
> > > > > On Wed, May 6, 2020 at 1:43 PM Edward Strobach - NOAA
Affiliate via
> > RT
> > > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> > > > > >
> > > > > > So my OUTPUT_BASE is here:
> > > > > >
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq.
> > > > > >
> > > > > > There is a conf directory where .conf files are stored in
aq
> > > > > > (OUTPUT_BASE/conf/aq) and pm (OUTPUT_BASE/conf/pm). There
are
> some
> > > > .conf
> > > > > > files from these runs.
> > > > > >
> > > > > > aq:
> > > > > > metplus_final_pb2nc_pointstat.conf
> > > > > > metplus_final_statanalysis_gatherbydate.conf
> > > > > >
> > > > > > pm:
> > > > > > metplus_final_pb2nc_pointstat.conf
> > > > > > metplus_final_statanalysis_gatherbydate.conf
> > > > > >
> > > > > > Any file in particular that you want to see? I don't have
the
> file
> > > you
> > > > > are
> > > > > > referring to.
> > > > > >
> > > > > > On Wed, May 6, 2020 at 3:22 PM George McCabe via RT <
> > > met_help at ucar.edu
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > That's it, thanks.
> > > > > > >
> > > > > > > Hmm, it looks like there is an error trying to add to
the log
> > file?
> > > > Did
> > > > > > > this run generate a log file? It would be found in the
logs
> > > directory
> > > > > > under
> > > > > > > the value you set for OUTPUT_BASE. If there is one, can
you
> send
> > > that
> > > > > to
> > > > > > > me? Also, please send the metplus_final.conf file in
your
> > > OUTPUT_BASE
> > > > > as
> > > > > > > well.
> > > > > > >
> > > > > > >
> > > > > > > On Wed, May 6, 2020 at 1:03 PM Edward Strobach - NOAA
Affiliate
> > via
> > > > RT
> > > > > <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
> > >
> > > > > > > >
> > > > > > > > Sure thing. I think you are referring to this?
> > > > > > > >
> > > > > > > > 05/06 17:34:09.831 metplus.PointStat
(command_builder.py:100)
> > > > ERROR:
> > > > > > > > (command_builder.py:437) Could not find FCST file
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
> > > > > > > > using template
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
> > > > > > > > 05/06 17:34:09.832 metplus.PointStat (met_util.py:192)
ERROR:
> > > Fatal
> > > > > > error
> > > > > > > > occurred
> > > > > > > > *Traceback (most recent call last):*
> > > > > > > > File
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > > > > > line 169, in run_metplus
> > > > > > > > process.run_all_times()
> > > > > > > > File
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > > > line 751, in run_all_times
> > > > > > > > util.loop_over_times_and_call(self.config, self)
> > > > > > > > File
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > > > > > line 834, in loop_over_times_and_call
> > > > > > > > process.run_at_time(input_dict)
> > > > > > > > File
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/compare_gridded_wrapper.py",
> > > > > > > > line 183, in run_at_time
> > > > > > > > self.run_at_time_once(time_info)
> > > > > > > > File
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/point_stat_wrapper.py",
> > > > > > > > line 146, in run_at_time_once
> > > > > > > > model_path = self.find_model(time_info,
var_list[0])
> > > > > > > > File
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > > > line 319, in find_model
> > > > > > > > return_list=return_list)
> > > > > > > > File
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > > > line 373, in find_data
> > > > > > > > return self.find_exact_file(**arg_dict)
> > > > > > > > File
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > > > line 437, in find_exact_file
> > > > > > > > self.log_error(msg)
> > > > > > > > File
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > > > line 100, in log_error
> > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >
self.logger.error(f"({os.path.basename(caller.filename)}:{caller.lineno})
> > > > > > > > {error_string}")
> > > > > > > > File
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > > > line 1335, in error
> > > > > > > > self._log(ERROR, msg, args, **kwargs)
> > > > > > > > File
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > > > line 1442, in _log
> > > > > > > > self.handle(record)
> > > > > > > > File
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > > > line 1452, in handle
> > > > > > > > self.callHandlers(record)
> > > > > > > > File
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > > > line 1514, in callHandlers
> > > > > > > > hdlr.handle(record)
> > > > > > > > File
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > > > line 863, in handle
> > > > > > > > self.emit(record)
> > > > > > > > File
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/produtil/log.py",
> > > > > > > > line 227, in emit
> > > > > > > > with open(self._logger,'at') as f:
> > > > > > > >
> > > > > > > > On Wed, May 6, 2020 at 2:55 PM George McCabe via RT <
> > > > > met_help at ucar.edu
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Edward,
> > > > > > > > >
> > > > > > > > > When you cloned the repo yourself, are you using the
> develop
> > > > branch
> > > > > > to
> > > > > > > > run?
> > > > > > > > > There may have been a bug introduced that is causing
that
> > fatal
> > > > > > error.
> > > > > > > > > Could you send me the stack trace that comes after
the
> fatal
> > > > > error? I
> > > > > > > > agree
> > > > > > > > > that it shouldn't crash if it can't find a file.
> > > > > > > > >
> > > > > > > > > On Wed, May 6, 2020 at 12:43 PM Edward Strobach -
NOAA
> > > Affiliate
> > > > > via
> > > > > > > RT <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
> > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks George. I've been busy trying to figure
things
> out
> > > and
> > > > > > forgot
> > > > > > > > to
> > > > > > > > > > respond. I talked with Perry about some of these
things
> > and
> > > he
> > > > > was
> > > > > > > > > > surprised that I was receiving this message when
running
> > his
> > > > > > script:
> > > > > > > > > >
> > > > > > > > > > 05/06 17:34:09.517 metplus.PointStat INFO:
Processing
> > > forecast
> > > > > > lead 0
> > > > > > > > > hours
> > > > > > > > > > 05/06 17:34:09.525 metplus.PointStat
> > (command_builder.py:407)
> > > > > > DEBUG:
> > > > > > > > > > Looking for FCST file
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
> > > > > > > > > > 05/06 17:34:09.831 metplus.PointStat
> > (command_builder.py:100)
> > > > > > ERROR:
> > > > > > > > > > (command_builder.py:437) Could not find FCST file
> > > > > > > > > > /gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/
> > > > > > > > > > *aqm.t00z.awpozcon.f00.148.grib2* using template
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
> > > > > > > > > > 05/06 17:34:09.832 metplus.PointStat
(met_util.py:192)
> > ERROR:
> > > > > Fatal
> > > > > > > > error
> > > > > > > > > > occurred
> > > > > > > > > >
> > > > > > > > > > The bolded item does not exist. When Perry runs
his
> script
> > > it
> > > > > > skips
> > > > > > > > this
> > > > > > > > > > file. When I run the script I get a fatal error.
I
> cloned
> > > > > metplus
> > > > > > > > just
> > > > > > > > > > the other day. I'm not sure if this is related to
item 2
> > > from
> > > > > your
> > > > > > > > email
> > > > > > > > > >
> > > > > > > > > > On Tue, May 5, 2020 at 2:37 PM George McCabe via
RT <
> > > > > > > met_help at ucar.edu
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Edward,
> > > > > > > > > > >
> > > > > > > > > > > 1) The LOG_MET_VERBOSITY variable sets the
verbosity
> > level
> > > > > when a
> > > > > > > MET
> > > > > > > > > > tool
> > > > > > > > > > > is run. To change the METplus logging level, set
> > > LOG_LEVEL. I
> > > > > > would
> > > > > > > > > > > recommend changing it to DEBUG to get more
information.
> > > > > > > > > > >
> > > > > > > > > > > 2) There is a bug in using the {now} tags for
> > > > > > > [INIT/VALID]_[BEG/END]
> > > > > > > > > that
> > > > > > > > > > > causes that crash. This has been fixed but we
have not
> > yet
> > > > > > > released a
> > > > > > > > > > > bugfix release. I will let the engineers know
that you
> > need
> > > > > this
> > > > > > > > > > > functionality and we will push the METplus 3.0.1
bugfix
> > > > release
> > > > > > > soon.
> > > > > > > > > You
> > > > > > > > > > > should be able to set the time explicitly, i.e.
> > 2020050500.
> > > > > > > > > > >
> > > > > > > > > > > 3) The {now} tag can be used in the VALID_BEG
and
> > VALID_END
> > > > > > config
> > > > > > > > > > > variables, but you will want to refer to init,
valid,
> > lead,
> > > > > etc.
> > > > > > in
> > > > > > > > > your
> > > > > > > > > > > INPUT_TEMPLATE and OUTPUT_TEMPLATE variables so
that it
> > > uses
> > > > > the
> > > > > > > time
> > > > > > > > > for
> > > > > > > > > > > each time loop. See this section for more
information:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#directory-
and-filename-template-info
> > > > > > > > > > >
> > > > > > > > > > > 4) The LEAD_SEQ is a list of forecast leads that
will
> be
> > > > > > processed
> > > > > > > > for
> > > > > > > > > > each
> > > > > > > > > > > run time. If you set this to 0 or unset it, then
it
> will
> > > just
> > > > > > > process
> > > > > > > > > > once
> > > > > > > > > > > for each run time.
> > > > > > > > > > >
> > > > > > > > > > > 5) There is documentation describing the time
looping
> > > > variables
> > > > > > > here:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#common-
config-variables
> > > > > > > > > > >
> > > > > > > > > > > This page should answer some of your questions
on how
> > this
> > > > > works.
> > > > > > > > > > >
> > > > > > > > > > > 6) I would recommend starting with the Example
use case
> > > found
> > > > > in
> > > > > > > > > > >
parm/use_cases/met_tool_wrapper/Example/Example.conf.
> > This
> > > > use
> > > > > > case
> > > > > > > > > loops
> > > > > > > > > > > over times, fills in the filename templates
based on
> each
> > > run
> > > > > > time,
> > > > > > > > and
> > > > > > > > > > > prints information to the logfile. You can copy
this
> use
> > > case
> > > > > and
> > > > > > > > > modify
> > > > > > > > > > it
> > > > > > > > > > > to see which filenames are searched for each run
time.
> > This
> > > > is
> > > > > a
> > > > > > > good
> > > > > > > > > way
> > > > > > > > > > > to understand how to configure METplus to find
the
> files
> > > you
> > > > > > need.
> > > > > > > > Once
> > > > > > > > > > > this makes sense, you can look at the PB2NC.conf
use
> case
> > > to
> > > > > see
> > > > > > > how
> > > > > > > > > the
> > > > > > > > > > > config variables control that wrapper.
> > > > > > > > > > >
> > > > > > > > > > > 7) METplus sets environment variables based on
its
> > > > > configuration
> > > > > > > > files.
> > > > > > > > > > > These environment variables are referenced in
the MET
> > > config
> > > > > file
> > > > > > > to
> > > > > > > > > set
> > > > > > > > > > > configurations in that file. If you turn on
DEBUG
> > logging,
> > > > you
> > > > > > can
> > > > > > > > see
> > > > > > > > > > what
> > > > > > > > > > > values are set for these environment variables
to get a
> > > > better
> > > > > > idea
> > > > > > > > of
> > > > > > > > > > how
> > > > > > > > > > > your METplus configuration will affect the MET
> > > configuration.
> > > > > > > > > > >
> > > > > > > > > > > Please let me know if you have any questions
regarding
> > any
> > > of
> > > > > > this.
> > > > > > > > > > >
> > > > > > > > > > > On Tue, May 5, 2020 at 11:46 AM Edward Strobach
- NOAA
> > > > > Affiliate
> > > > > > > via
> > > > > > > > > RT <
> > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
> > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Update:
> > > > > > > > > > > >
> > > > > > > > > > > > Trying to follow the log files that are
generated for
> > > > > guidance.
> > > > > > > I
> > > > > > > > > set
> > > > > > > > > > > the
> > > > > > > > > > > > verbosity to 7, but still can't understand why
I'm
> > > > receiving
> > > > > > this
> > > > > > > > > > > message:
> > > > > > > > > > > >
> > > > > > > > > > > > 05/05 17:41:59.861 metplus INFO: Running
METplus v3.0
> > > > called
> > > > > > with
> > > > > > > > > > > command:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/master_metplus.py
> > > > > > > > > > > > -c
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/use_cases/model_applications/air_quality/CMAQ_obsAirnow.conf
> > > > > > > > > > > > 05/05 17:41:59.882 metplus (met_util.py:776)
ERROR:
> > > > > > > > > > [INIT/VALID]_TIME_FMT
> > > > > > > > > > > > (%Y%m%d%H) does not match
[INIT/VALID]_[BEG/END]
> > > > (2020050200
> > > > > > > > > > > #2020050317)
> > > > > > > > > > > > 05/05 17:41:59.886 metplus.PB2NC
(met_util.py:192)
> > ERROR:
> > > > > Fatal
> > > > > > > > error
> > > > > > > > > > > > occurred
> > > > > > > > > > > >
> > > > > > > > > > > > I tried to start basic. If starting basic
worked,
> > then I
> > > > > would
> > > > > > > > have
> > > > > > > > > > made
> > > > > > > > > > > > it more complex:
> > > > > > > > > > > >
> > > > > > > > > > > > # time looping - options are INIT, VALID,
RETRO, and
> > > > REALTIME
> > > > > > > > > > > > LOG_MET_VERBOSITY = 7
> > > > > > > > > > > >
> > > > > > > > > > > > LOOP_BY = INIT
> > > > > > > > > > > >
> > > > > > > > > > > > # Format of VALID_BEG and VALID_END
> > > > > > > > > > > > INIT_TIME_FMT = %Y%m%d%H
> > > > > > > > > > > >
> > > > > > > > > > > > # Start time, end time, and time increment for
> METplus
> > > run
> > > > > > > > > > > > INIT_BEG = 2020050200
> > #{now?fmt=%Y%m%d%H?shift=-172800}
> > > > > > > > > > > > INIT_END = 2020050200
> #{now?fmt=%Y%m%d%H?shift=-172800}
> > > > > > > #-21600}
> > > > > > > > > > > > INIT_INCREMENT = 0
> > > > > > > > > > > > LEAD_SEQ = 0
> > > > > > > > > > > >
> > > > > > > > > > > > With how this looks, I would have thought that
only
> one
> > > > file
> > > > > > > would
> > > > > > > > be
> > > > > > > > > > > > processed. But it complains about a mismatch
between
> > the
> > > > > > > > > INIT_TIME_FMT
> > > > > > > > > > > and
> > > > > > > > > > > > INIT_BEG/INIT_END
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, May 5, 2020 at 12:01 PM
met_help at ucar.edu
> via
> > > RT <
> > > > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Greetings,
> > > > > > > > > > > > >
> > > > > > > > > > > > > This message has been automatically
generated in
> > > response
> > > > > to
> > > > > > > the
> > > > > > > > > > > creation
> > > > > > > > > > > > > of a trouble ticket regarding:
> > > > > > > > > > > > > "PB2NC_OFFSETS/related questions",
> > > > > > > > > > > > > a summary of which appears below.
> > > > > > > > > > > > >
> > > > > > > > > > > > > There is no need to reply to this message
right
> now.
> > > > Your
> > > > > > > ticket
> > > > > > > > > has
> > > > > > > > > > > > been
> > > > > > > > > > > > > assigned an ID of [rt.rap.ucar.edu #95164].
> > > > > > > > > > > > >
> > > > > > > > > > > > > Please include the string:
> > > > > > > > > > > > >
> > > > > > > > > > > > > [rt.rap.ucar.edu #95164]
> > > > > > > > > > > > >
> > > > > > > > > > > > > in the subject line of all future
correspondence
> > about
> > > > this
> > > > > > > > issue.
> > > > > > > > > To
> > > > > > > > > > > do
> > > > > > > > > > > > > so, you may reply to this message.
> > > > > > > > > > > > >
> > > > > > > > > > > > > For more information, please see:
> > > > > > > > > > > > >
> > > > > > > > > > > > > MET Online Tutorial:
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > >
> > >
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > > > > > > > > > > > >
> > > > > > > > > > > > > MET Users Guide:
> > > > > > > > > > > > >
> > > https://www.dtcenter.org/met/users/docs/overview.php
> > > > > > > > > > > > >
> > > > > > > > > > > > > MET FAQs:
> > > > > > > > > > > > >
> > > > > https://www.dtcenter.org/met/users/support/faqs/index.php
> > > > > > > > > > > > >
> > > > > > > > > > > > > MET-Help Email Archive:
> > > > > > > > > > > > >
http://mailman.ucar.edu/pipermail/met_help
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thank you,
> > > > > > > > > > > > > met_help at ucar.edu
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
>
-------------------------------------------------------------------------
> > > > > > > > > > > > > Good morning,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm trying to configure a file to process
prepbufr
> > data
> > > > to
> > > > > > > netcdf
> > > > > > > > > > > format.
> > > > > > > > > > > > > There are several files that are within the
> directory
> > > > that
> > > > > I
> > > > > > > want
> > > > > > > > > to
> > > > > > > > > > > > > process as shown by the bolded text from the
> > > > > > > > > > > > >
/gpfs/dell1/nco/ops/com/hourly/prod/hourly.20200503
> > > > > > directory:
> > > > > > > > > > > > >
> > > > > > > > > > > > > aqm.t12z.airnow.tm00.bufr_d
> > aqm.t12z.anowpm.pb.tm120
> > > > > > > > > > > > > aqm.t12z.status.tm024.bufr_d
> > > > > > > > > > > > > aqm.t12z.airnow.tm024.bufr_d
> > *aqm.t12z.prepbufr.tm00 *
> > > > > > > > > > > > > aqm.t12z.status.tm120.bufr_d
> > > > > > > > > > > > > aqm.t12z.airnow.tm120.bufr_d
> > *aqm.t12z.prepbufr.tm024*
> > > > > > > > > > > > > obsproc_version_for_t12z_aqm_dump_run
> > > > > > > > > > > > > *aqm.t12z.anowpm.pb.tm00*
> > aqm.t12z.prepbufr.tm120
> > > > > > > > > > > > > where_t12z_aqm_dump_ran
> > > > > > > > > > > > > *aqm.t12z.anowpm.pb.tm024 *
> > > > aqm.t12z.status.tm00.bufr_d
> > > > > > > > > > > > >
> > > > > > > > > > > > > Right now I'm focusing only on the *anowpm*
files
> > (two
> > > > > > files).
> > > > > > > > The
> > > > > > > > > > > tm00
> > > > > > > > > > > > is
> > > > > > > > > > > > > time centered on 12z for 05/03/2020, with
DHR =
> > ranging
> > > > > from
> > > > > > > -11
> > > > > > > > to
> > > > > > > > > > 12,
> > > > > > > > > > > > > covering the whole day. Similarly, the
tm024 file
> is
> > > > > > centered
> > > > > > > at
> > > > > > > > > > 12z,
> > > > > > > > > > > > but
> > > > > > > > > > > > > for 05/02/2020. I looked through the
metplus
> > glossary
> > > > and
> > > > > > > > examples
> > > > > > > > > > to
> > > > > > > > > > > > help
> > > > > > > > > > > > > with the process of building the configure
file.
> If
> > I
> > > > run
> > > > > > this
> > > > > > > > > > script
> > > > > > > > > > > > and
> > > > > > > > > > > > > desire the files from 05/03/2020, then I
should set
> > the
> > > > > > > > following:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > *[config]# time looping - options are INIT,
VALID,
> > > RETRO,
> > > > > and
> > > > > > > > > > > > > REALTIMELOOP_BY = VALID# Format of VALID_BEG
and
> > > > > > > > > > > VALID_ENDVALID_TIME_FMT
> > > > > > > > > > > > =
> > > > > > > > > > > > > %Y%m%d%H# Start time, end time, and time
increment
> > for
> > > > > > METplus
> > > > > > > > > > > > runVALID_BEG
> > > > > > > > > > > > > = {now?fmt=%Y%m%d%H?shift=-2d}VALID_END =
> > > > > > > > > > {now?fmt=%Y%m%d%H?shift=-2d}
> > > > > > > > > > > > > #-21600}VALID_INCREMENT = 24HLEAD_SEQ = 0,24
> > (((NOT
> > > > > SURE I
> > > > > > > > NEED
> > > > > > > > > > > THIS,
> > > > > > > > > > > > > BUT NOT SURE HOW THE LOOP WILL WORK WITHOUT
IT?)))*
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > *LOOP_ORDER = times# List of applications to
> > > > > runPROCESS_LIST
> > > > > > =
> > > > > > > > > > > > > PB2NCCONFIG_DIR =
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/met_configCONFIG_FILE
> > > > > > > > > > > > > =
> > > > {CONFIG_DIR}/PB2NCConfig_wrappedPB2NC_OBS_BUFR_VAR_LIST =
> > > > > > > DHR,
> > > > > > > > > RPT,
> > > > > > > > > > > > SID,
> > > > > > > > > > > > > YOB, XOB, TYPO, TPHR, QCIND, COPO, COPOPM*
> > > > > > > > > > > > >
> > > > > > > > > > > > > I've listed the variables I would like to
extract
> > from
> > > > each
> > > > > > > file
> > > > > > > > as
> > > > > > > > > > > well.
> > > > > > > > > > > > > It's not clear if I need the config file,
but I
> > listed
> > > it
> > > > > > just
> > > > > > > in
> > > > > > > > > > case.
> > > > > > > > > > > > > Here's what I have for the file names from
the
> input
> > > > > > directory
> > > > > > > > and
> > > > > > > > > > > what I
> > > > > > > > > > > > > would name it once converted to netcdf and
placed
> in
> > > the
> > > > > > output
> > > > > > > > > > > > directory:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
*INPUT_BASE=/gpfs/dell1/nco/ops/com/hourly/prod/hourly.{now?fmt=%Y%m%d?shift=-
2d}
> > > > > > > > > > > > > # -86400}PB2NC_INPUT_DIR =
> {INPUT_BASE}OUTPUT_BASE =
> > > > > > > > > > > > >
> > > > > > >
> /gpfs/dell2/emc/modeling/noscrub/Edward.StrobachPB2NC_OUTPUT_DIR
> > > > > > > > =
> > > > > > > > > > > > >
> > > > > {OUTPUT_BASE}/AirNow/pb2nc/hourly.{now?fmt=%Y%m%d?shift=-2d}
> > > > > > #
> > > > > > > > > > > > >
-86400}[filename_templates]PB2NC_INPUT_TEMPLATE =
> > > > > > > > > > > > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ}
(((RELATED TO
> LOOP
> > > > > > > > > > > > > CONFUSION)))PB2NC_OUTPUT_TEMPLATE =
> > > > > > > > > > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ}.nc*
> > > > > > > > > > > > >
> > > > > > > > > > > > > Let me know if more information is needed.
My main
> > > > > concerns
> > > > > > > here
> > > > > > > > > are
> > > > > > > > > > > 1)
> > > > > > > > > > > > > being able to process all the list files
correctly,
> > 2)
> > > > > making
> > > > > > > > sure
> > > > > > > > > > the
> > > > > > > > > > > > > looping is correct, and 3) making sure that
I've
> > > included
> > > > > > > > > everything
> > > > > > > > > > > > that I
> > > > > > > > > > > > > would need to process this step
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > Edward Strobach
> > > > > > > > > > > > > EMC/NCEP/NWS/
> > > > > > > > > > > > > IMSG Contractor
> > > > > > > > > > > > > Cubicle#: 2029
> > > > > > > > > > > > > 301-683-3717
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Edward Strobach
> > > > > > > > > > > > EMC/NCEP/NWS/
> > > > > > > > > > > > IMSG Contractor
> > > > > > > > > > > > Cubicle#: 2029
> > > > > > > > > > > > 301-683-3717
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > > 303-497-2768
> > > > > > > > > > > ---
> > > > > > > > > > > My working day may not be your working day.
Please do
> not
> > > > feel
> > > > > > > > obliged
> > > > > > > > > to
> > > > > > > > > > > reply to this email outside of your normal
working
> hours.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Edward Strobach
> > > > > > > > > > EMC/NCEP/NWS/
> > > > > > > > > > IMSG Contractor
> > > > > > > > > > Cubicle#: 2029
> > > > > > > > > > 301-683-3717
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > Research Applications Laboratory
> > > > > > > > > 303-497-2768
> > > > > > > > > ---
> > > > > > > > > My working day may not be your working day. Please
do not
> > feel
> > > > > > obliged
> > > > > > > to
> > > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Edward Strobach
> > > > > > > > EMC/NCEP/NWS/
> > > > > > > > IMSG Contractor
> > > > > > > > Cubicle#: 2029
> > > > > > > > 301-683-3717
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > George McCabe - Software Engineer III
> > > > > > > National Center for Atmospheric Research
> > > > > > > Research Applications Laboratory
> > > > > > > 303-497-2768
> > > > > > > ---
> > > > > > > My working day may not be your working day. Please do
not feel
> > > > obliged
> > > > > to
> > > > > > > reply to this email outside of your normal working
hours.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Edward Strobach
> > > > > > EMC/NCEP/NWS/
> > > > > > IMSG Contractor
> > > > > > Cubicle#: 2029
> > > > > > 301-683-3717
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > > --
> > > > Edward Strobach
> > > > EMC/NCEP/NWS/
> > > > IMSG Contractor
> > > > Cubicle#: 2029
> > > > 301-683-3717
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: PB2NC_OFFSETS/related questions
From: George McCabe
Time: Thu May 07 17:57:32 2020
Hi Edward,
Does 4:15pm EST on Wednesday work for you?
On Thu, May 7, 2020 at 5:38 PM Edward Strobach - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
>
> I understand. Are you good some time Tues. or Wed? No meetings for
me
> those days so any time works
>
> On Thu, May 7, 2020 at 7:21 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Edward,
> >
> > John and I have had a full day today and I am taking the day off
on
> Friday.
> > Let's schedule a time to meet next week. Do any days/time work
better for
> > you?
> >
> > On Thu, May 7, 2020 at 10:54 AM Edward Strobach - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> > >
> > > Removing the JLOGFILE from .bashrc moved me along nicely. I
think
> > > everything processed just fine.
> > >
> > > In terms of setting up a meeting, that would be great.
Sometimes a
> nudge
> > > in the right direction is all that is needed. My goals, both
short-term
> > and
> > > long-term, would be run this script, generate plots (I've seen
> > > plot_data_plane and plot_point_obs) both stats and maps, and set
this
> up
> > in
> > > a batch system so to process it with a cron job so that I have a
time
> > > series of stats for different regions, and maps with site
overlays.
> > > Eventually I can expand this to other fields if needed. Right
now I'm
> not
> > > exactly sure how to incorporate plotting utilities in this
process,
> but I
> > > assume it's just a matter of setting up another config file with
path
> > > information, processes to be used, and information related to
time. I
> > have
> > > a meeting in a few minutes but am free otherwise. We could find
a time
> > > that works for all of us, perhaps?
> > >
> > > On Thu, May 7, 2020 at 11:56 AM George McCabe via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Edward,
> > > >
> > > > Thanks for sending all of this information.
> > > >
> > > > I think the fatal error lies in this line in your bashrc:
> > > >
> > > > export JLOGFILE=~/
> > > >
> > > > I would recommend not setting JLOGFILE at all, as it isn't
necessary
> to
> > > run
> > > > METplus. I believe it should be set to a filepath to write job
logs,
> so
> > > > setting this to your home directory is causing the fatal
error. I am
> > > > guessing it only writes to the JLOGFILE when there is an
error, and
> it
> > > > can't write to ~/ because it is a directory not a file.
Removing that
> > > from
> > > > your bashrc should fix the fatal error you are experiencing.
> > > >
> > > > It might be beneficial to set up a meeting with you, me, and
John
> > Halley
> > > > Gotway (MET engineer) to share screens and step through your
current
> > > setup
> > > > to help you figure out how to set up what you want to do. We
have
> done
> > > this
> > > > with Perry in the past and it proved to be very productive.
Let me
> know
> > > if
> > > > you would like to do that and we can set up a time to meet.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Thu, May 7, 2020 at 7:32 AM Edward Strobach - NOAA
Affiliate via
> RT
> > <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> > > > >
> > > > > 1) I didn't see a metplus_final.conf file in OUTPUT_BASE,
but based
> > on
> > > > your
> > > > > last email, I am guessing you changed the value of
METPLUS_CONF to
> > > create
> > > > > those files you listed. Which of the pb2nc_pointstat conf
files was
> > > used
> > > > > for the runs that are producing the error? Could you send me
that
> > one?
> > > > When
> > > > > you run a use case, the name of the final conf is output to
the
> > screen
> > > > > towards the beginning of execution, but unfortunately not
added to
> > the
> > > > > logfile currently.
> > > > >
> > > > > They both produce the error. One of them looks for an ozone
file
> > which
> > > > > doesn't exist (aqm.t00z.awpozcon.f00.148.grib2) and the
other looks
> > > for a
> > > > > pm file which doesn't exist (aqm.t00z.pm25.f00.148.grib2).
*f00*
> are
> > > not
> > > > > saved as output from the CMAQ runs while the others (*fXY* )
are
> > being
> > > > > saved. I have *.conf files that are created for each
process
> > > > > (shared.conf_aq; aq.conf; pm.conf; and statanalysis.conf).
These
> are
> > > > based
> > > > > on inputs from my runs script. These are located here:
> > > > > /gpfs/dell2/ptmp/Edward.Strobach/metplus_aq
> > > > >
> > > > > When running the master_metplus.py command, I use several
*.conf
> > > > > files: *${MET_PLUS}/ush/master_metplus.py
> > > > > -c ${MET_PLUS}/parm/use_cases/grid_to_obs/grid_to_obs.conf
-c
> > > > >
${MET_PLUS}/parm/use_cases/grid_to_obs/examples/conus_surface.conf
> -c
> > > > > ${MET_PLUS_CONF}/pb2nc_aq.conf -c
> ${MET_PLUS_CONF}/point_stat_aq.conf
> > > -c
> > > > > ${MET_PLUS_TMP}/${model}.conf -c
${MET_PLUS_CONF}/shared.conf -c
> > > > > ${MET_PLUS_TMP}/shared.conf_aq -c
${MET_PLUS_CONF}/system_aq.conf*
> > > > >
> > > > > Here, model stands for "aq" or "pm"
> > > > >
> > > > > Lastly, all the conf files that are stored in the
OUTPUT_BASE/conf
> > can
> > > be
> > > > > found in the aq directory (two of them) and pm directory (2
of
> them).
> > > > > There is a log file located here:
> > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/logs/aq/20200504
> > > > > and
> > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/logs/pm/20200504.
> > > > > Both of them have this error message in them: 05/06
17:36:43.877
> > > > > metplus.StatAnalysis (stat_analysis_wrapper.py:237) DEBUG:
Items in
> > > these
> > > > > lists will be looped over: FCST_VALID_HOUR_LIST
> > > > >
> > > > > This, to me, is clearly related to the issue of trying to
find that
> > > file
> > > > > that doesn't exist. I'm not 100%, but I think the conf file
that
> you
> > > want
> > > > > is this: metplus_final_statanalysis_gatherbydate.conf. I
copied
> over
> > > the
> > > > > contents and stored them in the attachment below. I'm still
trying
> to
> > > > > understand what exactly is being processed based on the
contents
> > > inside.
> > > > >
> > > > >
> > > > > 2) It looks like your master_metplus.py command has a lot of
extra
> > conf
> > > > > files that don't need to be in there. The grid_to_obs conf
files
> are
> > > from
> > > > > an old version of METplus, so I would think they would cause
errors
> > > using
> > > > > deprecated variables. The corresponding use case in METplus
3.0 for
> > the
> > > > > upper_air.conf use case is:
> > > > >
> > > > > METplus/parm/use_cases/model_applications/medium_range/Point
> > > > > Stat_fcstGFS_obsGDAS_UpperAir_MultiField_PrepBufr.conf
> > > > >
> > > > > I don't really know, but I would assume that there would be
> specific
> > > > errors
> > > > > related to deprecated variables that would suggest as much.
I
> didn't
> > > see
> > > > > that though. I could attach the valid config helper, though,
just
> in
> > > > case,
> > > > > right?
> > > > >
> > > > > I would start with running that conf with a user config and
if that
> > > > works,
> > > > > copy the use case conf to another file I and start modifying
that
> > > instead
> > > > > of
> > > > > adding new confs onto the command. That will keep things a
little
> > more
> > > > > organized for you.
> > > > >
> > > > > I can look into trying that. I'm not very comfortable yet
with
> > metplus
> > > to
> > > > > make serious changes, so it might take a little bit of time
to
> change
> > > the
> > > > > set-up and make a test.
> > > > >
> > > > > 3) The error message is very odd. It looks like it was
logging just
> > > fine
> > > > up
> > > > > until that error, then it was trying to write to a log
"file" named
> > > > > /u/Edward.Strobach and crashing because that is a directory.
I'm
> not
> > > > > understanding why it would change the log file in the middle
of
> > > execution
> > > > > and on top of that still write the offending error to the
log. I am
> > > very
> > > > > baffled by that. I will try to recreate this error locally.
> > > > >
> > > > > Could it be how I have things set in my .bashrc? I have
these
> > related
> > > > > settings:
> > > > > module load python/3.6.3
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *# Add METplus to $PATHmodule use
> > > > >
> > >
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/modulefiles/module
> > > > > load met/9.0module load NCO/4.7.0module load
grib_util/1.1.1export
> > > > >
> > > > >
> > > >
> > >
> >
>
PATH=${PATH}:/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/METplus/METplus-
3.0/ushexport
> > > > > PATH=~/METplus/ush:$PATH# Optional JLOGFILEexport
JLOGFILE=~/module
> > > load
> > > > > prod_util/1.1.3*
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, May 6, 2020 at 6:46 PM George McCabe via RT <
> > met_help at ucar.edu
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Edward,
> > > > > >
> > > > > > Julie was able to login and send me some of your log files
and
> info
> > > > > while I
> > > > > > was in a meeting.
> > > > > >
> > > > > > 1) I didn't see a metplus_final.conf file in OUTPUT_BASE,
but
> based
> > > on
> > > > > your
> > > > > > last email, I am guessing you changed the value of
METPLUS_CONF
> to
> > > > create
> > > > > > those files you listed. Which of the pb2nc_pointstat conf
files
> was
> > > > used
> > > > > > for the runs that are producing the error? Could you send
me that
> > > one?
> > > > > When
> > > > > > you run a use case, the name of the final conf is output
to the
> > > screen
> > > > > > towards the beginning of execution, but unfortunately not
added
> to
> > > the
> > > > > > logfile currently.
> > > > > >
> > > > > >
> > > > > > 2) It looks like your master_metplus.py command has a lot
of
> extra
> > > conf
> > > > > > files that don't need to be in there. The grid_to_obs conf
files
> > are
> > > > from
> > > > > > an old version of METplus, so I would think they would
cause
> errors
> > > > using
> > > > > > deprecated variables. The corresponding use case in
METplus 3.0
> for
> > > the
> > > > > > upper_air.conf use case is:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
METplus/parm/use_cases/model_applications/medium_range/PointStat_fcstGFS_obsGDAS_UpperAir_MultiField_PrepBufr.conf
> > > > > >
> > > > > > I would start with running that conf with a user config
and if
> that
> > > > > works,
> > > > > > copy the use case conf to another file and start modifying
that
> > > instead
> > > > > of
> > > > > > adding new confs onto the command. That will keep things a
little
> > > more
> > > > > > organized for you.
> > > > > >
> > > > > >
> > > > > > 3) The error message is very odd. It looks like it was
logging
> just
> > > > fine
> > > > > up
> > > > > > until that error, then it was trying to write to a log
"file"
> named
> > > > > > /u/Edward.Strobach and crashing because that is a
directory. I'm
> > not
> > > > > > understanding why it would change the log file in the
middle of
> > > > execution
> > > > > > and on top of that still write the offending error to the
log. I
> am
> > > > very
> > > > > > baffled by that. I will try to recreate this error
locally.
> > > > > >
> > > > > > On Wed, May 6, 2020 at 1:43 PM Edward Strobach - NOAA
Affiliate
> via
> > > RT
> > > > <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
> >
> > > > > > >
> > > > > > > So my OUTPUT_BASE is here:
> > > > > > >
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq.
> > > > > > >
> > > > > > > There is a conf directory where .conf files are stored
in aq
> > > > > > > (OUTPUT_BASE/conf/aq) and pm (OUTPUT_BASE/conf/pm).
There are
> > some
> > > > > .conf
> > > > > > > files from these runs.
> > > > > > >
> > > > > > > aq:
> > > > > > > metplus_final_pb2nc_pointstat.conf
> > > > > > > metplus_final_statanalysis_gatherbydate.conf
> > > > > > >
> > > > > > > pm:
> > > > > > > metplus_final_pb2nc_pointstat.conf
> > > > > > > metplus_final_statanalysis_gatherbydate.conf
> > > > > > >
> > > > > > > Any file in particular that you want to see? I don't
have the
> > file
> > > > you
> > > > > > are
> > > > > > > referring to.
> > > > > > >
> > > > > > > On Wed, May 6, 2020 at 3:22 PM George McCabe via RT <
> > > > met_help at ucar.edu
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > That's it, thanks.
> > > > > > > >
> > > > > > > > Hmm, it looks like there is an error trying to add to
the log
> > > file?
> > > > > Did
> > > > > > > > this run generate a log file? It would be found in the
logs
> > > > directory
> > > > > > > under
> > > > > > > > the value you set for OUTPUT_BASE. If there is one,
can you
> > send
> > > > that
> > > > > > to
> > > > > > > > me? Also, please send the metplus_final.conf file in
your
> > > > OUTPUT_BASE
> > > > > > as
> > > > > > > > well.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, May 6, 2020 at 1:03 PM Edward Strobach - NOAA
> Affiliate
> > > via
> > > > > RT
> > > > > > <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
> > > >
> > > > > > > > >
> > > > > > > > > Sure thing. I think you are referring to this?
> > > > > > > > >
> > > > > > > > > 05/06 17:34:09.831 metplus.PointStat
> (command_builder.py:100)
> > > > > ERROR:
> > > > > > > > > (command_builder.py:437) Could not find FCST file
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
> > > > > > > > > using template
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
> > > > > > > > > 05/06 17:34:09.832 metplus.PointStat
(met_util.py:192)
> ERROR:
> > > > Fatal
> > > > > > > error
> > > > > > > > > occurred
> > > > > > > > > *Traceback (most recent call last):*
> > > > > > > > > File
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > > > > > > line 169, in run_metplus
> > > > > > > > > process.run_all_times()
> > > > > > > > > File
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > > > > line 751, in run_all_times
> > > > > > > > > util.loop_over_times_and_call(self.config, self)
> > > > > > > > > File
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > > > > > > line 834, in loop_over_times_and_call
> > > > > > > > > process.run_at_time(input_dict)
> > > > > > > > > File
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/compare_gridded_wrapper.py",
> > > > > > > > > line 183, in run_at_time
> > > > > > > > > self.run_at_time_once(time_info)
> > > > > > > > > File
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/point_stat_wrapper.py",
> > > > > > > > > line 146, in run_at_time_once
> > > > > > > > > model_path = self.find_model(time_info,
var_list[0])
> > > > > > > > > File
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > > > > line 319, in find_model
> > > > > > > > > return_list=return_list)
> > > > > > > > > File
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > > > > line 373, in find_data
> > > > > > > > > return self.find_exact_file(**arg_dict)
> > > > > > > > > File
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > > > > line 437, in find_exact_file
> > > > > > > > > self.log_error(msg)
> > > > > > > > > File
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > > > > line 100, in log_error
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
>
self.logger.error(f"({os.path.basename(caller.filename)}:{caller.lineno})
> > > > > > > > > {error_string}")
> > > > > > > > > File
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > > > > line 1335, in error
> > > > > > > > > self._log(ERROR, msg, args, **kwargs)
> > > > > > > > > File
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > > > > line 1442, in _log
> > > > > > > > > self.handle(record)
> > > > > > > > > File
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > > > > line 1452, in handle
> > > > > > > > > self.callHandlers(record)
> > > > > > > > > File
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > > > > line 1514, in callHandlers
> > > > > > > > > hdlr.handle(record)
> > > > > > > > > File
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > > > > line 863, in handle
> > > > > > > > > self.emit(record)
> > > > > > > > > File
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/produtil/log.py",
> > > > > > > > > line 227, in emit
> > > > > > > > > with open(self._logger,'at') as f:
> > > > > > > > >
> > > > > > > > > On Wed, May 6, 2020 at 2:55 PM George McCabe via RT
<
> > > > > > met_help at ucar.edu
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Edward,
> > > > > > > > > >
> > > > > > > > > > When you cloned the repo yourself, are you using
the
> > develop
> > > > > branch
> > > > > > > to
> > > > > > > > > run?
> > > > > > > > > > There may have been a bug introduced that is
causing that
> > > fatal
> > > > > > > error.
> > > > > > > > > > Could you send me the stack trace that comes after
the
> > fatal
> > > > > > error? I
> > > > > > > > > agree
> > > > > > > > > > that it shouldn't crash if it can't find a file.
> > > > > > > > > >
> > > > > > > > > > On Wed, May 6, 2020 at 12:43 PM Edward Strobach -
NOAA
> > > > Affiliate
> > > > > > via
> > > > > > > > RT <
> > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
> > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks George. I've been busy trying to figure
things
> > out
> > > > and
> > > > > > > forgot
> > > > > > > > > to
> > > > > > > > > > > respond. I talked with Perry about some of
these
> things
> > > and
> > > > he
> > > > > > was
> > > > > > > > > > > surprised that I was receiving this message when
> running
> > > his
> > > > > > > script:
> > > > > > > > > > >
> > > > > > > > > > > 05/06 17:34:09.517 metplus.PointStat INFO:
Processing
> > > > forecast
> > > > > > > lead 0
> > > > > > > > > > hours
> > > > > > > > > > > 05/06 17:34:09.525 metplus.PointStat
> > > (command_builder.py:407)
> > > > > > > DEBUG:
> > > > > > > > > > > Looking for FCST file
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
> > > > > > > > > > > 05/06 17:34:09.831 metplus.PointStat
> > > (command_builder.py:100)
> > > > > > > ERROR:
> > > > > > > > > > > (command_builder.py:437) Could not find FCST
file
> > > > > > > > > > > /gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/
> > > > > > > > > > > *aqm.t00z.awpozcon.f00.148.grib2* using template
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
> > > > > > > > > > > 05/06 17:34:09.832 metplus.PointStat
(met_util.py:192)
> > > ERROR:
> > > > > > Fatal
> > > > > > > > > error
> > > > > > > > > > > occurred
> > > > > > > > > > >
> > > > > > > > > > > The bolded item does not exist. When Perry runs
his
> > script
> > > > it
> > > > > > > skips
> > > > > > > > > this
> > > > > > > > > > > file. When I run the script I get a fatal
error. I
> > cloned
> > > > > > metplus
> > > > > > > > > just
> > > > > > > > > > > the other day. I'm not sure if this is related
to
> item 2
> > > > from
> > > > > > your
> > > > > > > > > email
> > > > > > > > > > >
> > > > > > > > > > > On Tue, May 5, 2020 at 2:37 PM George McCabe via
RT <
> > > > > > > > met_help at ucar.edu
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Edward,
> > > > > > > > > > > >
> > > > > > > > > > > > 1) The LOG_MET_VERBOSITY variable sets the
verbosity
> > > level
> > > > > > when a
> > > > > > > > MET
> > > > > > > > > > > tool
> > > > > > > > > > > > is run. To change the METplus logging level,
set
> > > > LOG_LEVEL. I
> > > > > > > would
> > > > > > > > > > > > recommend changing it to DEBUG to get more
> information.
> > > > > > > > > > > >
> > > > > > > > > > > > 2) There is a bug in using the {now} tags for
> > > > > > > > [INIT/VALID]_[BEG/END]
> > > > > > > > > > that
> > > > > > > > > > > > causes that crash. This has been fixed but we
have
> not
> > > yet
> > > > > > > > released a
> > > > > > > > > > > > bugfix release. I will let the engineers know
that
> you
> > > need
> > > > > > this
> > > > > > > > > > > > functionality and we will push the METplus
3.0.1
> bugfix
> > > > > release
> > > > > > > > soon.
> > > > > > > > > > You
> > > > > > > > > > > > should be able to set the time explicitly,
i.e.
> > > 2020050500.
> > > > > > > > > > > >
> > > > > > > > > > > > 3) The {now} tag can be used in the VALID_BEG
and
> > > VALID_END
> > > > > > > config
> > > > > > > > > > > > variables, but you will want to refer to init,
valid,
> > > lead,
> > > > > > etc.
> > > > > > > in
> > > > > > > > > > your
> > > > > > > > > > > > INPUT_TEMPLATE and OUTPUT_TEMPLATE variables
so that
> it
> > > > uses
> > > > > > the
> > > > > > > > time
> > > > > > > > > > for
> > > > > > > > > > > > each time loop. See this section for more
> information:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#directory-
and-filename-template-info
> > > > > > > > > > > >
> > > > > > > > > > > > 4) The LEAD_SEQ is a list of forecast leads
that will
> > be
> > > > > > > processed
> > > > > > > > > for
> > > > > > > > > > > each
> > > > > > > > > > > > run time. If you set this to 0 or unset it,
then it
> > will
> > > > just
> > > > > > > > process
> > > > > > > > > > > once
> > > > > > > > > > > > for each run time.
> > > > > > > > > > > >
> > > > > > > > > > > > 5) There is documentation describing the time
looping
> > > > > variables
> > > > > > > > here:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#common-
config-variables
> > > > > > > > > > > >
> > > > > > > > > > > > This page should answer some of your questions
on how
> > > this
> > > > > > works.
> > > > > > > > > > > >
> > > > > > > > > > > > 6) I would recommend starting with the Example
use
> case
> > > > found
> > > > > > in
> > > > > > > > > > > >
parm/use_cases/met_tool_wrapper/Example/Example.conf.
> > > This
> > > > > use
> > > > > > > case
> > > > > > > > > > loops
> > > > > > > > > > > > over times, fills in the filename templates
based on
> > each
> > > > run
> > > > > > > time,
> > > > > > > > > and
> > > > > > > > > > > > prints information to the logfile. You can
copy this
> > use
> > > > case
> > > > > > and
> > > > > > > > > > modify
> > > > > > > > > > > it
> > > > > > > > > > > > to see which filenames are searched for each
run
> time.
> > > This
> > > > > is
> > > > > > a
> > > > > > > > good
> > > > > > > > > > way
> > > > > > > > > > > > to understand how to configure METplus to find
the
> > files
> > > > you
> > > > > > > need.
> > > > > > > > > Once
> > > > > > > > > > > > this makes sense, you can look at the
PB2NC.conf use
> > case
> > > > to
> > > > > > see
> > > > > > > > how
> > > > > > > > > > the
> > > > > > > > > > > > config variables control that wrapper.
> > > > > > > > > > > >
> > > > > > > > > > > > 7) METplus sets environment variables based on
its
> > > > > > configuration
> > > > > > > > > files.
> > > > > > > > > > > > These environment variables are referenced in
the MET
> > > > config
> > > > > > file
> > > > > > > > to
> > > > > > > > > > set
> > > > > > > > > > > > configurations in that file. If you turn on
DEBUG
> > > logging,
> > > > > you
> > > > > > > can
> > > > > > > > > see
> > > > > > > > > > > what
> > > > > > > > > > > > values are set for these environment variables
to
> get a
> > > > > better
> > > > > > > idea
> > > > > > > > > of
> > > > > > > > > > > how
> > > > > > > > > > > > your METplus configuration will affect the MET
> > > > configuration.
> > > > > > > > > > > >
> > > > > > > > > > > > Please let me know if you have any questions
> regarding
> > > any
> > > > of
> > > > > > > this.
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, May 5, 2020 at 11:46 AM Edward
Strobach -
> NOAA
> > > > > > Affiliate
> > > > > > > > via
> > > > > > > > > > RT <
> > > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
> > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Update:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Trying to follow the log files that are
generated
> for
> > > > > > guidance.
> > > > > > > > I
> > > > > > > > > > set
> > > > > > > > > > > > the
> > > > > > > > > > > > > verbosity to 7, but still can't understand
why I'm
> > > > > receiving
> > > > > > > this
> > > > > > > > > > > > message:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 05/05 17:41:59.861 metplus INFO: Running
METplus
> v3.0
> > > > > called
> > > > > > > with
> > > > > > > > > > > > command:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/master_metplus.py
> > > > > > > > > > > > > -c
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/use_cases/model_applications/air_quality/CMAQ_obsAirnow.conf
> > > > > > > > > > > > > 05/05 17:41:59.882 metplus (met_util.py:776)
ERROR:
> > > > > > > > > > > [INIT/VALID]_TIME_FMT
> > > > > > > > > > > > > (%Y%m%d%H) does not match
[INIT/VALID]_[BEG/END]
> > > > > (2020050200
> > > > > > > > > > > > #2020050317)
> > > > > > > > > > > > > 05/05 17:41:59.886 metplus.PB2NC
(met_util.py:192)
> > > ERROR:
> > > > > > Fatal
> > > > > > > > > error
> > > > > > > > > > > > > occurred
> > > > > > > > > > > > >
> > > > > > > > > > > > > I tried to start basic. If starting basic
worked,
> > > then I
> > > > > > would
> > > > > > > > > have
> > > > > > > > > > > made
> > > > > > > > > > > > > it more complex:
> > > > > > > > > > > > >
> > > > > > > > > > > > > # time looping - options are INIT, VALID,
RETRO,
> and
> > > > > REALTIME
> > > > > > > > > > > > > LOG_MET_VERBOSITY = 7
> > > > > > > > > > > > >
> > > > > > > > > > > > > LOOP_BY = INIT
> > > > > > > > > > > > >
> > > > > > > > > > > > > # Format of VALID_BEG and VALID_END
> > > > > > > > > > > > > INIT_TIME_FMT = %Y%m%d%H
> > > > > > > > > > > > >
> > > > > > > > > > > > > # Start time, end time, and time increment
for
> > METplus
> > > > run
> > > > > > > > > > > > > INIT_BEG = 2020050200
> > > #{now?fmt=%Y%m%d%H?shift=-172800}
> > > > > > > > > > > > > INIT_END = 2020050200
> > #{now?fmt=%Y%m%d%H?shift=-172800}
> > > > > > > > #-21600}
> > > > > > > > > > > > > INIT_INCREMENT = 0
> > > > > > > > > > > > > LEAD_SEQ = 0
> > > > > > > > > > > > >
> > > > > > > > > > > > > With how this looks, I would have thought
that only
> > one
> > > > > file
> > > > > > > > would
> > > > > > > > > be
> > > > > > > > > > > > > processed. But it complains about a
mismatch
> between
> > > the
> > > > > > > > > > INIT_TIME_FMT
> > > > > > > > > > > > and
> > > > > > > > > > > > > INIT_BEG/INIT_END
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, May 5, 2020 at 12:01 PM
met_help at ucar.edu
> > via
> > > > RT <
> > > > > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Greetings,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This message has been automatically
generated in
> > > > response
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > > creation
> > > > > > > > > > > > > > of a trouble ticket regarding:
> > > > > > > > > > > > > > "PB2NC_OFFSETS/related questions",
> > > > > > > > > > > > > > a summary of which appears below.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > There is no need to reply to this message
right
> > now.
> > > > > Your
> > > > > > > > ticket
> > > > > > > > > > has
> > > > > > > > > > > > > been
> > > > > > > > > > > > > > assigned an ID of [rt.rap.ucar.edu
#95164].
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Please include the string:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [rt.rap.ucar.edu #95164]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > in the subject line of all future
correspondence
> > > about
> > > > > this
> > > > > > > > > issue.
> > > > > > > > > > To
> > > > > > > > > > > > do
> > > > > > > > > > > > > > so, you may reply to this message.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For more information, please see:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > MET Online Tutorial:
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > >
> > > >
https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > MET Users Guide:
> > > > > > > > > > > > > >
> > > > https://www.dtcenter.org/met/users/docs/overview.php
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > MET FAQs:
> > > > > > > > > > > > > >
> > > > > > https://www.dtcenter.org/met/users/support/faqs/index.php
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > MET-Help Email Archive:
> > > > > > > > > > > > > >
http://mailman.ucar.edu/pipermail/met_help
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you,
> > > > > > > > > > > > > > met_help at ucar.edu
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >
-------------------------------------------------------------------------
> > > > > > > > > > > > > > Good morning,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I'm trying to configure a file to process
> prepbufr
> > > data
> > > > > to
> > > > > > > > netcdf
> > > > > > > > > > > > format.
> > > > > > > > > > > > > > There are several files that are within
the
> > directory
> > > > > that
> > > > > > I
> > > > > > > > want
> > > > > > > > > > to
> > > > > > > > > > > > > > process as shown by the bolded text from
the
> > > > > > > > > > > > > >
> /gpfs/dell1/nco/ops/com/hourly/prod/hourly.20200503
> > > > > > > directory:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > aqm.t12z.airnow.tm00.bufr_d
> > > aqm.t12z.anowpm.pb.tm120
> > > > > > > > > > > > > > aqm.t12z.status.tm024.bufr_d
> > > > > > > > > > > > > > aqm.t12z.airnow.tm024.bufr_d
> > > *aqm.t12z.prepbufr.tm00 *
> > > > > > > > > > > > > > aqm.t12z.status.tm120.bufr_d
> > > > > > > > > > > > > > aqm.t12z.airnow.tm120.bufr_d
> > > *aqm.t12z.prepbufr.tm024*
> > > > > > > > > > > > > > obsproc_version_for_t12z_aqm_dump_run
> > > > > > > > > > > > > > *aqm.t12z.anowpm.pb.tm00*
> > > aqm.t12z.prepbufr.tm120
> > > > > > > > > > > > > > where_t12z_aqm_dump_ran
> > > > > > > > > > > > > > *aqm.t12z.anowpm.pb.tm024 *
> > > > > aqm.t12z.status.tm00.bufr_d
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Right now I'm focusing only on the
*anowpm* files
> > > (two
> > > > > > > files).
> > > > > > > > > The
> > > > > > > > > > > > tm00
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > time centered on 12z for 05/03/2020, with
DHR =
> > > ranging
> > > > > > from
> > > > > > > > -11
> > > > > > > > > to
> > > > > > > > > > > 12,
> > > > > > > > > > > > > > covering the whole day. Similarly, the
tm024
> file
> > is
> > > > > > > centered
> > > > > > > > at
> > > > > > > > > > > 12z,
> > > > > > > > > > > > > but
> > > > > > > > > > > > > > for 05/02/2020. I looked through the
metplus
> > > glossary
> > > > > and
> > > > > > > > > examples
> > > > > > > > > > > to
> > > > > > > > > > > > > help
> > > > > > > > > > > > > > with the process of building the configure
file.
> > If
> > > I
> > > > > run
> > > > > > > this
> > > > > > > > > > > script
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > desire the files from 05/03/2020, then I
should
> set
> > > the
> > > > > > > > > following:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *[config]# time looping - options are
INIT,
> VALID,
> > > > RETRO,
> > > > > > and
> > > > > > > > > > > > > > REALTIMELOOP_BY = VALID# Format of
VALID_BEG and
> > > > > > > > > > > > VALID_ENDVALID_TIME_FMT
> > > > > > > > > > > > > =
> > > > > > > > > > > > > > %Y%m%d%H# Start time, end time, and time
> increment
> > > for
> > > > > > > METplus
> > > > > > > > > > > > > runVALID_BEG
> > > > > > > > > > > > > > = {now?fmt=%Y%m%d%H?shift=-2d}VALID_END =
> > > > > > > > > > > {now?fmt=%Y%m%d%H?shift=-2d}
> > > > > > > > > > > > > > #-21600}VALID_INCREMENT = 24HLEAD_SEQ =
0,24
> > > (((NOT
> > > > > > SURE I
> > > > > > > > > NEED
> > > > > > > > > > > > THIS,
> > > > > > > > > > > > > > BUT NOT SURE HOW THE LOOP WILL WORK
WITHOUT
> IT?)))*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *LOOP_ORDER = times# List of applications
to
> > > > > > runPROCESS_LIST
> > > > > > > =
> > > > > > > > > > > > > > PB2NCCONFIG_DIR =
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/met_configCONFIG_FILE
> > > > > > > > > > > > > > =
> > > > > {CONFIG_DIR}/PB2NCConfig_wrappedPB2NC_OBS_BUFR_VAR_LIST =
> > > > > > > > DHR,
> > > > > > > > > > RPT,
> > > > > > > > > > > > > SID,
> > > > > > > > > > > > > > YOB, XOB, TYPO, TPHR, QCIND, COPO, COPOPM*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I've listed the variables I would like to
extract
> > > from
> > > > > each
> > > > > > > > file
> > > > > > > > > as
> > > > > > > > > > > > well.
> > > > > > > > > > > > > > It's not clear if I need the config file,
but I
> > > listed
> > > > it
> > > > > > > just
> > > > > > > > in
> > > > > > > > > > > case.
> > > > > > > > > > > > > > Here's what I have for the file names from
the
> > input
> > > > > > > directory
> > > > > > > > > and
> > > > > > > > > > > > what I
> > > > > > > > > > > > > > would name it once converted to netcdf and
placed
> > in
> > > > the
> > > > > > > output
> > > > > > > > > > > > > directory:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
*INPUT_BASE=/gpfs/dell1/nco/ops/com/hourly/prod/hourly.{now?fmt=%Y%m%d?shift=-
2d}
> > > > > > > > > > > > > > # -86400}PB2NC_INPUT_DIR =
> > {INPUT_BASE}OUTPUT_BASE =
> > > > > > > > > > > > > >
> > > > > > > >
> > /gpfs/dell2/emc/modeling/noscrub/Edward.StrobachPB2NC_OUTPUT_DIR
> > > > > > > > > =
> > > > > > > > > > > > > >
> > > > > > {OUTPUT_BASE}/AirNow/pb2nc/hourly.{now?fmt=%Y%m%d?shift=-
2d}
> > > > > > > #
> > > > > > > > > > > > > >
-86400}[filename_templates]PB2NC_INPUT_TEMPLATE =
> > > > > > > > > > > > > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ}
(((RELATED TO
> > LOOP
> > > > > > > > > > > > > > CONFUSION)))PB2NC_OUTPUT_TEMPLATE =
> > > > > > > > > > > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ}.nc*
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Let me know if more information is needed.
My
> main
> > > > > > concerns
> > > > > > > > here
> > > > > > > > > > are
> > > > > > > > > > > > 1)
> > > > > > > > > > > > > > being able to process all the list files
> correctly,
> > > 2)
> > > > > > making
> > > > > > > > > sure
> > > > > > > > > > > the
> > > > > > > > > > > > > > looping is correct, and 3) making sure
that I've
> > > > included
> > > > > > > > > > everything
> > > > > > > > > > > > > that I
> > > > > > > > > > > > > > would need to process this step
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > Edward Strobach
> > > > > > > > > > > > > > EMC/NCEP/NWS/
> > > > > > > > > > > > > > IMSG Contractor
> > > > > > > > > > > > > > Cubicle#: 2029
> > > > > > > > > > > > > > 301-683-3717
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > Edward Strobach
> > > > > > > > > > > > > EMC/NCEP/NWS/
> > > > > > > > > > > > > IMSG Contractor
> > > > > > > > > > > > > Cubicle#: 2029
> > > > > > > > > > > > > 301-683-3717
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > > > 303-497-2768
> > > > > > > > > > > > ---
> > > > > > > > > > > > My working day may not be your working day.
Please do
> > not
> > > > > feel
> > > > > > > > > obliged
> > > > > > > > > > to
> > > > > > > > > > > > reply to this email outside of your normal
working
> > hours.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Edward Strobach
> > > > > > > > > > > EMC/NCEP/NWS/
> > > > > > > > > > > IMSG Contractor
> > > > > > > > > > > Cubicle#: 2029
> > > > > > > > > > > 301-683-3717
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > 303-497-2768
> > > > > > > > > > ---
> > > > > > > > > > My working day may not be your working day. Please
do not
> > > feel
> > > > > > > obliged
> > > > > > > > to
> > > > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Edward Strobach
> > > > > > > > > EMC/NCEP/NWS/
> > > > > > > > > IMSG Contractor
> > > > > > > > > Cubicle#: 2029
> > > > > > > > > 301-683-3717
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > George McCabe - Software Engineer III
> > > > > > > > National Center for Atmospheric Research
> > > > > > > > Research Applications Laboratory
> > > > > > > > 303-497-2768
> > > > > > > > ---
> > > > > > > > My working day may not be your working day. Please do
not
> feel
> > > > > obliged
> > > > > > to
> > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Edward Strobach
> > > > > > > EMC/NCEP/NWS/
> > > > > > > IMSG Contractor
> > > > > > > Cubicle#: 2029
> > > > > > > 301-683-3717
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Edward Strobach
> > > > > EMC/NCEP/NWS/
> > > > > IMSG Contractor
> > > > > Cubicle#: 2029
> > > > > 301-683-3717
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: PB2NC_OFFSETS/related questions
From: Edward Strobach - NOAA Affiliate
Time: Thu May 07 20:11:49 2020
sure, that works
On Thu, May 7, 2020 at 7:57 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Edward,
>
> Does 4:15pm EST on Wednesday work for you?
>
> On Thu, May 7, 2020 at 5:38 PM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> >
> > I understand. Are you good some time Tues. or Wed? No meetings
for me
> > those days so any time works
> >
> > On Thu, May 7, 2020 at 7:21 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Edward,
> > >
> > > John and I have had a full day today and I am taking the day off
on
> > Friday.
> > > Let's schedule a time to meet next week. Do any days/time work
better
> for
> > > you?
> > >
> > > On Thu, May 7, 2020 at 10:54 AM Edward Strobach - NOAA Affiliate
via
> RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
>
> > > >
> > > > Removing the JLOGFILE from .bashrc moved me along nicely. I
think
> > > > everything processed just fine.
> > > >
> > > > In terms of setting up a meeting, that would be great.
Sometimes a
> > nudge
> > > > in the right direction is all that is needed. My goals, both
> short-term
> > > and
> > > > long-term, would be run this script, generate plots (I've seen
> > > > plot_data_plane and plot_point_obs) both stats and maps, and
set this
> > up
> > > in
> > > > a batch system so to process it with a cron job so that I have
a time
> > > > series of stats for different regions, and maps with site
overlays.
> > > > Eventually I can expand this to other fields if needed. Right
now I'm
> > not
> > > > exactly sure how to incorporate plotting utilities in this
process,
> > but I
> > > > assume it's just a matter of setting up another config file
with path
> > > > information, processes to be used, and information related to
time.
> I
> > > have
> > > > a meeting in a few minutes but am free otherwise. We could
find a
> time
> > > > that works for all of us, perhaps?
> > > >
> > > > On Thu, May 7, 2020 at 11:56 AM George McCabe via RT <
> > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Edward,
> > > > >
> > > > > Thanks for sending all of this information.
> > > > >
> > > > > I think the fatal error lies in this line in your bashrc:
> > > > >
> > > > > export JLOGFILE=~/
> > > > >
> > > > > I would recommend not setting JLOGFILE at all, as it isn't
> necessary
> > to
> > > > run
> > > > > METplus. I believe it should be set to a filepath to write
job
> logs,
> > so
> > > > > setting this to your home directory is causing the fatal
error. I
> am
> > > > > guessing it only writes to the JLOGFILE when there is an
error, and
> > it
> > > > > can't write to ~/ because it is a directory not a file.
Removing
> that
> > > > from
> > > > > your bashrc should fix the fatal error you are experiencing.
> > > > >
> > > > > It might be beneficial to set up a meeting with you, me, and
John
> > > Halley
> > > > > Gotway (MET engineer) to share screens and step through your
> current
> > > > setup
> > > > > to help you figure out how to set up what you want to do. We
have
> > done
> > > > this
> > > > > with Perry in the past and it proved to be very productive.
Let me
> > know
> > > > if
> > > > > you would like to do that and we can set up a time to meet.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > > On Thu, May 7, 2020 at 7:32 AM Edward Strobach - NOAA
Affiliate via
> > RT
> > > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164 >
> > > > > >
> > > > > > 1) I didn't see a metplus_final.conf file in OUTPUT_BASE,
but
> based
> > > on
> > > > > your
> > > > > > last email, I am guessing you changed the value of
METPLUS_CONF
> to
> > > > create
> > > > > > those files you listed. Which of the pb2nc_pointstat conf
files
> was
> > > > used
> > > > > > for the runs that are producing the error? Could you send
me that
> > > one?
> > > > > When
> > > > > > you run a use case, the name of the final conf is output
to the
> > > screen
> > > > > > towards the beginning of execution, but unfortunately not
added
> to
> > > the
> > > > > > logfile currently.
> > > > > >
> > > > > > They both produce the error. One of them looks for an
ozone file
> > > which
> > > > > > doesn't exist (aqm.t00z.awpozcon.f00.148.grib2) and the
other
> looks
> > > > for a
> > > > > > pm file which doesn't exist (aqm.t00z.pm25.f00.148.grib2).
*f00*
> > are
> > > > not
> > > > > > saved as output from the CMAQ runs while the others (*fXY*
) are
> > > being
> > > > > > saved. I have *.conf files that are created for each
process
> > > > > > (shared.conf_aq; aq.conf; pm.conf; and statanalysis.conf).
These
> > are
> > > > > based
> > > > > > on inputs from my runs script. These are located here:
> > > > > > /gpfs/dell2/ptmp/Edward.Strobach/metplus_aq
> > > > > >
> > > > > > When running the master_metplus.py command, I use several
*.conf
> > > > > > files: *${MET_PLUS}/ush/master_metplus.py
> > > > > > -c ${MET_PLUS}/parm/use_cases/grid_to_obs/grid_to_obs.conf
-c
> > > > > >
> ${MET_PLUS}/parm/use_cases/grid_to_obs/examples/conus_surface.conf
> > -c
> > > > > > ${MET_PLUS_CONF}/pb2nc_aq.conf -c
> > ${MET_PLUS_CONF}/point_stat_aq.conf
> > > > -c
> > > > > > ${MET_PLUS_TMP}/${model}.conf -c
${MET_PLUS_CONF}/shared.conf -c
> > > > > > ${MET_PLUS_TMP}/shared.conf_aq -c
> ${MET_PLUS_CONF}/system_aq.conf*
> > > > > >
> > > > > > Here, model stands for "aq" or "pm"
> > > > > >
> > > > > > Lastly, all the conf files that are stored in the
> OUTPUT_BASE/conf
> > > can
> > > > be
> > > > > > found in the aq directory (two of them) and pm directory
(2 of
> > them).
> > > > > > There is a log file located here:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/logs/aq/20200504
> > > > > > and
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/logs/pm/20200504.
> > > > > > Both of them have this error message in them: 05/06
17:36:43.877
> > > > > > metplus.StatAnalysis (stat_analysis_wrapper.py:237) DEBUG:
Items
> in
> > > > these
> > > > > > lists will be looped over: FCST_VALID_HOUR_LIST
> > > > > >
> > > > > > This, to me, is clearly related to the issue of trying to
find
> that
> > > > file
> > > > > > that doesn't exist. I'm not 100%, but I think the conf
file that
> > you
> > > > want
> > > > > > is this: metplus_final_statanalysis_gatherbydate.conf. I
copied
> > over
> > > > the
> > > > > > contents and stored them in the attachment below. I'm
still
> trying
> > to
> > > > > > understand what exactly is being processed based on the
contents
> > > > inside.
> > > > > >
> > > > > >
> > > > > > 2) It looks like your master_metplus.py command has a lot
of
> extra
> > > conf
> > > > > > files that don't need to be in there. The grid_to_obs conf
files
> > are
> > > > from
> > > > > > an old version of METplus, so I would think they would
cause
> errors
> > > > using
> > > > > > deprecated variables. The corresponding use case in
METplus 3.0
> for
> > > the
> > > > > > upper_air.conf use case is:
> > > > > >
> > > > > >
METplus/parm/use_cases/model_applications/medium_range/Point
> > > > > > Stat_fcstGFS_obsGDAS_UpperAir_MultiField_PrepBufr.conf
> > > > > >
> > > > > > I don't really know, but I would assume that there would
be
> > specific
> > > > > errors
> > > > > > related to deprecated variables that would suggest as
much. I
> > didn't
> > > > see
> > > > > > that though. I could attach the valid config helper,
though, just
> > in
> > > > > case,
> > > > > > right?
> > > > > >
> > > > > > I would start with running that conf with a user config
and if
> that
> > > > > works,
> > > > > > copy the use case conf to another file I and start
modifying that
> > > > instead
> > > > > > of
> > > > > > adding new confs onto the command. That will keep things a
little
> > > more
> > > > > > organized for you.
> > > > > >
> > > > > > I can look into trying that. I'm not very comfortable yet
with
> > > metplus
> > > > to
> > > > > > make serious changes, so it might take a little bit of
time to
> > change
> > > > the
> > > > > > set-up and make a test.
> > > > > >
> > > > > > 3) The error message is very odd. It looks like it was
logging
> just
> > > > fine
> > > > > up
> > > > > > until that error, then it was trying to write to a log
"file"
> named
> > > > > > /u/Edward.Strobach and crashing because that is a
directory. I'm
> > not
> > > > > > understanding why it would change the log file in the
middle of
> > > > execution
> > > > > > and on top of that still write the offending error to the
log. I
> am
> > > > very
> > > > > > baffled by that. I will try to recreate this error
locally.
> > > > > >
> > > > > > Could it be how I have things set in my .bashrc? I have
these
> > > related
> > > > > > settings:
> > > > > > module load python/3.6.3
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > *# Add METplus to $PATHmodule use
> > > > > >
> > > >
> >
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/modulefiles/module
> > > > > > load met/9.0module load NCO/4.7.0module load
> grib_util/1.1.1export
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
PATH=${PATH}:/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/METplus/METplus-
3.0/ushexport
> > > > > > PATH=~/METplus/ush:$PATH# Optional JLOGFILEexport
> JLOGFILE=~/module
> > > > load
> > > > > > prod_util/1.1.3*
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, May 6, 2020 at 6:46 PM George McCabe via RT <
> > > met_help at ucar.edu
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Edward,
> > > > > > >
> > > > > > > Julie was able to login and send me some of your log
files and
> > info
> > > > > > while I
> > > > > > > was in a meeting.
> > > > > > >
> > > > > > > 1) I didn't see a metplus_final.conf file in
OUTPUT_BASE, but
> > based
> > > > on
> > > > > > your
> > > > > > > last email, I am guessing you changed the value of
METPLUS_CONF
> > to
> > > > > create
> > > > > > > those files you listed. Which of the pb2nc_pointstat
conf files
> > was
> > > > > used
> > > > > > > for the runs that are producing the error? Could you
send me
> that
> > > > one?
> > > > > > When
> > > > > > > you run a use case, the name of the final conf is output
to the
> > > > screen
> > > > > > > towards the beginning of execution, but unfortunately
not added
> > to
> > > > the
> > > > > > > logfile currently.
> > > > > > >
> > > > > > >
> > > > > > > 2) It looks like your master_metplus.py command has a
lot of
> > extra
> > > > conf
> > > > > > > files that don't need to be in there. The grid_to_obs
conf
> files
> > > are
> > > > > from
> > > > > > > an old version of METplus, so I would think they would
cause
> > errors
> > > > > using
> > > > > > > deprecated variables. The corresponding use case in
METplus 3.0
> > for
> > > > the
> > > > > > > upper_air.conf use case is:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
METplus/parm/use_cases/model_applications/medium_range/PointStat_fcstGFS_obsGDAS_UpperAir_MultiField_PrepBufr.conf
> > > > > > >
> > > > > > > I would start with running that conf with a user config
and if
> > that
> > > > > > works,
> > > > > > > copy the use case conf to another file and start
modifying that
> > > > instead
> > > > > > of
> > > > > > > adding new confs onto the command. That will keep things
a
> little
> > > > more
> > > > > > > organized for you.
> > > > > > >
> > > > > > >
> > > > > > > 3) The error message is very odd. It looks like it was
logging
> > just
> > > > > fine
> > > > > > up
> > > > > > > until that error, then it was trying to write to a log
"file"
> > named
> > > > > > > /u/Edward.Strobach and crashing because that is a
directory.
> I'm
> > > not
> > > > > > > understanding why it would change the log file in the
middle of
> > > > > execution
> > > > > > > and on top of that still write the offending error to
the log.
> I
> > am
> > > > > very
> > > > > > > baffled by that. I will try to recreate this error
locally.
> > > > > > >
> > > > > > > On Wed, May 6, 2020 at 1:43 PM Edward Strobach - NOAA
Affiliate
> > via
> > > > RT
> > > > > <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
> > >
> > > > > > > >
> > > > > > > > So my OUTPUT_BASE is here:
> > > > > > > >
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq.
> > > > > > > >
> > > > > > > > There is a conf directory where .conf files are stored
in aq
> > > > > > > > (OUTPUT_BASE/conf/aq) and pm (OUTPUT_BASE/conf/pm).
There
> are
> > > some
> > > > > > .conf
> > > > > > > > files from these runs.
> > > > > > > >
> > > > > > > > aq:
> > > > > > > > metplus_final_pb2nc_pointstat.conf
> > > > > > > > metplus_final_statanalysis_gatherbydate.conf
> > > > > > > >
> > > > > > > > pm:
> > > > > > > > metplus_final_pb2nc_pointstat.conf
> > > > > > > > metplus_final_statanalysis_gatherbydate.conf
> > > > > > > >
> > > > > > > > Any file in particular that you want to see? I don't
have
> the
> > > file
> > > > > you
> > > > > > > are
> > > > > > > > referring to.
> > > > > > > >
> > > > > > > > On Wed, May 6, 2020 at 3:22 PM George McCabe via RT <
> > > > > met_help at ucar.edu
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > That's it, thanks.
> > > > > > > > >
> > > > > > > > > Hmm, it looks like there is an error trying to add
to the
> log
> > > > file?
> > > > > > Did
> > > > > > > > > this run generate a log file? It would be found in
the logs
> > > > > directory
> > > > > > > > under
> > > > > > > > > the value you set for OUTPUT_BASE. If there is one,
can you
> > > send
> > > > > that
> > > > > > > to
> > > > > > > > > me? Also, please send the metplus_final.conf file in
your
> > > > > OUTPUT_BASE
> > > > > > > as
> > > > > > > > > well.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, May 6, 2020 at 1:03 PM Edward Strobach -
NOAA
> > Affiliate
> > > > via
> > > > > > RT
> > > > > > > <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
> > > > >
> > > > > > > > > >
> > > > > > > > > > Sure thing. I think you are referring to this?
> > > > > > > > > >
> > > > > > > > > > 05/06 17:34:09.831 metplus.PointStat
> > (command_builder.py:100)
> > > > > > ERROR:
> > > > > > > > > > (command_builder.py:437) Could not find FCST file
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
> > > > > > > > > > using template
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
> > > > > > > > > > 05/06 17:34:09.832 metplus.PointStat
(met_util.py:192)
> > ERROR:
> > > > > Fatal
> > > > > > > > error
> > > > > > > > > > occurred
> > > > > > > > > > *Traceback (most recent call last):*
> > > > > > > > > > File
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > > > > > > > line 169, in run_metplus
> > > > > > > > > > process.run_all_times()
> > > > > > > > > > File
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > > > > > line 751, in run_all_times
> > > > > > > > > > util.loop_over_times_and_call(self.config,
self)
> > > > > > > > > > File
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > > > > > > > line 834, in loop_over_times_and_call
> > > > > > > > > > process.run_at_time(input_dict)
> > > > > > > > > > File
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/compare_gridded_wrapper.py",
> > > > > > > > > > line 183, in run_at_time
> > > > > > > > > > self.run_at_time_once(time_info)
> > > > > > > > > > File
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/point_stat_wrapper.py",
> > > > > > > > > > line 146, in run_at_time_once
> > > > > > > > > > model_path = self.find_model(time_info,
var_list[0])
> > > > > > > > > > File
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > > > > > line 319, in find_model
> > > > > > > > > > return_list=return_list)
> > > > > > > > > > File
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > > > > > line 373, in find_data
> > > > > > > > > > return self.find_exact_file(**arg_dict)
> > > > > > > > > > File
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > > > > > line 437, in find_exact_file
> > > > > > > > > > self.log_error(msg)
> > > > > > > > > > File
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/command_builder.py",
> > > > > > > > > > line 100, in log_error
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >
self.logger.error(f"({os.path.basename(caller.filename)}:{caller.lineno})
> > > > > > > > > > {error_string}")
> > > > > > > > > > File
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > > > > > line 1335, in error
> > > > > > > > > > self._log(ERROR, msg, args, **kwargs)
> > > > > > > > > > File
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > > > > > line 1442, in _log
> > > > > > > > > > self.handle(record)
> > > > > > > > > > File
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > > > > > line 1452, in handle
> > > > > > > > > > self.callHandlers(record)
> > > > > > > > > > File
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > > > > > line 1514, in callHandlers
> > > > > > > > > > hdlr.handle(record)
> > > > > > > > > > File
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/usrx/local/prod/packages/python/3.6.3/lib/python3.6/logging/__init__.py",
> > > > > > > > > > line 863, in handle
> > > > > > > > > > self.emit(record)
> > > > > > > > > > File
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/produtil/log.py",
> > > > > > > > > > line 227, in emit
> > > > > > > > > > with open(self._logger,'at') as f:
> > > > > > > > > >
> > > > > > > > > > On Wed, May 6, 2020 at 2:55 PM George McCabe via
RT <
> > > > > > > met_help at ucar.edu
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Edward,
> > > > > > > > > > >
> > > > > > > > > > > When you cloned the repo yourself, are you using
the
> > > develop
> > > > > > branch
> > > > > > > > to
> > > > > > > > > > run?
> > > > > > > > > > > There may have been a bug introduced that is
causing
> that
> > > > fatal
> > > > > > > > error.
> > > > > > > > > > > Could you send me the stack trace that comes
after the
> > > fatal
> > > > > > > error? I
> > > > > > > > > > agree
> > > > > > > > > > > that it shouldn't crash if it can't find a file.
> > > > > > > > > > >
> > > > > > > > > > > On Wed, May 6, 2020 at 12:43 PM Edward Strobach
- NOAA
> > > > > Affiliate
> > > > > > > via
> > > > > > > > > RT <
> > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
> > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks George. I've been busy trying to
figure
> things
> > > out
> > > > > and
> > > > > > > > forgot
> > > > > > > > > > to
> > > > > > > > > > > > respond. I talked with Perry about some of
these
> > things
> > > > and
> > > > > he
> > > > > > > was
> > > > > > > > > > > > surprised that I was receiving this message
when
> > running
> > > > his
> > > > > > > > script:
> > > > > > > > > > > >
> > > > > > > > > > > > 05/06 17:34:09.517 metplus.PointStat INFO:
Processing
> > > > > forecast
> > > > > > > > lead 0
> > > > > > > > > > > hours
> > > > > > > > > > > > 05/06 17:34:09.525 metplus.PointStat
> > > > (command_builder.py:407)
> > > > > > > > DEBUG:
> > > > > > > > > > > > Looking for FCST file
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/aqm.t00z.awpozcon.f00.148.grib2
> > > > > > > > > > > > 05/06 17:34:09.831 metplus.PointStat
> > > > (command_builder.py:100)
> > > > > > > > ERROR:
> > > > > > > > > > > > (command_builder.py:437) Could not find FCST
file
> > > > > > > > > > > > /gpfs/hps/nco/ops/com/aqm/prod/aqm.20200504/
> > > > > > > > > > > > *aqm.t00z.awpozcon.f00.148.grib2* using
template
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
aqm.{init?fmt=%Y%m%d}/aqm.t{init?fmt=%2H}z.awpozcon.f{lead?fmt=%2H}.148.grib2
> > > > > > > > > > > > 05/06 17:34:09.832 metplus.PointStat
> (met_util.py:192)
> > > > ERROR:
> > > > > > > Fatal
> > > > > > > > > > error
> > > > > > > > > > > > occurred
> > > > > > > > > > > >
> > > > > > > > > > > > The bolded item does not exist. When Perry
runs his
> > > script
> > > > > it
> > > > > > > > skips
> > > > > > > > > > this
> > > > > > > > > > > > file. When I run the script I get a fatal
error. I
> > > cloned
> > > > > > > metplus
> > > > > > > > > > just
> > > > > > > > > > > > the other day. I'm not sure if this is
related to
> > item 2
> > > > > from
> > > > > > > your
> > > > > > > > > > email
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, May 5, 2020 at 2:37 PM George McCabe
via RT <
> > > > > > > > > met_help at ucar.edu
> > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Edward,
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1) The LOG_MET_VERBOSITY variable sets the
> verbosity
> > > > level
> > > > > > > when a
> > > > > > > > > MET
> > > > > > > > > > > > tool
> > > > > > > > > > > > > is run. To change the METplus logging level,
set
> > > > > LOG_LEVEL. I
> > > > > > > > would
> > > > > > > > > > > > > recommend changing it to DEBUG to get more
> > information.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2) There is a bug in using the {now} tags
for
> > > > > > > > > [INIT/VALID]_[BEG/END]
> > > > > > > > > > > that
> > > > > > > > > > > > > causes that crash. This has been fixed but
we have
> > not
> > > > yet
> > > > > > > > > released a
> > > > > > > > > > > > > bugfix release. I will let the engineers
know that
> > you
> > > > need
> > > > > > > this
> > > > > > > > > > > > > functionality and we will push the METplus
3.0.1
> > bugfix
> > > > > > release
> > > > > > > > > soon.
> > > > > > > > > > > You
> > > > > > > > > > > > > should be able to set the time explicitly,
i.e.
> > > > 2020050500.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 3) The {now} tag can be used in the
VALID_BEG and
> > > > VALID_END
> > > > > > > > config
> > > > > > > > > > > > > variables, but you will want to refer to
init,
> valid,
> > > > lead,
> > > > > > > etc.
> > > > > > > > in
> > > > > > > > > > > your
> > > > > > > > > > > > > INPUT_TEMPLATE and OUTPUT_TEMPLATE variables
so
> that
> > it
> > > > > uses
> > > > > > > the
> > > > > > > > > time
> > > > > > > > > > > for
> > > > > > > > > > > > > each time loop. See this section for more
> > information:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#directory-
and-filename-template-info
> > > > > > > > > > > > >
> > > > > > > > > > > > > 4) The LEAD_SEQ is a list of forecast leads
that
> will
> > > be
> > > > > > > > processed
> > > > > > > > > > for
> > > > > > > > > > > > each
> > > > > > > > > > > > > run time. If you set this to 0 or unset it,
then it
> > > will
> > > > > just
> > > > > > > > > process
> > > > > > > > > > > > once
> > > > > > > > > > > > > for each run time.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 5) There is documentation describing the
time
> looping
> > > > > > variables
> > > > > > > > > here:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://ncar.github.io/METplus/Users_Guide/systemconfiguration.html#common-
config-variables
> > > > > > > > > > > > >
> > > > > > > > > > > > > This page should answer some of your
questions on
> how
> > > > this
> > > > > > > works.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 6) I would recommend starting with the
Example use
> > case
> > > > > found
> > > > > > > in
> > > > > > > > > > > > >
> parm/use_cases/met_tool_wrapper/Example/Example.conf.
> > > > This
> > > > > > use
> > > > > > > > case
> > > > > > > > > > > loops
> > > > > > > > > > > > > over times, fills in the filename templates
based
> on
> > > each
> > > > > run
> > > > > > > > time,
> > > > > > > > > > and
> > > > > > > > > > > > > prints information to the logfile. You can
copy
> this
> > > use
> > > > > case
> > > > > > > and
> > > > > > > > > > > modify
> > > > > > > > > > > > it
> > > > > > > > > > > > > to see which filenames are searched for each
run
> > time.
> > > > This
> > > > > > is
> > > > > > > a
> > > > > > > > > good
> > > > > > > > > > > way
> > > > > > > > > > > > > to understand how to configure METplus to
find the
> > > files
> > > > > you
> > > > > > > > need.
> > > > > > > > > > Once
> > > > > > > > > > > > > this makes sense, you can look at the
PB2NC.conf
> use
> > > case
> > > > > to
> > > > > > > see
> > > > > > > > > how
> > > > > > > > > > > the
> > > > > > > > > > > > > config variables control that wrapper.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 7) METplus sets environment variables based
on its
> > > > > > > configuration
> > > > > > > > > > files.
> > > > > > > > > > > > > These environment variables are referenced
in the
> MET
> > > > > config
> > > > > > > file
> > > > > > > > > to
> > > > > > > > > > > set
> > > > > > > > > > > > > configurations in that file. If you turn on
DEBUG
> > > > logging,
> > > > > > you
> > > > > > > > can
> > > > > > > > > > see
> > > > > > > > > > > > what
> > > > > > > > > > > > > values are set for these environment
variables to
> > get a
> > > > > > better
> > > > > > > > idea
> > > > > > > > > > of
> > > > > > > > > > > > how
> > > > > > > > > > > > > your METplus configuration will affect the
MET
> > > > > configuration.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Please let me know if you have any questions
> > regarding
> > > > any
> > > > > of
> > > > > > > > this.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, May 5, 2020 at 11:46 AM Edward
Strobach -
> > NOAA
> > > > > > > Affiliate
> > > > > > > > > via
> > > > > > > > > > > RT <
> > > > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95164
> > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Update:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Trying to follow the log files that are
generated
> > for
> > > > > > > guidance.
> > > > > > > > > I
> > > > > > > > > > > set
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > verbosity to 7, but still can't understand
why
> I'm
> > > > > > receiving
> > > > > > > > this
> > > > > > > > > > > > > message:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 05/05 17:41:59.861 metplus INFO: Running
METplus
> > v3.0
> > > > > > called
> > > > > > > > with
> > > > > > > > > > > > > command:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/master_metplus.py
> > > > > > > > > > > > > > -c
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/use_cases/model_applications/air_quality/CMAQ_obsAirnow.conf
> > > > > > > > > > > > > > 05/05 17:41:59.882 metplus
(met_util.py:776)
> ERROR:
> > > > > > > > > > > > [INIT/VALID]_TIME_FMT
> > > > > > > > > > > > > > (%Y%m%d%H) does not match
[INIT/VALID]_[BEG/END]
> > > > > > (2020050200
> > > > > > > > > > > > > #2020050317)
> > > > > > > > > > > > > > 05/05 17:41:59.886 metplus.PB2NC
> (met_util.py:192)
> > > > ERROR:
> > > > > > > Fatal
> > > > > > > > > > error
> > > > > > > > > > > > > > occurred
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I tried to start basic. If starting basic
> worked,
> > > > then I
> > > > > > > would
> > > > > > > > > > have
> > > > > > > > > > > > made
> > > > > > > > > > > > > > it more complex:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > # time looping - options are INIT, VALID,
RETRO,
> > and
> > > > > > REALTIME
> > > > > > > > > > > > > > LOG_MET_VERBOSITY = 7
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > LOOP_BY = INIT
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > # Format of VALID_BEG and VALID_END
> > > > > > > > > > > > > > INIT_TIME_FMT = %Y%m%d%H
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > # Start time, end time, and time increment
for
> > > METplus
> > > > > run
> > > > > > > > > > > > > > INIT_BEG = 2020050200
> > > > #{now?fmt=%Y%m%d%H?shift=-172800}
> > > > > > > > > > > > > > INIT_END = 2020050200
> > > #{now?fmt=%Y%m%d%H?shift=-172800}
> > > > > > > > > #-21600}
> > > > > > > > > > > > > > INIT_INCREMENT = 0
> > > > > > > > > > > > > > LEAD_SEQ = 0
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > With how this looks, I would have thought
that
> only
> > > one
> > > > > > file
> > > > > > > > > would
> > > > > > > > > > be
> > > > > > > > > > > > > > processed. But it complains about a
mismatch
> > between
> > > > the
> > > > > > > > > > > INIT_TIME_FMT
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > INIT_BEG/INIT_END
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, May 5, 2020 at 12:01 PM
> met_help at ucar.edu
> > > via
> > > > > RT <
> > > > > > > > > > > > > > met_help at ucar.edu>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Greetings,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This message has been automatically
generated
> in
> > > > > response
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > > > > creation
> > > > > > > > > > > > > > > of a trouble ticket regarding:
> > > > > > > > > > > > > > > "PB2NC_OFFSETS/related
questions",
> > > > > > > > > > > > > > > a summary of which appears below.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > There is no need to reply to this
message right
> > > now.
> > > > > > Your
> > > > > > > > > ticket
> > > > > > > > > > > has
> > > > > > > > > > > > > > been
> > > > > > > > > > > > > > > assigned an ID of [rt.rap.ucar.edu
#95164].
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Please include the string:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > [rt.rap.ucar.edu #95164]
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > in the subject line of all future
> correspondence
> > > > about
> > > > > > this
> > > > > > > > > > issue.
> > > > > > > > > > > To
> > > > > > > > > > > > > do
> > > > > > > > > > > > > > > so, you may reply to this message.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > For more information, please see:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > MET Online Tutorial:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > >
> > > > >
> https://www.dtcenter.org/met/users/support/online_tutorial/index.php
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > MET Users Guide:
> > > > > > > > > > > > > > >
> > > > > https://www.dtcenter.org/met/users/docs/overview.php
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > MET FAQs:
> > > > > > > > > > > > > > >
> > > > > > >
https://www.dtcenter.org/met/users/support/faqs/index.php
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > MET-Help Email Archive:
> > > > > > > > > > > > > > >
http://mailman.ucar.edu/pipermail/met_help
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thank you,
> > > > > > > > > > > > > > >
met_help at ucar.edu
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
>
-------------------------------------------------------------------------
> > > > > > > > > > > > > > > Good morning,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I'm trying to configure a file to
process
> > prepbufr
> > > > data
> > > > > > to
> > > > > > > > > netcdf
> > > > > > > > > > > > > format.
> > > > > > > > > > > > > > > There are several files that are within
the
> > > directory
> > > > > > that
> > > > > > > I
> > > > > > > > > want
> > > > > > > > > > > to
> > > > > > > > > > > > > > > process as shown by the bolded text from
the
> > > > > > > > > > > > > > >
> > /gpfs/dell1/nco/ops/com/hourly/prod/hourly.20200503
> > > > > > > > directory:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > aqm.t12z.airnow.tm00.bufr_d
> > > > aqm.t12z.anowpm.pb.tm120
> > > > > > > > > > > > > > > aqm.t12z.status.tm024.bufr_d
> > > > > > > > > > > > > > > aqm.t12z.airnow.tm024.bufr_d
> > > > *aqm.t12z.prepbufr.tm00 *
> > > > > > > > > > > > > > > aqm.t12z.status.tm120.bufr_d
> > > > > > > > > > > > > > > aqm.t12z.airnow.tm120.bufr_d
> > > > *aqm.t12z.prepbufr.tm024*
> > > > > > > > > > > > > > > obsproc_version_for_t12z_aqm_dump_run
> > > > > > > > > > > > > > > *aqm.t12z.anowpm.pb.tm00*
> > > > aqm.t12z.prepbufr.tm120
> > > > > > > > > > > > > > > where_t12z_aqm_dump_ran
> > > > > > > > > > > > > > > *aqm.t12z.anowpm.pb.tm024 *
> > > > > > aqm.t12z.status.tm00.bufr_d
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Right now I'm focusing only on the
*anowpm*
> files
> > > > (two
> > > > > > > > files).
> > > > > > > > > > The
> > > > > > > > > > > > > tm00
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > time centered on 12z for 05/03/2020,
with DHR =
> > > > ranging
> > > > > > > from
> > > > > > > > > -11
> > > > > > > > > > to
> > > > > > > > > > > > 12,
> > > > > > > > > > > > > > > covering the whole day. Similarly, the
tm024
> > file
> > > is
> > > > > > > > centered
> > > > > > > > > at
> > > > > > > > > > > > 12z,
> > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > for 05/02/2020. I looked through the
metplus
> > > > glossary
> > > > > > and
> > > > > > > > > > examples
> > > > > > > > > > > > to
> > > > > > > > > > > > > > help
> > > > > > > > > > > > > > > with the process of building the
configure
> file.
> > > If
> > > > I
> > > > > > run
> > > > > > > > this
> > > > > > > > > > > > script
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > desire the files from 05/03/2020, then I
should
> > set
> > > > the
> > > > > > > > > > following:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *[config]# time looping - options are
INIT,
> > VALID,
> > > > > RETRO,
> > > > > > > and
> > > > > > > > > > > > > > > REALTIMELOOP_BY = VALID# Format of
VALID_BEG
> and
> > > > > > > > > > > > > VALID_ENDVALID_TIME_FMT
> > > > > > > > > > > > > > =
> > > > > > > > > > > > > > > %Y%m%d%H# Start time, end time, and time
> > increment
> > > > for
> > > > > > > > METplus
> > > > > > > > > > > > > > runVALID_BEG
> > > > > > > > > > > > > > > = {now?fmt=%Y%m%d%H?shift=-2d}VALID_END
=
> > > > > > > > > > > > {now?fmt=%Y%m%d%H?shift=-2d}
> > > > > > > > > > > > > > > #-21600}VALID_INCREMENT = 24HLEAD_SEQ =
0,24
> > > > (((NOT
> > > > > > > SURE I
> > > > > > > > > > NEED
> > > > > > > > > > > > > THIS,
> > > > > > > > > > > > > > > BUT NOT SURE HOW THE LOOP WILL WORK
WITHOUT
> > IT?)))*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > *LOOP_ORDER = times# List of
applications to
> > > > > > > runPROCESS_LIST
> > > > > > > > =
> > > > > > > > > > > > > > > PB2NCCONFIG_DIR =
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/parm/met_configCONFIG_FILE
> > > > > > > > > > > > > > > =
> > > > > > {CONFIG_DIR}/PB2NCConfig_wrappedPB2NC_OBS_BUFR_VAR_LIST =
> > > > > > > > > DHR,
> > > > > > > > > > > RPT,
> > > > > > > > > > > > > > SID,
> > > > > > > > > > > > > > > YOB, XOB, TYPO, TPHR, QCIND, COPO,
COPOPM*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I've listed the variables I would like
to
> extract
> > > > from
> > > > > > each
> > > > > > > > > file
> > > > > > > > > > as
> > > > > > > > > > > > > well.
> > > > > > > > > > > > > > > It's not clear if I need the config
file, but I
> > > > listed
> > > > > it
> > > > > > > > just
> > > > > > > > > in
> > > > > > > > > > > > case.
> > > > > > > > > > > > > > > Here's what I have for the file names
from the
> > > input
> > > > > > > > directory
> > > > > > > > > > and
> > > > > > > > > > > > > what I
> > > > > > > > > > > > > > > would name it once converted to netcdf
and
> placed
> > > in
> > > > > the
> > > > > > > > output
> > > > > > > > > > > > > > directory:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
*INPUT_BASE=/gpfs/dell1/nco/ops/com/hourly/prod/hourly.{now?fmt=%Y%m%d?shift=-
2d}
> > > > > > > > > > > > > > > # -86400}PB2NC_INPUT_DIR =
> > > {INPUT_BASE}OUTPUT_BASE =
> > > > > > > > > > > > > > >
> > > > > > > > >
> > > /gpfs/dell2/emc/modeling/noscrub/Edward.StrobachPB2NC_OUTPUT_DIR
> > > > > > > > > > =
> > > > > > > > > > > > > > >
> > > > > > >
{OUTPUT_BASE}/AirNow/pb2nc/hourly.{now?fmt=%Y%m%d?shift=-2d}
> > > > > > > > #
> > > > > > > > > > > > > > >
> -86400}[filename_templates]PB2NC_INPUT_TEMPLATE =
> > > > > > > > > > > > > > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ}
(((RELATED
> TO
> > > LOOP
> > > > > > > > > > > > > > > CONFUSION)))PB2NC_OUTPUT_TEMPLATE =
> > > > > > > > > > > > > aqm.t12z.anowpm.pb.tm0{LEAD_SEQ}.nc*
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Let me know if more information is
needed. My
> > main
> > > > > > > concerns
> > > > > > > > > here
> > > > > > > > > > > are
> > > > > > > > > > > > > 1)
> > > > > > > > > > > > > > > being able to process all the list files
> > correctly,
> > > > 2)
> > > > > > > making
> > > > > > > > > > sure
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > looping is correct, and 3) making sure
that
> I've
> > > > > included
> > > > > > > > > > > everything
> > > > > > > > > > > > > > that I
> > > > > > > > > > > > > > > would need to process this step
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > Edward Strobach
> > > > > > > > > > > > > > > EMC/NCEP/NWS/
> > > > > > > > > > > > > > > IMSG Contractor
> > > > > > > > > > > > > > > Cubicle#: 2029
> > > > > > > > > > > > > > > 301-683-3717
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > Edward Strobach
> > > > > > > > > > > > > > EMC/NCEP/NWS/
> > > > > > > > > > > > > > IMSG Contractor
> > > > > > > > > > > > > > Cubicle#: 2029
> > > > > > > > > > > > > > 301-683-3717
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > > > > 303-497-2768
> > > > > > > > > > > > > ---
> > > > > > > > > > > > > My working day may not be your working day.
Please
> do
> > > not
> > > > > > feel
> > > > > > > > > > obliged
> > > > > > > > > > > to
> > > > > > > > > > > > > reply to this email outside of your normal
working
> > > hours.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Edward Strobach
> > > > > > > > > > > > EMC/NCEP/NWS/
> > > > > > > > > > > > IMSG Contractor
> > > > > > > > > > > > Cubicle#: 2029
> > > > > > > > > > > > 301-683-3717
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > > > Research Applications Laboratory
> > > > > > > > > > > 303-497-2768
> > > > > > > > > > > ---
> > > > > > > > > > > My working day may not be your working day.
Please do
> not
> > > > feel
> > > > > > > > obliged
> > > > > > > > > to
> > > > > > > > > > > reply to this email outside of your normal
working
> hours.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Edward Strobach
> > > > > > > > > > EMC/NCEP/NWS/
> > > > > > > > > > IMSG Contractor
> > > > > > > > > > Cubicle#: 2029
> > > > > > > > > > 301-683-3717
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > George McCabe - Software Engineer III
> > > > > > > > > National Center for Atmospheric Research
> > > > > > > > > Research Applications Laboratory
> > > > > > > > > 303-497-2768
> > > > > > > > > ---
> > > > > > > > > My working day may not be your working day. Please
do not
> > feel
> > > > > > obliged
> > > > > > > to
> > > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Edward Strobach
> > > > > > > > EMC/NCEP/NWS/
> > > > > > > > IMSG Contractor
> > > > > > > > Cubicle#: 2029
> > > > > > > > 301-683-3717
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > George McCabe - Software Engineer III
> > > > > > > National Center for Atmospheric Research
> > > > > > > Research Applications Laboratory
> > > > > > > 303-497-2768
> > > > > > > ---
> > > > > > > My working day may not be your working day. Please do
not feel
> > > > obliged
> > > > > to
> > > > > > > reply to this email outside of your normal working
hours.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Edward Strobach
> > > > > > EMC/NCEP/NWS/
> > > > > > IMSG Contractor
> > > > > > Cubicle#: 2029
> > > > > > 301-683-3717
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > > --
> > > > Edward Strobach
> > > > EMC/NCEP/NWS/
> > > > IMSG Contractor
> > > > Cubicle#: 2029
> > > > 301-683-3717
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
More information about the Met_help
mailing list