[Met_help] [rt.rap.ucar.edu #95996] History for StatAnalysis works intermittently with CMAQ meteorology files

Thu Oct 1 14:30:00 MDT 2020

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Good morning,

I've been attempting to set up verification for meteorology from the CMAQ
output.  In the past, I was able to successfully generate files for the
PB2NC, PointStat, and StatAnalysis steps.  In confirming the success of
these steps, I then set up my configuration to handle model runs with files
that follow the same naming structure.  The PB2NC and PointStat files are
always successful, while the StatAnalysis works intermittently.  Yesterday,
I was able to generate stat files from StatAnalysis for 2 out of my 4 model
runs.  Today, however, only one model run ended up being successful.  I'm
not sure why it would work for one day, then the next day it doesn't work.
The error is always the same.

07/23 00:32:16.941 metplus (config_launcher.py:520) DEBUG: Setting [config]
MODEL1_REFERENCE_NAME to default value: PROD.
07/23 00:32:16.942 metplus.StatAnalysis (stat_analysis_wrapper.py:235)
DEBUG: Items in these lists will be grouped together: MODEL_LIST,
DESC_LIST, FCST_LEAD_LIST, OBS_LEAD_LIST, FCST_INIT_HOUR_LIST,
OBS_VALID_HOUR_LIST, OBS_INIT_HOUR_LIST, FCST_VAR_LIST, OBS_VAR_LIST,
FCST_UNITS_LIST, OBS_UNITS_LIST, FCST_LEVEL_LIST, OBS_LEVEL_LIST,
VX_MASK_LIST, INTERP_MTHD_LIST, INTERP_PNTS_LIST, FCST_THRESH_LIST,
OBS_THRESH_LIST, COV_THRESH_LIST, ALPHA_LIST, LINE_TYPE_LIST
07/23 00:32:16.942 metplus.StatAnalysis (stat_analysis_wrapper.py:237)
DEBUG: Items in these lists will be looped over: FCST_VALID_HOUR_LIST
07/23 00:32:16.957 metplus.StatAnalysis (met_util.py:192) ERROR: Fatal
error occurred
Traceback (most recent call last):
  File
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
line 172, in run_metplus
    loop_over_times_and_call(config, processes)
  File
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
line 834, in loop_over_times_and_call
    process.run_at_time(input_dict)
  File
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
line 1872, in run_at_time
    self.run_stat_analysis_job(date, date, loop_by)
  File
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
line 1288, in run_stat_analysis_job
    runtime_settings_dict)
  File
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
line 836, in get_lookin_dir
    subprocess.check_output('ls -d '+dir_path_filled, shell=True)
  File "/usrx/local/dev/packages/python/3.6.3/lib/python3.6/subprocess.py",
line 336, in check_output
    **kwargs).stdout
  File "/usrx/local/dev/packages/python/3.6.3/lib/python3.6/subprocess.py",
line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'ls -d
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*'
returned non-zero exit status 2.

If the step worked, then the line that would follow "....looped over:
FCST_VALID_HOUR_LIST"  would be
07/23 14:55:12.158 metplus.StatAnalysis (stat_analysis_wrapper.py:788)
DEBUG: Building dump_row filename from user template:
{valid?fmt=%Y%m%d}/PARA13_BC_{valid_beg?fmt=%Y%m%d%H}.stat

If this step fails, it always happens at the first valid time (in this case
2020072000).  Following the file name template, I would have something like
this:

-rw-r--r-- 1 Edward.Strobach emcmodel 525632 Jul 23 14:55
PARA13_BC_2020072000.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 344951 Jul 23 14:55
PARA13_BC_2020072001.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 345726 Jul 23 14:55
PARA13_BC_2020072002.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 343403 Jul 23 14:55
PARA13_BC_2020072003.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 346222 Jul 23 14:55
PARA13_BC_2020072004.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 342297 Jul 23 14:55
PARA13_BC_2020072005.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 416853 Jul 23 14:55
PARA13_BC_2020072006.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 343406 Jul 23 14:55
PARA13_BC_2020072007.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 346832 Jul 23 14:55
PARA13_BC_2020072008.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 342291 Jul 23 14:55
PARA13_BC_2020072009.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 347004 Jul 23 14:55
PARA13_BC_2020072010.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 358938 Jul 23 14:55
PARA13_BC_2020072011.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 519089 Jul 23 14:55
PARA13_BC_2020072012.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 414343 Jul 23 14:55
PARA13_BC_2020072013.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 437188 Jul 23 14:55
PARA13_BC_2020072014.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 435365 Jul 23 14:56
PARA13_BC_2020072015.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 433295 Jul 23 14:56
PARA13_BC_2020072016.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 433698 Jul 23 14:56
PARA13_BC_2020072017.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 508369 Jul 23 14:56
PARA13_BC_2020072018.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 447083 Jul 23 14:56
PARA13_BC_2020072019.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 441350 Jul 23 14:56
PARA13_BC_2020072020.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 437671 Jul 23 14:56
PARA13_BC_2020072021.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 416085 Jul 23 14:56
PARA13_BC_2020072022.stat
-rw-r--r-- 1 Edward.Strobach emcmodel 444785 Jul 23 14:56
PARA13_BC_2020072023.stat

You can see these are indeed populated, and that these are the only ones
generated.  If all files were generated, then I would have PROD_BC_*stat,
PROD_*stat, and PARA13_*stat.  The day before both PROD and PARA13_BC
worked.  I've also had cases where all model runs were successful.  I'm not
sure why this would be the case.  I don't see anything indicating a problem
within the point stat files themselves..
-- 
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: StatAnalysis works intermittently with CMAQ meteorology files
From: Minna Win
Time: Thu Jul 23 09:27:36 2020

Hello Edward,

It looks like you have some issues with your StatAnalysis runs.  I've
assigned this issue to John Halley Gotway.  Please allow a few
business
days for a full response.

Regards,
Minna
---------------
Minna Win
National Center for Atmospheric Research
Developmental Testbed Center
Phone: 303-497-8423
Fax:   303-497-8401

On Thu, Jul 23, 2020 at 9:18 AM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:

>
> Thu Jul 23 09:18:05 2020: Request 95996 was acted upon.
> Transaction: Ticket created by edward.strobach at noaa.gov
>        Queue: met_help
>      Subject: StatAnalysis works intermittently with CMAQ
meteorology files
>        Owner: Nobody
>   Requestors: edward.strobach at noaa.gov
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996 >
>
>
> Good morning,
>
> I've been attempting to set up verification for meteorology from the
CMAQ
> output.  In the past, I was able to successfully generate files for
the
> PB2NC, PointStat, and StatAnalysis steps.  In confirming the success
of
> these steps, I then set up my configuration to handle model runs
with files
> that follow the same naming structure.  The PB2NC and PointStat
files are
> always successful, while the StatAnalysis works intermittently.
Yesterday,
> I was able to generate stat files from StatAnalysis for 2 out of my
4 model
> runs.  Today, however, only one model run ended up being successful.
I'm
> not sure why it would work for one day, then the next day it doesn't
work.
> The error is always the same.
>
> 07/23 00:32:16.941 metplus (config_launcher.py:520) DEBUG: Setting
[config]
> MODEL1_REFERENCE_NAME to default value: PROD.
> 07/23 00:32:16.942 metplus.StatAnalysis
(stat_analysis_wrapper.py:235)
> DEBUG: Items in these lists will be grouped together: MODEL_LIST,
> DESC_LIST, FCST_LEAD_LIST, OBS_LEAD_LIST, FCST_INIT_HOUR_LIST,
> OBS_VALID_HOUR_LIST, OBS_INIT_HOUR_LIST, FCST_VAR_LIST,
OBS_VAR_LIST,
> FCST_UNITS_LIST, OBS_UNITS_LIST, FCST_LEVEL_LIST, OBS_LEVEL_LIST,
> VX_MASK_LIST, INTERP_MTHD_LIST, INTERP_PNTS_LIST, FCST_THRESH_LIST,
> OBS_THRESH_LIST, COV_THRESH_LIST, ALPHA_LIST, LINE_TYPE_LIST
> 07/23 00:32:16.942 metplus.StatAnalysis
(stat_analysis_wrapper.py:237)
> DEBUG: Items in these lists will be looped over:
FCST_VALID_HOUR_LIST
> 07/23 00:32:16.957 metplus.StatAnalysis (met_util.py:192) ERROR:
Fatal
> error occurred
> Traceback (most recent call last):
>   File
>
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> line 172, in run_metplus
>     loop_over_times_and_call(config, processes)
>   File
>
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> line 834, in loop_over_times_and_call
>     process.run_at_time(input_dict)
>   File
>
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> line 1872, in run_at_time
>     self.run_stat_analysis_job(date, date, loop_by)
>   File
>
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> line 1288, in run_stat_analysis_job
>     runtime_settings_dict)
>   File
>
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> line 836, in get_lookin_dir
>     subprocess.check_output('ls -d '+dir_path_filled, shell=True)
>   File
"/usrx/local/dev/packages/python/3.6.3/lib/python3.6/subprocess.py",
> line 336, in check_output
>     **kwargs).stdout
>   File
"/usrx/local/dev/packages/python/3.6.3/lib/python3.6/subprocess.py",
> line 418, in run
>     output=stdout, stderr=stderr)
> subprocess.CalledProcessError: Command 'ls -d
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*'
> returned non-zero exit status 2.
>
> If the step worked, then the line that would follow "....looped
over:
> FCST_VALID_HOUR_LIST"  would be
> 07/23 14:55:12.158 metplus.StatAnalysis
(stat_analysis_wrapper.py:788)
> DEBUG: Building dump_row filename from user template:
> {valid?fmt=%Y%m%d}/PARA13_BC_{valid_beg?fmt=%Y%m%d%H}.stat
>
> If this step fails, it always happens at the first valid time (in
this case
> 2020072000).  Following the file name template, I would have
something like
> this:
>
> -rw-r--r-- 1 Edward.Strobach emcmodel 525632 Jul 23 14:55
> PARA13_BC_2020072000.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 344951 Jul 23 14:55
> PARA13_BC_2020072001.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 345726 Jul 23 14:55
> PARA13_BC_2020072002.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 343403 Jul 23 14:55
> PARA13_BC_2020072003.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 346222 Jul 23 14:55
> PARA13_BC_2020072004.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 342297 Jul 23 14:55
> PARA13_BC_2020072005.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 416853 Jul 23 14:55
> PARA13_BC_2020072006.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 343406 Jul 23 14:55
> PARA13_BC_2020072007.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 346832 Jul 23 14:55
> PARA13_BC_2020072008.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 342291 Jul 23 14:55
> PARA13_BC_2020072009.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 347004 Jul 23 14:55
> PARA13_BC_2020072010.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 358938 Jul 23 14:55
> PARA13_BC_2020072011.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 519089 Jul 23 14:55
> PARA13_BC_2020072012.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 414343 Jul 23 14:55
> PARA13_BC_2020072013.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 437188 Jul 23 14:55
> PARA13_BC_2020072014.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 435365 Jul 23 14:56
> PARA13_BC_2020072015.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 433295 Jul 23 14:56
> PARA13_BC_2020072016.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 433698 Jul 23 14:56
> PARA13_BC_2020072017.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 508369 Jul 23 14:56
> PARA13_BC_2020072018.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 447083 Jul 23 14:56
> PARA13_BC_2020072019.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 441350 Jul 23 14:56
> PARA13_BC_2020072020.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 437671 Jul 23 14:56
> PARA13_BC_2020072021.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 416085 Jul 23 14:56
> PARA13_BC_2020072022.stat
> -rw-r--r-- 1 Edward.Strobach emcmodel 444785 Jul 23 14:56
> PARA13_BC_2020072023.stat
>
> You can see these are indeed populated, and that these are the only
ones
> generated.  If all files were generated, then I would have
PROD_BC_*stat,
> PROD_*stat, and PARA13_*stat.  The day before both PROD and
PARA13_BC
> worked.  I've also had cases where all model runs were successful.
I'm not
> sure why this would be the case.  I don't see anything indicating a
problem
> within the point stat files themselves..
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>

------------------------------------------------
Subject: StatAnalysis works intermittently with CMAQ meteorology files
From: John Halley Gotway
Time: Thu Jul 23 12:33:43 2020

Hi Ed (and George),

I see you're getting errors from the METplus Stat-Analysis wrapper.
Based
on the log message you sent, it's erroring out when listing those
files:

ls -d
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*

But when I list them now, I see 112 of them:

[John.H.Gotway at m72a1 ~]$ ls -d
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*
| wc -w
112

I would guess this has something to do with timing. Perhaps if this
wrapper
is run prior to those files being written, then there's no work to do.
I do
think the logic should be improved to better handle this case. I
suspect
the bad return status is causing a problem. When there's no match, you
get
"No such file or directory" and a bad return status of 2:

[John.H.Gotway at m72a1 ~]$ ls -d
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*BAD*

ls: cannot access
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*BAD*:
No such file or directory
[John.H.Gotway at m72a1 ~]$ echo $?
2

George, we should enhance this Stat-Analysis wrapper to check for no
matching files. And then apply more sane logic instead or a useful log
message instead of exiting.

Who's the right person to do that?

Thanks,
John

On Thu, Jul 23, 2020 at 9:28 AM Minna Win via RT <met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996 >
>
> Hello Edward,
>
> It looks like you have some issues with your StatAnalysis runs.
I've
> assigned this issue to John Halley Gotway.  Please allow a few
business
> days for a full response.
>
> Regards,
> Minna
> ---------------
> Minna Win
> National Center for Atmospheric Research
> Developmental Testbed Center
> Phone: 303-497-8423
> Fax:   303-497-8401
>
>
>
> On Thu, Jul 23, 2020 at 9:18 AM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Jul 23 09:18:05 2020: Request 95996 was acted upon.
> > Transaction: Ticket created by edward.strobach at noaa.gov
> >        Queue: met_help
> >      Subject: StatAnalysis works intermittently with CMAQ
meteorology
> files
> >        Owner: Nobody
> >   Requestors: edward.strobach at noaa.gov
> >       Status: new
> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996 >
> >
> >
> > Good morning,
> >
> > I've been attempting to set up verification for meteorology from
the CMAQ
> > output.  In the past, I was able to successfully generate files
for the
> > PB2NC, PointStat, and StatAnalysis steps.  In confirming the
success of
> > these steps, I then set up my configuration to handle model runs
with
> files
> > that follow the same naming structure.  The PB2NC and PointStat
files are
> > always successful, while the StatAnalysis works intermittently.
> Yesterday,
> > I was able to generate stat files from StatAnalysis for 2 out of
my 4
> model
> > runs.  Today, however, only one model run ended up being
successful.  I'm
> > not sure why it would work for one day, then the next day it
doesn't
> work.
> > The error is always the same.
> >
> > 07/23 00:32:16.941 metplus (config_launcher.py:520) DEBUG: Setting
> [config]
> > MODEL1_REFERENCE_NAME to default value: PROD.
> > 07/23 00:32:16.942 metplus.StatAnalysis
(stat_analysis_wrapper.py:235)
> > DEBUG: Items in these lists will be grouped together: MODEL_LIST,
> > DESC_LIST, FCST_LEAD_LIST, OBS_LEAD_LIST, FCST_INIT_HOUR_LIST,
> > OBS_VALID_HOUR_LIST, OBS_INIT_HOUR_LIST, FCST_VAR_LIST,
OBS_VAR_LIST,
> > FCST_UNITS_LIST, OBS_UNITS_LIST, FCST_LEVEL_LIST, OBS_LEVEL_LIST,
> > VX_MASK_LIST, INTERP_MTHD_LIST, INTERP_PNTS_LIST,
FCST_THRESH_LIST,
> > OBS_THRESH_LIST, COV_THRESH_LIST, ALPHA_LIST, LINE_TYPE_LIST
> > 07/23 00:32:16.942 metplus.StatAnalysis
(stat_analysis_wrapper.py:237)
> > DEBUG: Items in these lists will be looped over:
FCST_VALID_HOUR_LIST
> > 07/23 00:32:16.957 metplus.StatAnalysis (met_util.py:192) ERROR:
Fatal
> > error occurred
> > Traceback (most recent call last):
> >   File
> >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > line 172, in run_metplus
> >     loop_over_times_and_call(config, processes)
> >   File
> >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > line 834, in loop_over_times_and_call
> >     process.run_at_time(input_dict)
> >   File
> >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > line 1872, in run_at_time
> >     self.run_stat_analysis_job(date, date, loop_by)
> >   File
> >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > line 1288, in run_stat_analysis_job
> >     runtime_settings_dict)
> >   File
> >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > line 836, in get_lookin_dir
> >     subprocess.check_output('ls -d '+dir_path_filled, shell=True)
> >   File
> "/usrx/local/dev/packages/python/3.6.3/lib/python3.6/subprocess.py",
> > line 336, in check_output
> >     **kwargs).stdout
> >   File
> "/usrx/local/dev/packages/python/3.6.3/lib/python3.6/subprocess.py",
> > line 418, in run
> >     output=stdout, stderr=stderr)
> > subprocess.CalledProcessError: Command 'ls -d
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*'
> > returned non-zero exit status 2.
> >
> > If the step worked, then the line that would follow "....looped
over:
> > FCST_VALID_HOUR_LIST"  would be
> > 07/23 14:55:12.158 metplus.StatAnalysis
(stat_analysis_wrapper.py:788)
> > DEBUG: Building dump_row filename from user template:
> > {valid?fmt=%Y%m%d}/PARA13_BC_{valid_beg?fmt=%Y%m%d%H}.stat
> >
> > If this step fails, it always happens at the first valid time (in
this
> case
> > 2020072000).  Following the file name template, I would have
something
> like
> > this:
> >
> > -rw-r--r-- 1 Edward.Strobach emcmodel 525632 Jul 23 14:55
> > PARA13_BC_2020072000.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 344951 Jul 23 14:55
> > PARA13_BC_2020072001.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 345726 Jul 23 14:55
> > PARA13_BC_2020072002.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 343403 Jul 23 14:55
> > PARA13_BC_2020072003.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 346222 Jul 23 14:55
> > PARA13_BC_2020072004.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 342297 Jul 23 14:55
> > PARA13_BC_2020072005.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 416853 Jul 23 14:55
> > PARA13_BC_2020072006.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 343406 Jul 23 14:55
> > PARA13_BC_2020072007.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 346832 Jul 23 14:55
> > PARA13_BC_2020072008.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 342291 Jul 23 14:55
> > PARA13_BC_2020072009.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 347004 Jul 23 14:55
> > PARA13_BC_2020072010.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 358938 Jul 23 14:55
> > PARA13_BC_2020072011.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 519089 Jul 23 14:55
> > PARA13_BC_2020072012.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 414343 Jul 23 14:55
> > PARA13_BC_2020072013.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 437188 Jul 23 14:55
> > PARA13_BC_2020072014.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 435365 Jul 23 14:56
> > PARA13_BC_2020072015.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 433295 Jul 23 14:56
> > PARA13_BC_2020072016.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 433698 Jul 23 14:56
> > PARA13_BC_2020072017.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 508369 Jul 23 14:56
> > PARA13_BC_2020072018.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 447083 Jul 23 14:56
> > PARA13_BC_2020072019.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 441350 Jul 23 14:56
> > PARA13_BC_2020072020.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 437671 Jul 23 14:56
> > PARA13_BC_2020072021.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 416085 Jul 23 14:56
> > PARA13_BC_2020072022.stat
> > -rw-r--r-- 1 Edward.Strobach emcmodel 444785 Jul 23 14:56
> > PARA13_BC_2020072023.stat
> >
> > You can see these are indeed populated, and that these are the
only ones
> > generated.  If all files were generated, then I would have
PROD_BC_*stat,
> > PROD_*stat, and PARA13_*stat.  The day before both PROD and
PARA13_BC
> > worked.  I've also had cases where all model runs were successful.
I'm
> not
> > sure why this would be the case.  I don't see anything indicating
a
> problem
> > within the point stat files themselves..
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
>

------------------------------------------------
Subject: StatAnalysis works intermittently with CMAQ meteorology files
From: Edward Strobach - NOAA Affiliate
Time: Thu Jul 23 12:55:44 2020

I see.  It threw me off because the issue is not consistent.  I've
looked
in other files related to the StatAnalysis process and thought that
all
fields stored inside the stat files generated by PointStat would be
carried
over to stat files generated by StatAnalysis.  I forgot to mention
that
this is not the case, and that the PBL field and 10-m wind does not
appear
to be stored in those files for some reason.  There's nothing in the
related files that would indicate why unfortunately.

On Thu, Jul 23, 2020 at 2:33 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:

> Hi Ed (and George),
>
> I see you're getting errors from the METplus Stat-Analysis wrapper.
Based
> on the log message you sent, it's erroring out when listing those
files:
>
> ls -d
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*
>
> But when I list them now, I see 112 of them:
>
> [John.H.Gotway at m72a1 ~]$ ls -d
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*
> | wc -w
> 112
>
> I would guess this has something to do with timing. Perhaps if this
wrapper
> is run prior to those files being written, then there's no work to
do. I do
> think the logic should be improved to better handle this case. I
suspect
> the bad return status is causing a problem. When there's no match,
you get
> "No such file or directory" and a bad return status of 2:
>
> [John.H.Gotway at m72a1 ~]$ ls -d
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*BAD*
>
> ls: cannot access
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*BAD*:
> No such file or directory
> [John.H.Gotway at m72a1 ~]$ echo $?
> 2
>
> George, we should enhance this Stat-Analysis wrapper to check for no
> matching files. And then apply more sane logic instead or a useful
log
> message instead of exiting.
>
> Who's the right person to do that?
>
> Thanks,
> John
>
> On Thu, Jul 23, 2020 at 9:28 AM Minna Win via RT <met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996 >
> >
> > Hello Edward,
> >
> > It looks like you have some issues with your StatAnalysis runs.
I've
> > assigned this issue to John Halley Gotway.  Please allow a few
business
> > days for a full response.
> >
> > Regards,
> > Minna
> > ---------------
> > Minna Win
> > National Center for Atmospheric Research
> > Developmental Testbed Center
> > Phone: 303-497-8423
> > Fax:   303-497-8401
> >
> >
> >
> > On Thu, Jul 23, 2020 at 9:18 AM Edward Strobach - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Jul 23 09:18:05 2020: Request 95996 was acted upon.
> > > Transaction: Ticket created by edward.strobach at noaa.gov
> > >        Queue: met_help
> > >      Subject: StatAnalysis works intermittently with CMAQ
meteorology
> > files
> > >        Owner: Nobody
> > >   Requestors: edward.strobach at noaa.gov
> > >       Status: new
> > >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996
> >
> > >
> > >
> > > Good morning,
> > >
> > > I've been attempting to set up verification for meteorology from
the
> CMAQ
> > > output.  In the past, I was able to successfully generate files
for the
> > > PB2NC, PointStat, and StatAnalysis steps.  In confirming the
success of
> > > these steps, I then set up my configuration to handle model runs
with
> > files
> > > that follow the same naming structure.  The PB2NC and PointStat
files
> are
> > > always successful, while the StatAnalysis works intermittently.
> > Yesterday,
> > > I was able to generate stat files from StatAnalysis for 2 out of
my 4
> > model
> > > runs.  Today, however, only one model run ended up being
successful.
> I'm
> > > not sure why it would work for one day, then the next day it
doesn't
> > work.
> > > The error is always the same.
> > >
> > > 07/23 00:32:16.941 metplus (config_launcher.py:520) DEBUG:
Setting
> > [config]
> > > MODEL1_REFERENCE_NAME to default value: PROD.
> > > 07/23 00:32:16.942 metplus.StatAnalysis
(stat_analysis_wrapper.py:235)
> > > DEBUG: Items in these lists will be grouped together:
MODEL_LIST,
> > > DESC_LIST, FCST_LEAD_LIST, OBS_LEAD_LIST, FCST_INIT_HOUR_LIST,
> > > OBS_VALID_HOUR_LIST, OBS_INIT_HOUR_LIST, FCST_VAR_LIST,
OBS_VAR_LIST,
> > > FCST_UNITS_LIST, OBS_UNITS_LIST, FCST_LEVEL_LIST,
OBS_LEVEL_LIST,
> > > VX_MASK_LIST, INTERP_MTHD_LIST, INTERP_PNTS_LIST,
FCST_THRESH_LIST,
> > > OBS_THRESH_LIST, COV_THRESH_LIST, ALPHA_LIST, LINE_TYPE_LIST
> > > 07/23 00:32:16.942 metplus.StatAnalysis
(stat_analysis_wrapper.py:237)
> > > DEBUG: Items in these lists will be looped over:
FCST_VALID_HOUR_LIST
> > > 07/23 00:32:16.957 metplus.StatAnalysis (met_util.py:192) ERROR:
Fatal
> > > error occurred
> > > Traceback (most recent call last):
> > >   File
> > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > line 172, in run_metplus
> > >     loop_over_times_and_call(config, processes)
> > >   File
> > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > line 834, in loop_over_times_and_call
> > >     process.run_at_time(input_dict)
> > >   File
> > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > > line 1872, in run_at_time
> > >     self.run_stat_analysis_job(date, date, loop_by)
> > >   File
> > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > > line 1288, in run_stat_analysis_job
> > >     runtime_settings_dict)
> > >   File
> > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > > line 836, in get_lookin_dir
> > >     subprocess.check_output('ls -d '+dir_path_filled,
shell=True)
> > >   File
> >
"/usrx/local/dev/packages/python/3.6.3/lib/python3.6/subprocess.py",
> > > line 336, in check_output
> > >     **kwargs).stdout
> > >   File
> >
"/usrx/local/dev/packages/python/3.6.3/lib/python3.6/subprocess.py",
> > > line 418, in run
> > >     output=stdout, stderr=stderr)
> > > subprocess.CalledProcessError: Command 'ls -d
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*'
> > > returned non-zero exit status 2.
> > >
> > > If the step worked, then the line that would follow "....looped
over:
> > > FCST_VALID_HOUR_LIST"  would be
> > > 07/23 14:55:12.158 metplus.StatAnalysis
(stat_analysis_wrapper.py:788)
> > > DEBUG: Building dump_row filename from user template:
> > > {valid?fmt=%Y%m%d}/PARA13_BC_{valid_beg?fmt=%Y%m%d%H}.stat
> > >
> > > If this step fails, it always happens at the first valid time
(in this
> > case
> > > 2020072000).  Following the file name template, I would have
something
> > like
> > > this:
> > >
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 525632 Jul 23 14:55
> > > PARA13_BC_2020072000.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 344951 Jul 23 14:55
> > > PARA13_BC_2020072001.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 345726 Jul 23 14:55
> > > PARA13_BC_2020072002.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 343403 Jul 23 14:55
> > > PARA13_BC_2020072003.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 346222 Jul 23 14:55
> > > PARA13_BC_2020072004.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 342297 Jul 23 14:55
> > > PARA13_BC_2020072005.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 416853 Jul 23 14:55
> > > PARA13_BC_2020072006.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 343406 Jul 23 14:55
> > > PARA13_BC_2020072007.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 346832 Jul 23 14:55
> > > PARA13_BC_2020072008.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 342291 Jul 23 14:55
> > > PARA13_BC_2020072009.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 347004 Jul 23 14:55
> > > PARA13_BC_2020072010.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 358938 Jul 23 14:55
> > > PARA13_BC_2020072011.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 519089 Jul 23 14:55
> > > PARA13_BC_2020072012.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 414343 Jul 23 14:55
> > > PARA13_BC_2020072013.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 437188 Jul 23 14:55
> > > PARA13_BC_2020072014.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 435365 Jul 23 14:56
> > > PARA13_BC_2020072015.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 433295 Jul 23 14:56
> > > PARA13_BC_2020072016.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 433698 Jul 23 14:56
> > > PARA13_BC_2020072017.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 508369 Jul 23 14:56
> > > PARA13_BC_2020072018.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 447083 Jul 23 14:56
> > > PARA13_BC_2020072019.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 441350 Jul 23 14:56
> > > PARA13_BC_2020072020.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 437671 Jul 23 14:56
> > > PARA13_BC_2020072021.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 416085 Jul 23 14:56
> > > PARA13_BC_2020072022.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 444785 Jul 23 14:56
> > > PARA13_BC_2020072023.stat
> > >
> > > You can see these are indeed populated, and that these are the
only
> ones
> > > generated.  If all files were generated, then I would have
> PROD_BC_*stat,
> > > PROD_*stat, and PARA13_*stat.  The day before both PROD and
PARA13_BC
> > > worked.  I've also had cases where all model runs were
successful.  I'm
> > not
> > > sure why this would be the case.  I don't see anything
indicating a
> > problem
> > > within the point stat files themselves..
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> > >
> >
> >
>
>

--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

------------------------------------------------
Subject: StatAnalysis works intermittently with CMAQ meteorology files
From: George McCabe
Time: Thu Jul 23 12:56:37 2020

Hi Edward (and John),

I recently completed a huge overhaul of the StatAnalysis wrapper. It
looks
like you are using an older version of METplus that doesn't include
these
changes. It is available in the METplus 3.1-beta2 release. These
changes
should hopefully catch something misconfigured earlier in execution to
prevent a crash like this and instead provide a nicer error message
letting
you know what went wrong. Either way, I agree that we should add error
checking to prevent this crash. I will try to recreate this locally.

I would try two things:

1) Running this case with the beta2 release to see if the crash still
occurs
2) Search for "ERROR" earlier in execution to see if something failed
earlier in execution that may explain why you don't have any files in
that
directory.

- George

On Thu, Jul 23, 2020 at 12:33 PM John Halley Gotway via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996 >
>
> Hi Ed (and George),
>
> I see you're getting errors from the METplus Stat-Analysis wrapper.
Based
> on the log message you sent, it's erroring out when listing those
files:
>
> ls -d
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*
>
> But when I list them now, I see 112 of them:
>
> [John.H.Gotway at m72a1 ~]$ ls -d
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*
> | wc -w
> 112
>
> I would guess this has something to do with timing. Perhaps if this
wrapper
> is run prior to those files being written, then there's no work to
do. I do
> think the logic should be improved to better handle this case. I
suspect
> the bad return status is causing a problem. When there's no match,
you get
> "No such file or directory" and a bad return status of 2:
>
> [John.H.Gotway at m72a1 ~]$ ls -d
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*BAD*
>
> ls: cannot access
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*BAD*:
> No such file or directory
> [John.H.Gotway at m72a1 ~]$ echo $?
> 2
>
> George, we should enhance this Stat-Analysis wrapper to check for no
> matching files. And then apply more sane logic instead or a useful
log
> message instead of exiting.
>
> Who's the right person to do that?
>
> Thanks,
> John
>
> On Thu, Jul 23, 2020 at 9:28 AM Minna Win via RT <met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996 >
> >
> > Hello Edward,
> >
> > It looks like you have some issues with your StatAnalysis runs.
I've
> > assigned this issue to John Halley Gotway.  Please allow a few
business
> > days for a full response.
> >
> > Regards,
> > Minna
> > ---------------
> > Minna Win
> > National Center for Atmospheric Research
> > Developmental Testbed Center
> > Phone: 303-497-8423
> > Fax:   303-497-8401
> >
> >
> >
> > On Thu, Jul 23, 2020 at 9:18 AM Edward Strobach - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Jul 23 09:18:05 2020: Request 95996 was acted upon.
> > > Transaction: Ticket created by edward.strobach at noaa.gov
> > >        Queue: met_help
> > >      Subject: StatAnalysis works intermittently with CMAQ
meteorology
> > files
> > >        Owner: Nobody
> > >   Requestors: edward.strobach at noaa.gov
> > >       Status: new
> > >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996
> >
> > >
> > >
> > > Good morning,
> > >
> > > I've been attempting to set up verification for meteorology from
the
> CMAQ
> > > output.  In the past, I was able to successfully generate files
for the
> > > PB2NC, PointStat, and StatAnalysis steps.  In confirming the
success of
> > > these steps, I then set up my configuration to handle model runs
with
> > files
> > > that follow the same naming structure.  The PB2NC and PointStat
files
> are
> > > always successful, while the StatAnalysis works intermittently.
> > Yesterday,
> > > I was able to generate stat files from StatAnalysis for 2 out of
my 4
> > model
> > > runs.  Today, however, only one model run ended up being
successful.
> I'm
> > > not sure why it would work for one day, then the next day it
doesn't
> > work.
> > > The error is always the same.
> > >
> > > 07/23 00:32:16.941 metplus (config_launcher.py:520) DEBUG:
Setting
> > [config]
> > > MODEL1_REFERENCE_NAME to default value: PROD.
> > > 07/23 00:32:16.942 metplus.StatAnalysis
(stat_analysis_wrapper.py:235)
> > > DEBUG: Items in these lists will be grouped together:
MODEL_LIST,
> > > DESC_LIST, FCST_LEAD_LIST, OBS_LEAD_LIST, FCST_INIT_HOUR_LIST,
> > > OBS_VALID_HOUR_LIST, OBS_INIT_HOUR_LIST, FCST_VAR_LIST,
OBS_VAR_LIST,
> > > FCST_UNITS_LIST, OBS_UNITS_LIST, FCST_LEVEL_LIST,
OBS_LEVEL_LIST,
> > > VX_MASK_LIST, INTERP_MTHD_LIST, INTERP_PNTS_LIST,
FCST_THRESH_LIST,
> > > OBS_THRESH_LIST, COV_THRESH_LIST, ALPHA_LIST, LINE_TYPE_LIST
> > > 07/23 00:32:16.942 metplus.StatAnalysis
(stat_analysis_wrapper.py:237)
> > > DEBUG: Items in these lists will be looped over:
FCST_VALID_HOUR_LIST
> > > 07/23 00:32:16.957 metplus.StatAnalysis (met_util.py:192) ERROR:
Fatal
> > > error occurred
> > > Traceback (most recent call last):
> > >   File
> > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > line 172, in run_metplus
> > >     loop_over_times_and_call(config, processes)
> > >   File
> > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > line 834, in loop_over_times_and_call
> > >     process.run_at_time(input_dict)
> > >   File
> > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > > line 1872, in run_at_time
> > >     self.run_stat_analysis_job(date, date, loop_by)
> > >   File
> > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > > line 1288, in run_stat_analysis_job
> > >     runtime_settings_dict)
> > >   File
> > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > > line 836, in get_lookin_dir
> > >     subprocess.check_output('ls -d '+dir_path_filled,
shell=True)
> > >   File
> >
"/usrx/local/dev/packages/python/3.6.3/lib/python3.6/subprocess.py",
> > > line 336, in check_output
> > >     **kwargs).stdout
> > >   File
> >
"/usrx/local/dev/packages/python/3.6.3/lib/python3.6/subprocess.py",
> > > line 418, in run
> > >     output=stdout, stderr=stderr)
> > > subprocess.CalledProcessError: Command 'ls -d
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*'
> > > returned non-zero exit status 2.
> > >
> > > If the step worked, then the line that would follow "....looped
over:
> > > FCST_VALID_HOUR_LIST"  would be
> > > 07/23 14:55:12.158 metplus.StatAnalysis
(stat_analysis_wrapper.py:788)
> > > DEBUG: Building dump_row filename from user template:
> > > {valid?fmt=%Y%m%d}/PARA13_BC_{valid_beg?fmt=%Y%m%d%H}.stat
> > >
> > > If this step fails, it always happens at the first valid time
(in this
> > case
> > > 2020072000).  Following the file name template, I would have
something
> > like
> > > this:
> > >
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 525632 Jul 23 14:55
> > > PARA13_BC_2020072000.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 344951 Jul 23 14:55
> > > PARA13_BC_2020072001.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 345726 Jul 23 14:55
> > > PARA13_BC_2020072002.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 343403 Jul 23 14:55
> > > PARA13_BC_2020072003.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 346222 Jul 23 14:55
> > > PARA13_BC_2020072004.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 342297 Jul 23 14:55
> > > PARA13_BC_2020072005.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 416853 Jul 23 14:55
> > > PARA13_BC_2020072006.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 343406 Jul 23 14:55
> > > PARA13_BC_2020072007.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 346832 Jul 23 14:55
> > > PARA13_BC_2020072008.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 342291 Jul 23 14:55
> > > PARA13_BC_2020072009.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 347004 Jul 23 14:55
> > > PARA13_BC_2020072010.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 358938 Jul 23 14:55
> > > PARA13_BC_2020072011.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 519089 Jul 23 14:55
> > > PARA13_BC_2020072012.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 414343 Jul 23 14:55
> > > PARA13_BC_2020072013.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 437188 Jul 23 14:55
> > > PARA13_BC_2020072014.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 435365 Jul 23 14:56
> > > PARA13_BC_2020072015.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 433295 Jul 23 14:56
> > > PARA13_BC_2020072016.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 433698 Jul 23 14:56
> > > PARA13_BC_2020072017.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 508369 Jul 23 14:56
> > > PARA13_BC_2020072018.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 447083 Jul 23 14:56
> > > PARA13_BC_2020072019.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 441350 Jul 23 14:56
> > > PARA13_BC_2020072020.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 437671 Jul 23 14:56
> > > PARA13_BC_2020072021.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 416085 Jul 23 14:56
> > > PARA13_BC_2020072022.stat
> > > -rw-r--r-- 1 Edward.Strobach emcmodel 444785 Jul 23 14:56
> > > PARA13_BC_2020072023.stat
> > >
> > > You can see these are indeed populated, and that these are the
only
> ones
> > > generated.  If all files were generated, then I would have
> PROD_BC_*stat,
> > > PROD_*stat, and PARA13_*stat.  The day before both PROD and
PARA13_BC
> > > worked.  I've also had cases where all model runs were
successful.  I'm
> > not
> > > sure why this would be the case.  I don't see anything
indicating a
> > problem
> > > within the point stat files themselves..
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> > >
> >
> >
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: StatAnalysis works intermittently with CMAQ meteorology files
From: Edward Strobach - NOAA Affiliate
Time: Thu Jul 23 12:59:20 2020

I've done the second point in the past, which has helped most of the
time
since the error messages tend to be intuitive.  Do I need to reclone
something or should I just specify a new version in my file system?  I
haven't had to do this before.

On Thu, Jul 23, 2020 at 2:56 PM George McCabe via RT
<met_help at ucar.edu>
wrote:

> Hi Edward (and John),
>
> I recently completed a huge overhaul of the StatAnalysis wrapper. It
looks
> like you are using an older version of METplus that doesn't include
these
> changes. It is available in the METplus 3.1-beta2 release. These
changes
> should hopefully catch something misconfigured earlier in execution
to
> prevent a crash like this and instead provide a nicer error message
letting
> you know what went wrong. Either way, I agree that we should add
error
> checking to prevent this crash. I will try to recreate this locally.
>
> I would try two things:
>
> 1) Running this case with the beta2 release to see if the crash
still
> occurs
> 2) Search for "ERROR" earlier in execution to see if something
failed
> earlier in execution that may explain why you don't have any files
in that
> directory.
>
> - George
>
> On Thu, Jul 23, 2020 at 12:33 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996 >
> >
> > Hi Ed (and George),
> >
> > I see you're getting errors from the METplus Stat-Analysis
wrapper. Based
> > on the log message you sent, it's erroring out when listing those
files:
> >
> > ls -d
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*
> >
> > But when I list them now, I see 112 of them:
> >
> > [John.H.Gotway at m72a1 ~]$ ls -d
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*
> > | wc -w
> > 112
> >
> > I would guess this has something to do with timing. Perhaps if
this
> wrapper
> > is run prior to those files being written, then there's no work to
do. I
> do
> > think the logic should be improved to better handle this case. I
suspect
> > the bad return status is causing a problem. When there's no match,
you
> get
> > "No such file or directory" and a bad return status of 2:
> >
> > [John.H.Gotway at m72a1 ~]$ ls -d
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*BAD*
> >
> > ls: cannot access
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*BAD*:
> > No such file or directory
> > [John.H.Gotway at m72a1 ~]$ echo $?
> > 2
> >
> > George, we should enhance this Stat-Analysis wrapper to check for
no
> > matching files. And then apply more sane logic instead or a useful
log
> > message instead of exiting.
> >
> > Who's the right person to do that?
> >
> > Thanks,
> > John
> >
> > On Thu, Jul 23, 2020 at 9:28 AM Minna Win via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996 >
> > >
> > > Hello Edward,
> > >
> > > It looks like you have some issues with your StatAnalysis runs.
I've
> > > assigned this issue to John Halley Gotway.  Please allow a few
business
> > > days for a full response.
> > >
> > > Regards,
> > > Minna
> > > ---------------
> > > Minna Win
> > > National Center for Atmospheric Research
> > > Developmental Testbed Center
> > > Phone: 303-497-8423
> > > Fax:   303-497-8401
> > >
> > >
> > >
> > > On Thu, Jul 23, 2020 at 9:18 AM Edward Strobach - NOAA Affiliate
via
> RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Thu Jul 23 09:18:05 2020: Request 95996 was acted upon.
> > > > Transaction: Ticket created by edward.strobach at noaa.gov
> > > >        Queue: met_help
> > > >      Subject: StatAnalysis works intermittently with CMAQ
meteorology
> > > files
> > > >        Owner: Nobody
> > > >   Requestors: edward.strobach at noaa.gov
> > > >       Status: new
> > > >  Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996
> > >
> > > >
> > > >
> > > > Good morning,
> > > >
> > > > I've been attempting to set up verification for meteorology
from the
> > CMAQ
> > > > output.  In the past, I was able to successfully generate
files for
> the
> > > > PB2NC, PointStat, and StatAnalysis steps.  In confirming the
success
> of
> > > > these steps, I then set up my configuration to handle model
runs with
> > > files
> > > > that follow the same naming structure.  The PB2NC and
PointStat files
> > are
> > > > always successful, while the StatAnalysis works
intermittently.
> > > Yesterday,
> > > > I was able to generate stat files from StatAnalysis for 2 out
of my 4
> > > model
> > > > runs.  Today, however, only one model run ended up being
successful.
> > I'm
> > > > not sure why it would work for one day, then the next day it
doesn't
> > > work.
> > > > The error is always the same.
> > > >
> > > > 07/23 00:32:16.941 metplus (config_launcher.py:520) DEBUG:
Setting
> > > [config]
> > > > MODEL1_REFERENCE_NAME to default value: PROD.
> > > > 07/23 00:32:16.942 metplus.StatAnalysis
> (stat_analysis_wrapper.py:235)
> > > > DEBUG: Items in these lists will be grouped together:
MODEL_LIST,
> > > > DESC_LIST, FCST_LEAD_LIST, OBS_LEAD_LIST, FCST_INIT_HOUR_LIST,
> > > > OBS_VALID_HOUR_LIST, OBS_INIT_HOUR_LIST, FCST_VAR_LIST,
OBS_VAR_LIST,
> > > > FCST_UNITS_LIST, OBS_UNITS_LIST, FCST_LEVEL_LIST,
OBS_LEVEL_LIST,
> > > > VX_MASK_LIST, INTERP_MTHD_LIST, INTERP_PNTS_LIST,
FCST_THRESH_LIST,
> > > > OBS_THRESH_LIST, COV_THRESH_LIST, ALPHA_LIST, LINE_TYPE_LIST
> > > > 07/23 00:32:16.942 metplus.StatAnalysis
> (stat_analysis_wrapper.py:237)
> > > > DEBUG: Items in these lists will be looped over:
FCST_VALID_HOUR_LIST
> > > > 07/23 00:32:16.957 metplus.StatAnalysis (met_util.py:192)
ERROR:
> Fatal
> > > > error occurred
> > > > Traceback (most recent call last):
> > > >   File
> > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > line 172, in run_metplus
> > > >     loop_over_times_and_call(config, processes)
> > > >   File
> > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > line 834, in loop_over_times_and_call
> > > >     process.run_at_time(input_dict)
> > > >   File
> > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > > > line 1872, in run_at_time
> > > >     self.run_stat_analysis_job(date, date, loop_by)
> > > >   File
> > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > > > line 1288, in run_stat_analysis_job
> > > >     runtime_settings_dict)
> > > >   File
> > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > > > line 836, in get_lookin_dir
> > > >     subprocess.check_output('ls -d '+dir_path_filled,
shell=True)
> > > >   File
> > >
"/usrx/local/dev/packages/python/3.6.3/lib/python3.6/subprocess.py",
> > > > line 336, in check_output
> > > >     **kwargs).stdout
> > > >   File
> > >
"/usrx/local/dev/packages/python/3.6.3/lib/python3.6/subprocess.py",
> > > > line 418, in run
> > > >     output=stdout, stderr=stderr)
> > > > subprocess.CalledProcessError: Command 'ls -d
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*'
> > > > returned non-zero exit status 2.
> > > >
> > > > If the step worked, then the line that would follow
"....looped over:
> > > > FCST_VALID_HOUR_LIST"  would be
> > > > 07/23 14:55:12.158 metplus.StatAnalysis
> (stat_analysis_wrapper.py:788)
> > > > DEBUG: Building dump_row filename from user template:
> > > > {valid?fmt=%Y%m%d}/PARA13_BC_{valid_beg?fmt=%Y%m%d%H}.stat
> > > >
> > > > If this step fails, it always happens at the first valid time
(in
> this
> > > case
> > > > 2020072000).  Following the file name template, I would have
> something
> > > like
> > > > this:
> > > >
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 525632 Jul 23 14:55
> > > > PARA13_BC_2020072000.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 344951 Jul 23 14:55
> > > > PARA13_BC_2020072001.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 345726 Jul 23 14:55
> > > > PARA13_BC_2020072002.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 343403 Jul 23 14:55
> > > > PARA13_BC_2020072003.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 346222 Jul 23 14:55
> > > > PARA13_BC_2020072004.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 342297 Jul 23 14:55
> > > > PARA13_BC_2020072005.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 416853 Jul 23 14:55
> > > > PARA13_BC_2020072006.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 343406 Jul 23 14:55
> > > > PARA13_BC_2020072007.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 346832 Jul 23 14:55
> > > > PARA13_BC_2020072008.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 342291 Jul 23 14:55
> > > > PARA13_BC_2020072009.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 347004 Jul 23 14:55
> > > > PARA13_BC_2020072010.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 358938 Jul 23 14:55
> > > > PARA13_BC_2020072011.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 519089 Jul 23 14:55
> > > > PARA13_BC_2020072012.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 414343 Jul 23 14:55
> > > > PARA13_BC_2020072013.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 437188 Jul 23 14:55
> > > > PARA13_BC_2020072014.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 435365 Jul 23 14:56
> > > > PARA13_BC_2020072015.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 433295 Jul 23 14:56
> > > > PARA13_BC_2020072016.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 433698 Jul 23 14:56
> > > > PARA13_BC_2020072017.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 508369 Jul 23 14:56
> > > > PARA13_BC_2020072018.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 447083 Jul 23 14:56
> > > > PARA13_BC_2020072019.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 441350 Jul 23 14:56
> > > > PARA13_BC_2020072020.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 437671 Jul 23 14:56
> > > > PARA13_BC_2020072021.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 416085 Jul 23 14:56
> > > > PARA13_BC_2020072022.stat
> > > > -rw-r--r-- 1 Edward.Strobach emcmodel 444785 Jul 23 14:56
> > > > PARA13_BC_2020072023.stat
> > > >
> > > > You can see these are indeed populated, and that these are the
only
> > ones
> > > > generated.  If all files were generated, then I would have
> > PROD_BC_*stat,
> > > > PROD_*stat, and PARA13_*stat.  The day before both PROD and
PARA13_BC
> > > > worked.  I've also had cases where all model runs were
successful.
> I'm
> > > not
> > > > sure why this would be the case.  I don't see anything
indicating a
> > > problem
> > > > within the point stat files themselves..
> > > > --
> > > > Edward Strobach
> > > > EMC/NCEP/NWS/
> > > > IMSG Contractor
> > > > Cubicle#: 2029
> > > > 301-683-3717
> > > >
> > > >
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

------------------------------------------------
Subject: StatAnalysis works intermittently with CMAQ meteorology files
From: John Halley Gotway
Time: Thu Jul 23 14:26:27 2020

Hopefully, you can just point to the updated version of METplus and
give
that a shot.

John

On Thu, Jul 23, 2020 at 12:59 PM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996 >
>
> I've done the second point in the past, which has helped most of the
time
> since the error messages tend to be intuitive.  Do I need to reclone
> something or should I just specify a new version in my file system?
I
> haven't had to do this before.
>
> On Thu, Jul 23, 2020 at 2:56 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Edward (and John),
> >
> > I recently completed a huge overhaul of the StatAnalysis wrapper.
It
> looks
> > like you are using an older version of METplus that doesn't
include these
> > changes. It is available in the METplus 3.1-beta2 release. These
changes
> > should hopefully catch something misconfigured earlier in
execution to
> > prevent a crash like this and instead provide a nicer error
message
> letting
> > you know what went wrong. Either way, I agree that we should add
error
> > checking to prevent this crash. I will try to recreate this
locally.
> >
> > I would try two things:
> >
> > 1) Running this case with the beta2 release to see if the crash
still
> > occurs
> > 2) Search for "ERROR" earlier in execution to see if something
failed
> > earlier in execution that may explain why you don't have any files
in
> that
> > directory.
> >
> > - George
> >
> > On Thu, Jul 23, 2020 at 12:33 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996 >
> > >
> > > Hi Ed (and George),
> > >
> > > I see you're getting errors from the METplus Stat-Analysis
wrapper.
> Based
> > > on the log message you sent, it's erroring out when listing
those
> files:
> > >
> > > ls -d
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*
> > >
> > > But when I list them now, I see 112 of them:
> > >
> > > [John.H.Gotway at m72a1 ~]$ ls -d
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*
> > > | wc -w
> > > 112
> > >
> > > I would guess this has something to do with timing. Perhaps if
this
> > wrapper
> > > is run prior to those files being written, then there's no work
to do.
> I
> > do
> > > think the logic should be improved to better handle this case. I
> suspect
> > > the bad return status is causing a problem. When there's no
match, you
> > get
> > > "No such file or directory" and a bad return status of 2:
> > >
> > > [John.H.Gotway at m72a1 ~]$ ls -d
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*BAD*
> > >
> > > ls: cannot access
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*BAD*:
> > > No such file or directory
> > > [John.H.Gotway at m72a1 ~]$ echo $?
> > > 2
> > >
> > > George, we should enhance this Stat-Analysis wrapper to check
for no
> > > matching files. And then apply more sane logic instead or a
useful log
> > > message instead of exiting.
> > >
> > > Who's the right person to do that?
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Jul 23, 2020 at 9:28 AM Minna Win via RT
<met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996
>
> > > >
> > > > Hello Edward,
> > > >
> > > > It looks like you have some issues with your StatAnalysis
runs.  I've
> > > > assigned this issue to John Halley Gotway.  Please allow a few
> business
> > > > days for a full response.
> > > >
> > > > Regards,
> > > > Minna
> > > > ---------------
> > > > Minna Win
> > > > National Center for Atmospheric Research
> > > > Developmental Testbed Center
> > > > Phone: 303-497-8423
> > > > Fax:   303-497-8401
> > > >
> > > >
> > > >
> > > > On Thu, Jul 23, 2020 at 9:18 AM Edward Strobach - NOAA
Affiliate via
> > RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Thu Jul 23 09:18:05 2020: Request 95996 was acted upon.
> > > > > Transaction: Ticket created by edward.strobach at noaa.gov
> > > > >        Queue: met_help
> > > > >      Subject: StatAnalysis works intermittently with CMAQ
> meteorology
> > > > files
> > > > >        Owner: Nobody
> > > > >   Requestors: edward.strobach at noaa.gov
> > > > >       Status: new
> > > > >  Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996
> > > >
> > > > >
> > > > >
> > > > > Good morning,
> > > > >
> > > > > I've been attempting to set up verification for meteorology
from
> the
> > > CMAQ
> > > > > output.  In the past, I was able to successfully generate
files for
> > the
> > > > > PB2NC, PointStat, and StatAnalysis steps.  In confirming the
> success
> > of
> > > > > these steps, I then set up my configuration to handle model
runs
> with
> > > > files
> > > > > that follow the same naming structure.  The PB2NC and
PointStat
> files
> > > are
> > > > > always successful, while the StatAnalysis works
intermittently.
> > > > Yesterday,
> > > > > I was able to generate stat files from StatAnalysis for 2
out of
> my 4
> > > > model
> > > > > runs.  Today, however, only one model run ended up being
> successful.
> > > I'm
> > > > > not sure why it would work for one day, then the next day it
> doesn't
> > > > work.
> > > > > The error is always the same.
> > > > >
> > > > > 07/23 00:32:16.941 metplus (config_launcher.py:520) DEBUG:
Setting
> > > > [config]
> > > > > MODEL1_REFERENCE_NAME to default value: PROD.
> > > > > 07/23 00:32:16.942 metplus.StatAnalysis
> > (stat_analysis_wrapper.py:235)
> > > > > DEBUG: Items in these lists will be grouped together:
MODEL_LIST,
> > > > > DESC_LIST, FCST_LEAD_LIST, OBS_LEAD_LIST,
FCST_INIT_HOUR_LIST,
> > > > > OBS_VALID_HOUR_LIST, OBS_INIT_HOUR_LIST, FCST_VAR_LIST,
> OBS_VAR_LIST,
> > > > > FCST_UNITS_LIST, OBS_UNITS_LIST, FCST_LEVEL_LIST,
OBS_LEVEL_LIST,
> > > > > VX_MASK_LIST, INTERP_MTHD_LIST, INTERP_PNTS_LIST,
FCST_THRESH_LIST,
> > > > > OBS_THRESH_LIST, COV_THRESH_LIST, ALPHA_LIST, LINE_TYPE_LIST
> > > > > 07/23 00:32:16.942 metplus.StatAnalysis
> > (stat_analysis_wrapper.py:237)
> > > > > DEBUG: Items in these lists will be looped over:
> FCST_VALID_HOUR_LIST
> > > > > 07/23 00:32:16.957 metplus.StatAnalysis (met_util.py:192)
ERROR:
> > Fatal
> > > > > error occurred
> > > > > Traceback (most recent call last):
> > > > >   File
> > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > > line 172, in run_metplus
> > > > >     loop_over_times_and_call(config, processes)
> > > > >   File
> > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > > line 834, in loop_over_times_and_call
> > > > >     process.run_at_time(input_dict)
> > > > >   File
> > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > > > > line 1872, in run_at_time
> > > > >     self.run_stat_analysis_job(date, date, loop_by)
> > > > >   File
> > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > > > > line 1288, in run_stat_analysis_job
> > > > >     runtime_settings_dict)
> > > > >   File
> > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > > > > line 836, in get_lookin_dir
> > > > >     subprocess.check_output('ls -d '+dir_path_filled,
shell=True)
> > > > >   File
> > > >
"/usrx/local/dev/packages/python/3.6.3/lib/python3.6/subprocess.py",
> > > > > line 336, in check_output
> > > > >     **kwargs).stdout
> > > > >   File
> > > >
"/usrx/local/dev/packages/python/3.6.3/lib/python3.6/subprocess.py",
> > > > > line 418, in run
> > > > >     output=stdout, stderr=stderr)
> > > > > subprocess.CalledProcessError: Command 'ls -d
> > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*'
> > > > > returned non-zero exit status 2.
> > > > >
> > > > > If the step worked, then the line that would follow
"....looped
> over:
> > > > > FCST_VALID_HOUR_LIST"  would be
> > > > > 07/23 14:55:12.158 metplus.StatAnalysis
> > (stat_analysis_wrapper.py:788)
> > > > > DEBUG: Building dump_row filename from user template:
> > > > > {valid?fmt=%Y%m%d}/PARA13_BC_{valid_beg?fmt=%Y%m%d%H}.stat
> > > > >
> > > > > If this step fails, it always happens at the first valid
time (in
> > this
> > > > case
> > > > > 2020072000).  Following the file name template, I would have
> > something
> > > > like
> > > > > this:
> > > > >
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 525632 Jul 23 14:55
> > > > > PARA13_BC_2020072000.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 344951 Jul 23 14:55
> > > > > PARA13_BC_2020072001.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 345726 Jul 23 14:55
> > > > > PARA13_BC_2020072002.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 343403 Jul 23 14:55
> > > > > PARA13_BC_2020072003.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 346222 Jul 23 14:55
> > > > > PARA13_BC_2020072004.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 342297 Jul 23 14:55
> > > > > PARA13_BC_2020072005.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 416853 Jul 23 14:55
> > > > > PARA13_BC_2020072006.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 343406 Jul 23 14:55
> > > > > PARA13_BC_2020072007.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 346832 Jul 23 14:55
> > > > > PARA13_BC_2020072008.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 342291 Jul 23 14:55
> > > > > PARA13_BC_2020072009.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 347004 Jul 23 14:55
> > > > > PARA13_BC_2020072010.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 358938 Jul 23 14:55
> > > > > PARA13_BC_2020072011.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 519089 Jul 23 14:55
> > > > > PARA13_BC_2020072012.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 414343 Jul 23 14:55
> > > > > PARA13_BC_2020072013.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 437188 Jul 23 14:55
> > > > > PARA13_BC_2020072014.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 435365 Jul 23 14:56
> > > > > PARA13_BC_2020072015.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 433295 Jul 23 14:56
> > > > > PARA13_BC_2020072016.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 433698 Jul 23 14:56
> > > > > PARA13_BC_2020072017.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 508369 Jul 23 14:56
> > > > > PARA13_BC_2020072018.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 447083 Jul 23 14:56
> > > > > PARA13_BC_2020072019.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 441350 Jul 23 14:56
> > > > > PARA13_BC_2020072020.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 437671 Jul 23 14:56
> > > > > PARA13_BC_2020072021.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 416085 Jul 23 14:56
> > > > > PARA13_BC_2020072022.stat
> > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 444785 Jul 23 14:56
> > > > > PARA13_BC_2020072023.stat
> > > > >
> > > > > You can see these are indeed populated, and that these are
the only
> > > ones
> > > > > generated.  If all files were generated, then I would have
> > > PROD_BC_*stat,
> > > > > PROD_*stat, and PARA13_*stat.  The day before both PROD and
> PARA13_BC
> > > > > worked.  I've also had cases where all model runs were
successful.
> > I'm
> > > > not
> > > > > sure why this would be the case.  I don't see anything
indicating a
> > > > problem
> > > > > within the point stat files themselves..
> > > > > --
> > > > > Edward Strobach
> > > > > EMC/NCEP/NWS/
> > > > > IMSG Contractor
> > > > > Cubicle#: 2029
> > > > > 301-683-3717
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>

------------------------------------------------
Subject: StatAnalysis works intermittently with CMAQ meteorology files
From: Edward Strobach - NOAA Affiliate
Time: Thu Jul 23 15:05:58 2020

Great, I'll try to find the newest location and give that a try.

On Thu, Jul 23, 2020 at 4:26 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:

> Hopefully, you can just point to the updated version of METplus and
give
> that a shot.
>
> John
>
> On Thu, Jul 23, 2020 at 12:59 PM Edward Strobach - NOAA Affiliate
via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996 >
> >
> > I've done the second point in the past, which has helped most of
the time
> > since the error messages tend to be intuitive.  Do I need to
reclone
> > something or should I just specify a new version in my file
system?  I
> > haven't had to do this before.
> >
> > On Thu, Jul 23, 2020 at 2:56 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Edward (and John),
> > >
> > > I recently completed a huge overhaul of the StatAnalysis
wrapper. It
> > looks
> > > like you are using an older version of METplus that doesn't
include
> these
> > > changes. It is available in the METplus 3.1-beta2 release. These
> changes
> > > should hopefully catch something misconfigured earlier in
execution to
> > > prevent a crash like this and instead provide a nicer error
message
> > letting
> > > you know what went wrong. Either way, I agree that we should add
error
> > > checking to prevent this crash. I will try to recreate this
locally.
> > >
> > > I would try two things:
> > >
> > > 1) Running this case with the beta2 release to see if the crash
still
> > > occurs
> > > 2) Search for "ERROR" earlier in execution to see if something
failed
> > > earlier in execution that may explain why you don't have any
files in
> > that
> > > directory.
> > >
> > > - George
> > >
> > > On Thu, Jul 23, 2020 at 12:33 PM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996
>
> > > >
> > > > Hi Ed (and George),
> > > >
> > > > I see you're getting errors from the METplus Stat-Analysis
wrapper.
> > Based
> > > > on the log message you sent, it's erroring out when listing
those
> > files:
> > > >
> > > > ls -d
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*
> > > >
> > > > But when I list them now, I see 112 of them:
> > > >
> > > > [John.H.Gotway at m72a1 ~]$ ls -d
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*
> > > > | wc -w
> > > > 112
> > > >
> > > > I would guess this has something to do with timing. Perhaps if
this
> > > wrapper
> > > > is run prior to those files being written, then there's no
work to
> do.
> > I
> > > do
> > > > think the logic should be improved to better handle this case.
I
> > suspect
> > > > the bad return status is causing a problem. When there's no
match,
> you
> > > get
> > > > "No such file or directory" and a bad return status of 2:
> > > >
> > > > [John.H.Gotway at m72a1 ~]$ ls -d
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*BAD*
> > > >
> > > > ls: cannot access
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*BAD*:
> > > > No such file or directory
> > > > [John.H.Gotway at m72a1 ~]$ echo $?
> > > > 2
> > > >
> > > > George, we should enhance this Stat-Analysis wrapper to check
for no
> > > > matching files. And then apply more sane logic instead or a
useful
> log
> > > > message instead of exiting.
> > > >
> > > > Who's the right person to do that?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Thu, Jul 23, 2020 at 9:28 AM Minna Win via RT
<met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996 >
> > > > >
> > > > > Hello Edward,
> > > > >
> > > > > It looks like you have some issues with your StatAnalysis
runs.
> I've
> > > > > assigned this issue to John Halley Gotway.  Please allow a
few
> > business
> > > > > days for a full response.
> > > > >
> > > > > Regards,
> > > > > Minna
> > > > > ---------------
> > > > > Minna Win
> > > > > National Center for Atmospheric Research
> > > > > Developmental Testbed Center
> > > > > Phone: 303-497-8423
> > > > > Fax:   303-497-8401
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jul 23, 2020 at 9:18 AM Edward Strobach - NOAA
Affiliate
> via
> > > RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Thu Jul 23 09:18:05 2020: Request 95996 was acted upon.
> > > > > > Transaction: Ticket created by edward.strobach at noaa.gov
> > > > > >        Queue: met_help
> > > > > >      Subject: StatAnalysis works intermittently with CMAQ
> > meteorology
> > > > > files
> > > > > >        Owner: Nobody
> > > > > >   Requestors: edward.strobach at noaa.gov
> > > > > >       Status: new
> > > > > >  Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996
> > > > >
> > > > > >
> > > > > >
> > > > > > Good morning,
> > > > > >
> > > > > > I've been attempting to set up verification for
meteorology from
> > the
> > > > CMAQ
> > > > > > output.  In the past, I was able to successfully generate
files
> for
> > > the
> > > > > > PB2NC, PointStat, and StatAnalysis steps.  In confirming
the
> > success
> > > of
> > > > > > these steps, I then set up my configuration to handle
model runs
> > with
> > > > > files
> > > > > > that follow the same naming structure.  The PB2NC and
PointStat
> > files
> > > > are
> > > > > > always successful, while the StatAnalysis works
intermittently.
> > > > > Yesterday,
> > > > > > I was able to generate stat files from StatAnalysis for 2
out of
> > my 4
> > > > > model
> > > > > > runs.  Today, however, only one model run ended up being
> > successful.
> > > > I'm
> > > > > > not sure why it would work for one day, then the next day
it
> > doesn't
> > > > > work.
> > > > > > The error is always the same.
> > > > > >
> > > > > > 07/23 00:32:16.941 metplus (config_launcher.py:520) DEBUG:
> Setting
> > > > > [config]
> > > > > > MODEL1_REFERENCE_NAME to default value: PROD.
> > > > > > 07/23 00:32:16.942 metplus.StatAnalysis
> > > (stat_analysis_wrapper.py:235)
> > > > > > DEBUG: Items in these lists will be grouped together:
MODEL_LIST,
> > > > > > DESC_LIST, FCST_LEAD_LIST, OBS_LEAD_LIST,
FCST_INIT_HOUR_LIST,
> > > > > > OBS_VALID_HOUR_LIST, OBS_INIT_HOUR_LIST, FCST_VAR_LIST,
> > OBS_VAR_LIST,
> > > > > > FCST_UNITS_LIST, OBS_UNITS_LIST, FCST_LEVEL_LIST,
OBS_LEVEL_LIST,
> > > > > > VX_MASK_LIST, INTERP_MTHD_LIST, INTERP_PNTS_LIST,
> FCST_THRESH_LIST,
> > > > > > OBS_THRESH_LIST, COV_THRESH_LIST, ALPHA_LIST,
LINE_TYPE_LIST
> > > > > > 07/23 00:32:16.942 metplus.StatAnalysis
> > > (stat_analysis_wrapper.py:237)
> > > > > > DEBUG: Items in these lists will be looped over:
> > FCST_VALID_HOUR_LIST
> > > > > > 07/23 00:32:16.957 metplus.StatAnalysis (met_util.py:192)
ERROR:
> > > Fatal
> > > > > > error occurred
> > > > > > Traceback (most recent call last):
> > > > > >   File
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > > > line 172, in run_metplus
> > > > > >     loop_over_times_and_call(config, processes)
> > > > > >   File
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > > > line 834, in loop_over_times_and_call
> > > > > >     process.run_at_time(input_dict)
> > > > > >   File
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > > > > > line 1872, in run_at_time
> > > > > >     self.run_stat_analysis_job(date, date, loop_by)
> > > > > >   File
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > > > > > line 1288, in run_stat_analysis_job
> > > > > >     runtime_settings_dict)
> > > > > >   File
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > > > > > line 836, in get_lookin_dir
> > > > > >     subprocess.check_output('ls -d '+dir_path_filled,
shell=True)
> > > > > >   File
> > > > >
> "/usrx/local/dev/packages/python/3.6.3/lib/python3.6/subprocess.py",
> > > > > > line 336, in check_output
> > > > > >     **kwargs).stdout
> > > > > >   File
> > > > >
> "/usrx/local/dev/packages/python/3.6.3/lib/python3.6/subprocess.py",
> > > > > > line 418, in run
> > > > > >     output=stdout, stderr=stderr)
> > > > > > subprocess.CalledProcessError: Command 'ls -d
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*'
> > > > > > returned non-zero exit status 2.
> > > > > >
> > > > > > If the step worked, then the line that would follow
"....looped
> > over:
> > > > > > FCST_VALID_HOUR_LIST"  would be
> > > > > > 07/23 14:55:12.158 metplus.StatAnalysis
> > > (stat_analysis_wrapper.py:788)
> > > > > > DEBUG: Building dump_row filename from user template:
> > > > > > {valid?fmt=%Y%m%d}/PARA13_BC_{valid_beg?fmt=%Y%m%d%H}.stat
> > > > > >
> > > > > > If this step fails, it always happens at the first valid
time (in
> > > this
> > > > > case
> > > > > > 2020072000).  Following the file name template, I would
have
> > > something
> > > > > like
> > > > > > this:
> > > > > >
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 525632 Jul 23 14:55
> > > > > > PARA13_BC_2020072000.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 344951 Jul 23 14:55
> > > > > > PARA13_BC_2020072001.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 345726 Jul 23 14:55
> > > > > > PARA13_BC_2020072002.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 343403 Jul 23 14:55
> > > > > > PARA13_BC_2020072003.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 346222 Jul 23 14:55
> > > > > > PARA13_BC_2020072004.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 342297 Jul 23 14:55
> > > > > > PARA13_BC_2020072005.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 416853 Jul 23 14:55
> > > > > > PARA13_BC_2020072006.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 343406 Jul 23 14:55
> > > > > > PARA13_BC_2020072007.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 346832 Jul 23 14:55
> > > > > > PARA13_BC_2020072008.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 342291 Jul 23 14:55
> > > > > > PARA13_BC_2020072009.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 347004 Jul 23 14:55
> > > > > > PARA13_BC_2020072010.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 358938 Jul 23 14:55
> > > > > > PARA13_BC_2020072011.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 519089 Jul 23 14:55
> > > > > > PARA13_BC_2020072012.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 414343 Jul 23 14:55
> > > > > > PARA13_BC_2020072013.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 437188 Jul 23 14:55
> > > > > > PARA13_BC_2020072014.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 435365 Jul 23 14:56
> > > > > > PARA13_BC_2020072015.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 433295 Jul 23 14:56
> > > > > > PARA13_BC_2020072016.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 433698 Jul 23 14:56
> > > > > > PARA13_BC_2020072017.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 508369 Jul 23 14:56
> > > > > > PARA13_BC_2020072018.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 447083 Jul 23 14:56
> > > > > > PARA13_BC_2020072019.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 441350 Jul 23 14:56
> > > > > > PARA13_BC_2020072020.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 437671 Jul 23 14:56
> > > > > > PARA13_BC_2020072021.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 416085 Jul 23 14:56
> > > > > > PARA13_BC_2020072022.stat
> > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 444785 Jul 23 14:56
> > > > > > PARA13_BC_2020072023.stat
> > > > > >
> > > > > > You can see these are indeed populated, and that these are
the
> only
> > > > ones
> > > > > > generated.  If all files were generated, then I would have
> > > > PROD_BC_*stat,
> > > > > > PROD_*stat, and PARA13_*stat.  The day before both PROD
and
> > PARA13_BC
> > > > > > worked.  I've also had cases where all model runs were
> successful.
> > > I'm
> > > > > not
> > > > > > sure why this would be the case.  I don't see anything
> indicating a
> > > > > problem
> > > > > > within the point stat files themselves..
> > > > > > --
> > > > > > Edward Strobach
> > > > > > EMC/NCEP/NWS/
> > > > > > IMSG Contractor
> > > > > > Cubicle#: 2029
> > > > > > 301-683-3717
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
>

--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

------------------------------------------------
Subject: StatAnalysis works intermittently with CMAQ meteorology files
From: George McCabe
Time: Thu Jul 23 15:40:51 2020

I confirmed that the wrapper will still crash if the wildcard
expression
did not match any files. I created a GitHub issue to make this fix:

https://github.com/NCAR/METplus/issues/538

It should be a quick fix to implement and test, so it should be
available
in the 3.1 release. We may also cut a 3.1-beta3 release for testing
purposes and we could provide you with that version so you can test
that
your case is handled correctly.

Thanks,
George

On Thu, Jul 23, 2020 at 3:06 PM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996 >
>
> Great, I'll try to find the newest location and give that a try.
>
> On Thu, Jul 23, 2020 at 4:26 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Hopefully, you can just point to the updated version of METplus
and give
> > that a shot.
> >
> > John
> >
> > On Thu, Jul 23, 2020 at 12:59 PM Edward Strobach - NOAA Affiliate
via RT
> <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996 >
> > >
> > > I've done the second point in the past, which has helped most of
the
> time
> > > since the error messages tend to be intuitive.  Do I need to
reclone
> > > something or should I just specify a new version in my file
system?  I
> > > haven't had to do this before.
> > >
> > > On Thu, Jul 23, 2020 at 2:56 PM George McCabe via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Edward (and John),
> > > >
> > > > I recently completed a huge overhaul of the StatAnalysis
wrapper. It
> > > looks
> > > > like you are using an older version of METplus that doesn't
include
> > these
> > > > changes. It is available in the METplus 3.1-beta2 release.
These
> > changes
> > > > should hopefully catch something misconfigured earlier in
execution
> to
> > > > prevent a crash like this and instead provide a nicer error
message
> > > letting
> > > > you know what went wrong. Either way, I agree that we should
add
> error
> > > > checking to prevent this crash. I will try to recreate this
locally.
> > > >
> > > > I would try two things:
> > > >
> > > > 1) Running this case with the beta2 release to see if the
crash still
> > > > occurs
> > > > 2) Search for "ERROR" earlier in execution to see if something
failed
> > > > earlier in execution that may explain why you don't have any
files in
> > > that
> > > > directory.
> > > >
> > > > - George
> > > >
> > > > On Thu, Jul 23, 2020 at 12:33 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996 >
> > > > >
> > > > > Hi Ed (and George),
> > > > >
> > > > > I see you're getting errors from the METplus Stat-Analysis
wrapper.
> > > Based
> > > > > on the log message you sent, it's erroring out when listing
those
> > > files:
> > > > >
> > > > > ls -d
> > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*
> > > > >
> > > > > But when I list them now, I see 112 of them:
> > > > >
> > > > > [John.H.Gotway at m72a1 ~]$ ls -d
> > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*
> > > > > | wc -w
> > > > > 112
> > > > >
> > > > > I would guess this has something to do with timing. Perhaps
if this
> > > > wrapper
> > > > > is run prior to those files being written, then there's no
work to
> > do.
> > > I
> > > > do
> > > > > think the logic should be improved to better handle this
case. I
> > > suspect
> > > > > the bad return status is causing a problem. When there's no
match,
> > you
> > > > get
> > > > > "No such file or directory" and a bad return status of 2:
> > > > >
> > > > > [John.H.Gotway at m72a1 ~]$ ls -d
> > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*BAD*
> > > > >
> > > > > ls: cannot access
> > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*BAD*:
> > > > > No such file or directory
> > > > > [John.H.Gotway at m72a1 ~]$ echo $?
> > > > > 2
> > > > >
> > > > > George, we should enhance this Stat-Analysis wrapper to
check for
> no
> > > > > matching files. And then apply more sane logic instead or a
useful
> > log
> > > > > message instead of exiting.
> > > > >
> > > > > Who's the right person to do that?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Thu, Jul 23, 2020 at 9:28 AM Minna Win via RT <
> met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996 >
> > > > > >
> > > > > > Hello Edward,
> > > > > >
> > > > > > It looks like you have some issues with your StatAnalysis
runs.
> > I've
> > > > > > assigned this issue to John Halley Gotway.  Please allow a
few
> > > business
> > > > > > days for a full response.
> > > > > >
> > > > > > Regards,
> > > > > > Minna
> > > > > > ---------------
> > > > > > Minna Win
> > > > > > National Center for Atmospheric Research
> > > > > > Developmental Testbed Center
> > > > > > Phone: 303-497-8423
> > > > > > Fax:   303-497-8401
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Jul 23, 2020 at 9:18 AM Edward Strobach - NOAA
Affiliate
> > via
> > > > RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > Thu Jul 23 09:18:05 2020: Request 95996 was acted upon.
> > > > > > > Transaction: Ticket created by edward.strobach at noaa.gov
> > > > > > >        Queue: met_help
> > > > > > >      Subject: StatAnalysis works intermittently with
CMAQ
> > > meteorology
> > > > > > files
> > > > > > >        Owner: Nobody
> > > > > > >   Requestors: edward.strobach at noaa.gov
> > > > > > >       Status: new
> > > > > > >  Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Good morning,
> > > > > > >
> > > > > > > I've been attempting to set up verification for
meteorology
> from
> > > the
> > > > > CMAQ
> > > > > > > output.  In the past, I was able to successfully
generate files
> > for
> > > > the
> > > > > > > PB2NC, PointStat, and StatAnalysis steps.  In confirming
the
> > > success
> > > > of
> > > > > > > these steps, I then set up my configuration to handle
model
> runs
> > > with
> > > > > > files
> > > > > > > that follow the same naming structure.  The PB2NC and
PointStat
> > > files
> > > > > are
> > > > > > > always successful, while the StatAnalysis works
intermittently.
> > > > > > Yesterday,
> > > > > > > I was able to generate stat files from StatAnalysis for
2 out
> of
> > > my 4
> > > > > > model
> > > > > > > runs.  Today, however, only one model run ended up being
> > > successful.
> > > > > I'm
> > > > > > > not sure why it would work for one day, then the next
day it
> > > doesn't
> > > > > > work.
> > > > > > > The error is always the same.
> > > > > > >
> > > > > > > 07/23 00:32:16.941 metplus (config_launcher.py:520)
DEBUG:
> > Setting
> > > > > > [config]
> > > > > > > MODEL1_REFERENCE_NAME to default value: PROD.
> > > > > > > 07/23 00:32:16.942 metplus.StatAnalysis
> > > > (stat_analysis_wrapper.py:235)
> > > > > > > DEBUG: Items in these lists will be grouped together:
> MODEL_LIST,
> > > > > > > DESC_LIST, FCST_LEAD_LIST, OBS_LEAD_LIST,
FCST_INIT_HOUR_LIST,
> > > > > > > OBS_VALID_HOUR_LIST, OBS_INIT_HOUR_LIST, FCST_VAR_LIST,
> > > OBS_VAR_LIST,
> > > > > > > FCST_UNITS_LIST, OBS_UNITS_LIST, FCST_LEVEL_LIST,
> OBS_LEVEL_LIST,
> > > > > > > VX_MASK_LIST, INTERP_MTHD_LIST, INTERP_PNTS_LIST,
> > FCST_THRESH_LIST,
> > > > > > > OBS_THRESH_LIST, COV_THRESH_LIST, ALPHA_LIST,
LINE_TYPE_LIST
> > > > > > > 07/23 00:32:16.942 metplus.StatAnalysis
> > > > (stat_analysis_wrapper.py:237)
> > > > > > > DEBUG: Items in these lists will be looped over:
> > > FCST_VALID_HOUR_LIST
> > > > > > > 07/23 00:32:16.957 metplus.StatAnalysis
(met_util.py:192)
> ERROR:
> > > > Fatal
> > > > > > > error occurred
> > > > > > > Traceback (most recent call last):
> > > > > > >   File
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > > > > line 172, in run_metplus
> > > > > > >     loop_over_times_and_call(config, processes)
> > > > > > >   File
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > > > > line 834, in loop_over_times_and_call
> > > > > > >     process.run_at_time(input_dict)
> > > > > > >   File
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > > > > > > line 1872, in run_at_time
> > > > > > >     self.run_stat_analysis_job(date, date, loop_by)
> > > > > > >   File
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > > > > > > line 1288, in run_stat_analysis_job
> > > > > > >     runtime_settings_dict)
> > > > > > >   File
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > > > > > > line 836, in get_lookin_dir
> > > > > > >     subprocess.check_output('ls -d '+dir_path_filled,
> shell=True)
> > > > > > >   File
> > > > > >
> >
"/usrx/local/dev/packages/python/3.6.3/lib/python3.6/subprocess.py",
> > > > > > > line 336, in check_output
> > > > > > >     **kwargs).stdout
> > > > > > >   File
> > > > > >
> >
"/usrx/local/dev/packages/python/3.6.3/lib/python3.6/subprocess.py",
> > > > > > > line 418, in run
> > > > > > >     output=stdout, stderr=stderr)
> > > > > > > subprocess.CalledProcessError: Command 'ls -d
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*'
> > > > > > > returned non-zero exit status 2.
> > > > > > >
> > > > > > > If the step worked, then the line that would follow
"....looped
> > > over:
> > > > > > > FCST_VALID_HOUR_LIST"  would be
> > > > > > > 07/23 14:55:12.158 metplus.StatAnalysis
> > > > (stat_analysis_wrapper.py:788)
> > > > > > > DEBUG: Building dump_row filename from user template:
> > > > > > >
{valid?fmt=%Y%m%d}/PARA13_BC_{valid_beg?fmt=%Y%m%d%H}.stat
> > > > > > >
> > > > > > > If this step fails, it always happens at the first valid
time
> (in
> > > > this
> > > > > > case
> > > > > > > 2020072000).  Following the file name template, I would
have
> > > > something
> > > > > > like
> > > > > > > this:
> > > > > > >
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 525632 Jul 23
14:55
> > > > > > > PARA13_BC_2020072000.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 344951 Jul 23
14:55
> > > > > > > PARA13_BC_2020072001.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 345726 Jul 23
14:55
> > > > > > > PARA13_BC_2020072002.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 343403 Jul 23
14:55
> > > > > > > PARA13_BC_2020072003.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 346222 Jul 23
14:55
> > > > > > > PARA13_BC_2020072004.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 342297 Jul 23
14:55
> > > > > > > PARA13_BC_2020072005.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 416853 Jul 23
14:55
> > > > > > > PARA13_BC_2020072006.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 343406 Jul 23
14:55
> > > > > > > PARA13_BC_2020072007.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 346832 Jul 23
14:55
> > > > > > > PARA13_BC_2020072008.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 342291 Jul 23
14:55
> > > > > > > PARA13_BC_2020072009.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 347004 Jul 23
14:55
> > > > > > > PARA13_BC_2020072010.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 358938 Jul 23
14:55
> > > > > > > PARA13_BC_2020072011.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 519089 Jul 23
14:55
> > > > > > > PARA13_BC_2020072012.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 414343 Jul 23
14:55
> > > > > > > PARA13_BC_2020072013.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 437188 Jul 23
14:55
> > > > > > > PARA13_BC_2020072014.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 435365 Jul 23
14:56
> > > > > > > PARA13_BC_2020072015.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 433295 Jul 23
14:56
> > > > > > > PARA13_BC_2020072016.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 433698 Jul 23
14:56
> > > > > > > PARA13_BC_2020072017.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 508369 Jul 23
14:56
> > > > > > > PARA13_BC_2020072018.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 447083 Jul 23
14:56
> > > > > > > PARA13_BC_2020072019.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 441350 Jul 23
14:56
> > > > > > > PARA13_BC_2020072020.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 437671 Jul 23
14:56
> > > > > > > PARA13_BC_2020072021.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 416085 Jul 23
14:56
> > > > > > > PARA13_BC_2020072022.stat
> > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 444785 Jul 23
14:56
> > > > > > > PARA13_BC_2020072023.stat
> > > > > > >
> > > > > > > You can see these are indeed populated, and that these
are the
> > only
> > > > > ones
> > > > > > > generated.  If all files were generated, then I would
have
> > > > > PROD_BC_*stat,
> > > > > > > PROD_*stat, and PARA13_*stat.  The day before both PROD
and
> > > PARA13_BC
> > > > > > > worked.  I've also had cases where all model runs were
> > successful.
> > > > I'm
> > > > > > not
> > > > > > > sure why this would be the case.  I don't see anything
> > indicating a
> > > > > > problem
> > > > > > > within the point stat files themselves..
> > > > > > > --
> > > > > > > Edward Strobach
> > > > > > > EMC/NCEP/NWS/
> > > > > > > IMSG Contractor
> > > > > > > Cubicle#: 2029
> > > > > > > 301-683-3717
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> > >
> >
> >
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: StatAnalysis works intermittently with CMAQ meteorology files
From: Edward Strobach - NOAA Affiliate
Time: Thu Jul 23 15:44:43 2020

Interesting.  I went ahead and tried and got the same results.  Thanks
for
letting me know

On Thu, Jul 23, 2020 at 5:41 PM George McCabe via RT
<met_help at ucar.edu>
wrote:

> I confirmed that the wrapper will still crash if the wildcard
expression
> did not match any files. I created a GitHub issue to make this fix:
>
> https://github.com/NCAR/METplus/issues/538
>
> It should be a quick fix to implement and test, so it should be
available
> in the 3.1 release. We may also cut a 3.1-beta3 release for testing
> purposes and we could provide you with that version so you can test
that
> your case is handled correctly.
>
> Thanks,
> George
>
> On Thu, Jul 23, 2020 at 3:06 PM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996 >
> >
> > Great, I'll try to find the newest location and give that a try.
> >
> > On Thu, Jul 23, 2020 at 4:26 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Hopefully, you can just point to the updated version of METplus
and
> give
> > > that a shot.
> > >
> > > John
> > >
> > > On Thu, Jul 23, 2020 at 12:59 PM Edward Strobach - NOAA
Affiliate via
> RT
> > <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996
>
> > > >
> > > > I've done the second point in the past, which has helped most
of the
> > time
> > > > since the error messages tend to be intuitive.  Do I need to
reclone
> > > > something or should I just specify a new version in my file
system?
> I
> > > > haven't had to do this before.
> > > >
> > > > On Thu, Jul 23, 2020 at 2:56 PM George McCabe via RT <
> > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Edward (and John),
> > > > >
> > > > > I recently completed a huge overhaul of the StatAnalysis
wrapper.
> It
> > > > looks
> > > > > like you are using an older version of METplus that doesn't
include
> > > these
> > > > > changes. It is available in the METplus 3.1-beta2 release.
These
> > > changes
> > > > > should hopefully catch something misconfigured earlier in
execution
> > to
> > > > > prevent a crash like this and instead provide a nicer error
message
> > > > letting
> > > > > you know what went wrong. Either way, I agree that we should
add
> > error
> > > > > checking to prevent this crash. I will try to recreate this
> locally.
> > > > >
> > > > > I would try two things:
> > > > >
> > > > > 1) Running this case with the beta2 release to see if the
crash
> still
> > > > > occurs
> > > > > 2) Search for "ERROR" earlier in execution to see if
something
> failed
> > > > > earlier in execution that may explain why you don't have any
files
> in
> > > > that
> > > > > directory.
> > > > >
> > > > > - George
> > > > >
> > > > > On Thu, Jul 23, 2020 at 12:33 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996 >
> > > > > >
> > > > > > Hi Ed (and George),
> > > > > >
> > > > > > I see you're getting errors from the METplus Stat-Analysis
> wrapper.
> > > > Based
> > > > > > on the log message you sent, it's erroring out when
listing those
> > > > files:
> > > > > >
> > > > > > ls -d
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*
> > > > > >
> > > > > > But when I list them now, I see 112 of them:
> > > > > >
> > > > > > [John.H.Gotway at m72a1 ~]$ ls -d
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*
> > > > > > | wc -w
> > > > > > 112
> > > > > >
> > > > > > I would guess this has something to do with timing.
Perhaps if
> this
> > > > > wrapper
> > > > > > is run prior to those files being written, then there's no
work
> to
> > > do.
> > > > I
> > > > > do
> > > > > > think the logic should be improved to better handle this
case. I
> > > > suspect
> > > > > > the bad return status is causing a problem. When there's
no
> match,
> > > you
> > > > > get
> > > > > > "No such file or directory" and a bad return status of 2:
> > > > > >
> > > > > > [John.H.Gotway at m72a1 ~]$ ls -d
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*BAD*
> > > > > >
> > > > > > ls: cannot access
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*BAD*:
> > > > > > No such file or directory
> > > > > > [John.H.Gotway at m72a1 ~]$ echo $?
> > > > > > 2
> > > > > >
> > > > > > George, we should enhance this Stat-Analysis wrapper to
check for
> > no
> > > > > > matching files. And then apply more sane logic instead or
a
> useful
> > > log
> > > > > > message instead of exiting.
> > > > > >
> > > > > > Who's the right person to do that?
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Thu, Jul 23, 2020 at 9:28 AM Minna Win via RT <
> > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996
> >
> > > > > > >
> > > > > > > Hello Edward,
> > > > > > >
> > > > > > > It looks like you have some issues with your
StatAnalysis runs.
> > > I've
> > > > > > > assigned this issue to John Halley Gotway.  Please allow
a few
> > > > business
> > > > > > > days for a full response.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Minna
> > > > > > > ---------------
> > > > > > > Minna Win
> > > > > > > National Center for Atmospheric Research
> > > > > > > Developmental Testbed Center
> > > > > > > Phone: 303-497-8423
> > > > > > > Fax:   303-497-8401
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jul 23, 2020 at 9:18 AM Edward Strobach - NOAA
> Affiliate
> > > via
> > > > > RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Thu Jul 23 09:18:05 2020: Request 95996 was acted
upon.
> > > > > > > > Transaction: Ticket created by
edward.strobach at noaa.gov
> > > > > > > >        Queue: met_help
> > > > > > > >      Subject: StatAnalysis works intermittently with
CMAQ
> > > > meteorology
> > > > > > > files
> > > > > > > >        Owner: Nobody
> > > > > > > >   Requestors: edward.strobach at noaa.gov
> > > > > > > >       Status: new
> > > > > > > >  Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95996
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Good morning,
> > > > > > > >
> > > > > > > > I've been attempting to set up verification for
meteorology
> > from
> > > > the
> > > > > > CMAQ
> > > > > > > > output.  In the past, I was able to successfully
generate
> files
> > > for
> > > > > the
> > > > > > > > PB2NC, PointStat, and StatAnalysis steps.  In
confirming the
> > > > success
> > > > > of
> > > > > > > > these steps, I then set up my configuration to handle
model
> > runs
> > > > with
> > > > > > > files
> > > > > > > > that follow the same naming structure.  The PB2NC and
> PointStat
> > > > files
> > > > > > are
> > > > > > > > always successful, while the StatAnalysis works
> intermittently.
> > > > > > > Yesterday,
> > > > > > > > I was able to generate stat files from StatAnalysis
for 2 out
> > of
> > > > my 4
> > > > > > > model
> > > > > > > > runs.  Today, however, only one model run ended up
being
> > > > successful.
> > > > > > I'm
> > > > > > > > not sure why it would work for one day, then the next
day it
> > > > doesn't
> > > > > > > work.
> > > > > > > > The error is always the same.
> > > > > > > >
> > > > > > > > 07/23 00:32:16.941 metplus (config_launcher.py:520)
DEBUG:
> > > Setting
> > > > > > > [config]
> > > > > > > > MODEL1_REFERENCE_NAME to default value: PROD.
> > > > > > > > 07/23 00:32:16.942 metplus.StatAnalysis
> > > > > (stat_analysis_wrapper.py:235)
> > > > > > > > DEBUG: Items in these lists will be grouped together:
> > MODEL_LIST,
> > > > > > > > DESC_LIST, FCST_LEAD_LIST, OBS_LEAD_LIST,
> FCST_INIT_HOUR_LIST,
> > > > > > > > OBS_VALID_HOUR_LIST, OBS_INIT_HOUR_LIST,
FCST_VAR_LIST,
> > > > OBS_VAR_LIST,
> > > > > > > > FCST_UNITS_LIST, OBS_UNITS_LIST, FCST_LEVEL_LIST,
> > OBS_LEVEL_LIST,
> > > > > > > > VX_MASK_LIST, INTERP_MTHD_LIST, INTERP_PNTS_LIST,
> > > FCST_THRESH_LIST,
> > > > > > > > OBS_THRESH_LIST, COV_THRESH_LIST, ALPHA_LIST,
LINE_TYPE_LIST
> > > > > > > > 07/23 00:32:16.942 metplus.StatAnalysis
> > > > > (stat_analysis_wrapper.py:237)
> > > > > > > > DEBUG: Items in these lists will be looped over:
> > > > FCST_VALID_HOUR_LIST
> > > > > > > > 07/23 00:32:16.957 metplus.StatAnalysis
(met_util.py:192)
> > ERROR:
> > > > > Fatal
> > > > > > > > error occurred
> > > > > > > > Traceback (most recent call last):
> > > > > > > >   File
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > > > > > line 172, in run_metplus
> > > > > > > >     loop_over_times_and_call(config, processes)
> > > > > > > >   File
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/met_util.py",
> > > > > > > > line 834, in loop_over_times_and_call
> > > > > > > >     process.run_at_time(input_dict)
> > > > > > > >   File
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > > > > > > > line 1872, in run_at_time
> > > > > > > >     self.run_stat_analysis_job(date, date, loop_by)
> > > > > > > >   File
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > > > > > > > line 1288, in run_stat_analysis_job
> > > > > > > >     runtime_settings_dict)
> > > > > > > >   File
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
"/gpfs/dell2/emc/modeling/save/Edward.Strobach/MetPlus/METplus/ush/stat_analysis_wrapper.py",
> > > > > > > > line 836, in get_lookin_dir
> > > > > > > >     subprocess.check_output('ls -d '+dir_path_filled,
> > shell=True)
> > > > > > > >   File
> > > > > > >
> > >
"/usrx/local/dev/packages/python/3.6.3/lib/python3.6/subprocess.py",
> > > > > > > > line 336, in check_output
> > > > > > > >     **kwargs).stdout
> > > > > > > >   File
> > > > > > >
> > >
"/usrx/local/dev/packages/python/3.6.3/lib/python3.6/subprocess.py",
> > > > > > > > line 418, in run
> > > > > > > >     output=stdout, stderr=stderr)
> > > > > > > > subprocess.CalledProcessError: Command 'ls -d
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ_Meteorology/aqm/stat/prod/*20200720*'
> > > > > > > > returned non-zero exit status 2.
> > > > > > > >
> > > > > > > > If the step worked, then the line that would follow
> "....looped
> > > > over:
> > > > > > > > FCST_VALID_HOUR_LIST"  would be
> > > > > > > > 07/23 14:55:12.158 metplus.StatAnalysis
> > > > > (stat_analysis_wrapper.py:788)
> > > > > > > > DEBUG: Building dump_row filename from user template:
> > > > > > > >
{valid?fmt=%Y%m%d}/PARA13_BC_{valid_beg?fmt=%Y%m%d%H}.stat
> > > > > > > >
> > > > > > > > If this step fails, it always happens at the first
valid time
> > (in
> > > > > this
> > > > > > > case
> > > > > > > > 2020072000).  Following the file name template, I
would have
> > > > > something
> > > > > > > like
> > > > > > > > this:
> > > > > > > >
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 525632 Jul 23
14:55
> > > > > > > > PARA13_BC_2020072000.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 344951 Jul 23
14:55
> > > > > > > > PARA13_BC_2020072001.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 345726 Jul 23
14:55
> > > > > > > > PARA13_BC_2020072002.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 343403 Jul 23
14:55
> > > > > > > > PARA13_BC_2020072003.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 346222 Jul 23
14:55
> > > > > > > > PARA13_BC_2020072004.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 342297 Jul 23
14:55
> > > > > > > > PARA13_BC_2020072005.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 416853 Jul 23
14:55
> > > > > > > > PARA13_BC_2020072006.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 343406 Jul 23
14:55
> > > > > > > > PARA13_BC_2020072007.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 346832 Jul 23
14:55
> > > > > > > > PARA13_BC_2020072008.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 342291 Jul 23
14:55
> > > > > > > > PARA13_BC_2020072009.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 347004 Jul 23
14:55
> > > > > > > > PARA13_BC_2020072010.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 358938 Jul 23
14:55
> > > > > > > > PARA13_BC_2020072011.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 519089 Jul 23
14:55
> > > > > > > > PARA13_BC_2020072012.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 414343 Jul 23
14:55
> > > > > > > > PARA13_BC_2020072013.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 437188 Jul 23
14:55
> > > > > > > > PARA13_BC_2020072014.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 435365 Jul 23
14:56
> > > > > > > > PARA13_BC_2020072015.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 433295 Jul 23
14:56
> > > > > > > > PARA13_BC_2020072016.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 433698 Jul 23
14:56
> > > > > > > > PARA13_BC_2020072017.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 508369 Jul 23
14:56
> > > > > > > > PARA13_BC_2020072018.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 447083 Jul 23
14:56
> > > > > > > > PARA13_BC_2020072019.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 441350 Jul 23
14:56
> > > > > > > > PARA13_BC_2020072020.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 437671 Jul 23
14:56
> > > > > > > > PARA13_BC_2020072021.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 416085 Jul 23
14:56
> > > > > > > > PARA13_BC_2020072022.stat
> > > > > > > > -rw-r--r-- 1 Edward.Strobach emcmodel 444785 Jul 23
14:56
> > > > > > > > PARA13_BC_2020072023.stat
> > > > > > > >
> > > > > > > > You can see these are indeed populated, and that these
are
> the
> > > only
> > > > > > ones
> > > > > > > > generated.  If all files were generated, then I would
have
> > > > > > PROD_BC_*stat,
> > > > > > > > PROD_*stat, and PARA13_*stat.  The day before both
PROD and
> > > > PARA13_BC
> > > > > > > > worked.  I've also had cases where all model runs were
> > > successful.
> > > > > I'm
> > > > > > > not
> > > > > > > > sure why this would be the case.  I don't see anything
> > > indicating a
> > > > > > > problem
> > > > > > > > within the point stat files themselves..
> > > > > > > > --
> > > > > > > > Edward Strobach
> > > > > > > > EMC/NCEP/NWS/
> > > > > > > > IMSG Contractor
> > > > > > > > Cubicle#: 2029
> > > > > > > > 301-683-3717
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > > --
> > > > Edward Strobach
> > > > EMC/NCEP/NWS/
> > > > IMSG Contractor
> > > > Cubicle#: 2029
> > > > 301-683-3717
> > > >
> > > >
> > >
> > >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717

------------------------------------------------