[Met_help] [rt.rap.ucar.edu #99563] History for zero matched pairs when processing the AQ inline data

Thu Apr 15 12:28:54 MDT 2021

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Good morning,

I'm attempting to process the data from here: "
*/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/aqm.20190810*
"

The files follow the "aqm.t12z.metcro2d.HH.nc" format and are written in a
format that METplus can handle.  The processing steps include pb2nc,
pointstat, and statanalysis.  The issue occurs at pointstat.  The pb2nc
step runs just fine.

Looking in the netcdf files that I created above confirms that the fields I
need are there:

*>>> Dataset('aqm.t12z.metcro2d.02.nc
<http://aqm.t12z.metcro2d.02.nc>')<class 'netCDF4._netCDF4.Dataset'>root
group (NETCDF4 data model, file format HDF5):    FileOrigins: NA
MET_version: V9.1    hemisphere: N    Projection: Lambert Conformal    nx:
191    ny: 97 grid_points    dimensions(sizes): lat(97), lon(191)
variables(dimensions): float32 LAT(lat,lon), float32 LON(lat,lon), float32
WDIR10(lat,lon), float32 WSPD10(lat,lon), float32 TMP(lat,lon), float32
DPT(lat,lon), float32 OZCON(lat,lon), float32 PMTF(lat,lon)    groups: *

You can also see that the meta data is not overly descriptive as in the
past.  I'm not sure if that is problematic or not.  I do know that whatever
is not specified in the file is filled by -9999 or -999 values.

Looking at the log file I find the following for every region, for both
PM2.5 and ozone:

*DEBUG 2: Processing OZCONA8 versus COPO/A8, for observation type AIRNOW,
over region WEST, for interpolation method BILIN(4), using 0 matched
pairs.DEBUG 3: Number of matched pairs   = 0DEBUG 3: Observations processed
   = 59531DEBUG 3: Rejected: station id      = 0DEBUG 3: Rejected: obs type
       = 0DEBUG 3: Rejected: valid time      = 57003DEBUG 3: Rejected: bad
obs value   = 0DEBUG 3: Rejected: off the grid    = 2528DEBUG 3: Rejected:
topography      = 0DEBUG 3: Rejected: level mismatch  = 0DEBUG 3: Rejected:
quality marker  = 0DEBUG 3: Rejected: message type    = 0DEBUG 3: Rejected:
masking region  = 0DEBUG 3: Rejected: bad fcst value  = 0DEBUG 3: Rejected:
bad climo mean  = 0DEBUG 3: Rejected: bad climo stdev = 0DEBUG 3: Rejected:
duplicates      = 0*

There's no errors to report, rather there are just no matched pairs.  Since
there are only 24 hours being processed for this data, then only 24 hours
worth of data are being added to the prebufr*nc files.  You can also see in
the log file that the appropriate files are being selected according to
date:

*DEBUG 1: Forecast File:
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/aqm.20190810/aqm.t12z.metcro2d.10.nc
<http://aqm.t12z.metcro2d.10.nc>DEBUG 1: Observation File:
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r112_v1/aq/prepbufr.aqm.20190810.nc
<http://prepbufr.aqm.20190810.nc>DEBUG 7: check_nc_data_2d() count:
valid=18527, zero=0, missing=0, 18527 == 18527*

I'm having a hard time figuring out why there are no matched pairs.  A
couple of other things to note related to an earlier statement about meta
data..

Recall that I said the files weren't overly descriptive.  You can see that
the -9999 is being filed for many of the definitions that are not
specifically being set in the created netcdf files:

*DEBUG 4: Lambert Conformal Grid Data:DEBUG 4:    hemisphere: NDEBUG 4:
scale_lat_1: -9999DEBUG 4:   scale_lat_2: -9999DEBUG 4:       lat_pin:
-9999DEBUG 4:       lon_pin: 9999DEBUG 4:         x_pin: -9999DEBUG 4:
    y_pin: -9999DEBUG 4:    lon_orient: 9999DEBUG 4:          d_km:
-9999DEBUG 4:          r_km: -9999DEBUG 4:            nx: 191DEBUG 4:
     ny: 97DEBUG 4:     so2_angle: 0DEBUG 4:*

I'm not sure if that is playing a role here.  Printing out the ozone field
from the netcdf file you can see that other information like long_name,
level, etc. are given.

*>>> Dataset('aqm.t12z.metcro2d.02.nc
<http://aqm.t12z.metcro2d.02.nc>').variables['OZCON']<class
'netCDF4._netCDF4.Variable'>float32 OZCON(lat, lon)    long_name: OZCON
standard_name: Surface Ozone    units: ppb    level: A8    init_time:
20190810_120000    init_time_ut: 1565438400    valid_time: 20190810_140000
  valid_time_ut: 1565445600unlimited dimensions: current shape = (97,
191)filling on, default _FillValue of 9.969209968386869e+36 used*

Is there something I'm not thinking about here?  I can provide more details
if needed.  Hopefully this description is a good start.  I do recall having
a similar issue in the past but couldn't locate the exchange going over
this problem.

Edward Strobach
EMC/NCEP/NWS/ IMSG
Coding Logic:  if, then, else

----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: zero matched pairs when processing the AQ inline data
From: Julie Prestopnik
Time: Tue Apr 13 12:19:57 2021

Hi Edward.

I see that you are having trouble getting matched pairs from Point-
Stat.

Let's start by taking a look at exactly where the log messages are
pointing
us. Looking at one of your log files:
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210413144310

Looking at the first call of point_stat, I see the following output
indicating that all of the 21,468 observations were rejected due to
the
valid_time, which is why there are no matched pairs for this case:

> DEBUG 3: Number of matched pairs   = 0
> DEBUG 3: Observations processed    = 21468
> DEBUG 3: Rejected: station id      = 0
> DEBUG 3: Rejected: obs type        = 0
> DEBUG 3: Rejected: valid time      = 21468
> DEBUG 3: Rejected: bad obs value   = 0
> DEBUG 3: Rejected: off the grid    = 0
> DEBUG 3: Rejected: topography      = 0
> DEBUG 3: Rejected: level mismatch  = 0
> DEBUG 3: Rejected: quality marker  = 0
> DEBUG 3: Rejected: message type    = 0
> DEBUG 3: Rejected: masking region  = 0
> DEBUG 3: Rejected: bad fcst value  = 0
> DEBUG 3: Rejected: bad climo mean  = 0
> DEBUG 3: Rejected: bad climo stdev = 0
> DEBUG 3: Rejected: duplicates      = 0

This is similar to the situation in the Rejected values that you had
pasted
in, except some of those observations were rejected because they were
off
of the grid, which is why there are no matched pairs for this case:

> DEBUG 3: Observations processed = 59531
> DEBUG 3: Rejected: valid time      = 57003
> DEBUG 3: Rejected: off the grid    = 2528

I wasn't sure which log file that came from, so I just used your most
recent log file to get the example above.

You'll want to make sure that when you run Point-Stat, the forecast
valid time matches up correctly with the valid time of point
observations.
I may not be the best person to help with that, but I did find
something
that may be a part of the problem.

Here is the call to point_stat for the case with the 21,468
observations
rejected due to valid time:
point_stat -v 7 \
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/\
aqm.20190809/aqm.t12z.metcro2d.12.nc \
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r112_v1/pm/
prepbufr.pm.20190810.nc
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
\
-outdir
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/stat/pm/r112_v1

I can see that
the
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
configuration file is being used.

In that configuration file I see:

> obs_window = {
>    beg = 0;
>    end = 0;
> }

whereas in many of the other Point-Stat configuration files in that
directory I see:

> obs_window = {
>    beg = ${OBS_WINDOW_BEGIN};
>    end = ${OBS_WINDOW_END};
> }

In
your
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
file, I see the following set:

> OBS_WINDOW_BEGIN = -2700
> OBS_WINDOW_END = 2700

and I was wondering if you are intending for METplus to pass on these
values in the run to Point-Stat?  If so, you would need to change the
values of zero to ${OBS_WINDOW_BEGIN} and ${OBS_WINDOW_END},
respectively.
Perhaps that is the cause of the zero matched pairs due to valid time?

I hope this helps!

Julie

On Tue, Apr 13, 2021 at 9:27 AM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:

>
> Tue Apr 13 09:27:33 2021: Request 99563 was acted upon.
> Transaction: Ticket created by edward.strobach at noaa.gov
>        Queue: met_help
>      Subject: zero matched pairs when processing the AQ inline data
>        Owner: Nobody
>   Requestors: edward.strobach at noaa.gov
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99563 >
>
>
> Good morning,
>
> I'm attempting to process the data from here: "
>
>
*/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/aqm.20190810*
> "
>
> The files follow the "aqm.t12z.metcro2d.HH.nc" format and are
written in a
> format that METplus can handle.  The processing steps include pb2nc,
> pointstat, and statanalysis.  The issue occurs at pointstat.  The
pb2nc
> step runs just fine.
>
> Looking in the netcdf files that I created above confirms that the
fields I
> need are there:
>
>
>
>
>
>
>
>
>
>
>
>
> *>>> Dataset('aqm.t12z.metcro2d.02.nc
> <http://aqm.t12z.metcro2d.02.nc>')<class
'netCDF4._netCDF4.Dataset'>root
> group (NETCDF4 data model, file format HDF5):    FileOrigins: NA
> MET_version: V9.1    hemisphere: N    Projection: Lambert Conformal
nx:
> 191    ny: 97 grid_points    dimensions(sizes): lat(97), lon(191)
> variables(dimensions): float32 LAT(lat,lon), float32 LON(lat,lon),
float32
> WDIR10(lat,lon), float32 WSPD10(lat,lon), float32 TMP(lat,lon),
float32
> DPT(lat,lon), float32 OZCON(lat,lon), float32 PMTF(lat,lon)
groups: *
>
> You can also see that the meta data is not overly descriptive as in
the
> past.  I'm not sure if that is problematic or not.  I do know that
whatever
> is not specified in the file is filled by -9999 or -999 values.
>
> Looking at the log file I find the following for every region, for
both
> PM2.5 and ozone:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *DEBUG 2: Processing OZCONA8 versus COPO/A8, for observation type
AIRNOW,
> over region WEST, for interpolation method BILIN(4), using 0 matched
> pairs.DEBUG 3: Number of matched pairs   = 0DEBUG 3: Observations
processed
>    = 59531DEBUG 3: Rejected: station id      = 0DEBUG 3: Rejected:
obs type
>        = 0DEBUG 3: Rejected: valid time      = 57003DEBUG 3:
Rejected: bad
> obs value   = 0DEBUG 3: Rejected: off the grid    = 2528DEBUG 3:
Rejected:
> topography      = 0DEBUG 3: Rejected: level mismatch  = 0DEBUG 3:
Rejected:
> quality marker  = 0DEBUG 3: Rejected: message type    = 0DEBUG 3:
Rejected:
> masking region  = 0DEBUG 3: Rejected: bad fcst value  = 0DEBUG 3:
Rejected:
> bad climo mean  = 0DEBUG 3: Rejected: bad climo stdev = 0DEBUG 3:
Rejected:
> duplicates      = 0*
>
> There's no errors to report, rather there are just no matched pairs.
Since
> there are only 24 hours being processed for this data, then only 24
hours
> worth of data are being added to the prebufr*nc files.  You can also
see in
> the log file that the appropriate files are being selected according
to
> date:
>
>
>
> *DEBUG 1: Forecast File:
>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/aqm.20190810/
> aqm.t12z.metcro2d.10.nc
> <http://aqm.t12z.metcro2d.10.nc>DEBUG 1: Observation File:
>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r112_v1/aq/
> prepbufr.aqm.20190810.nc
> <http://prepbufr.aqm.20190810.nc>DEBUG 7: check_nc_data_2d() count:
> valid=18527, zero=0, missing=0, 18527 == 18527*
>
> I'm having a hard time figuring out why there are no matched pairs.
A
> couple of other things to note related to an earlier statement about
meta
> data..
>
> Recall that I said the files weren't overly descriptive.  You can
see that
> the -9999 is being filed for many of the definitions that are not
> specifically being set in the created netcdf files:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *DEBUG 4: Lambert Conformal Grid Data:DEBUG 4:    hemisphere: NDEBUG
4:
> scale_lat_1: -9999DEBUG 4:   scale_lat_2: -9999DEBUG 4:
lat_pin:
> -9999DEBUG 4:       lon_pin: 9999DEBUG 4:         x_pin: -9999DEBUG
4:
>     y_pin: -9999DEBUG 4:    lon_orient: 9999DEBUG 4:          d_km:
> -9999DEBUG 4:          r_km: -9999DEBUG 4:            nx: 191DEBUG
4:
>      ny: 97DEBUG 4:     so2_angle: 0DEBUG 4:*
>
> I'm not sure if that is playing a role here.  Printing out the ozone
field
> from the netcdf file you can see that other information like
long_name,
> level, etc. are given.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *>>> Dataset('aqm.t12z.metcro2d.02.nc
> <http://aqm.t12z.metcro2d.02.nc>').variables['OZCON']<class
> 'netCDF4._netCDF4.Variable'>float32 OZCON(lat, lon)    long_name:
OZCON
> standard_name: Surface Ozone    units: ppb    level: A8
init_time:
> 20190810_120000    init_time_ut: 1565438400    valid_time:
20190810_140000
>   valid_time_ut: 1565445600unlimited dimensions: current shape =
(97,
> 191)filling on, default _FillValue of 9.969209968386869e+36 used*
>
> Is there something I'm not thinking about here?  I can provide more
details
> if needed.  Hopefully this description is a good start.  I do recall
having
> a similar issue in the past but couldn't locate the exchange going
over
> this problem.
>
>
>
>
> Edward Strobach
> EMC/NCEP/NWS/ IMSG
> Coding Logic:  if, then, else
>
>

--
Julie Prestopnik (she/her)
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Email: jpresto at ucar.edu

My working day may not be your working day.  Please do not feel
obliged to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: zero matched pairs when processing the AQ inline data
From: Edward Strobach - NOAA Affiliate
Time: Tue Apr 13 13:17:02 2021

Thank you for your help.

I see from my metplus_final_pb2nc_pointstat.conf located in my
"/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/conf/r112_v1/aq"
directory are references to the time window for both the pb2nc and
pointstat step:

*PB2NC_WINDOW_BEGIN = -86400PB2NC_WINDOW_END = 86400*

and

*OBS_WINDOW_BEGIN = -86400OBS_WINDOW_END = 86400*

It's interesting that you note the settings for OBS_WINDOW* in the
/*gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
file* are set to 2700.  I'm not sure what's setting that.  I thought
it was
being set based on what's in the shared.conf found in my
run_metplus_aq.sh
script located here:
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/scripts.

I thought that the OBS_WINDOW* definitions set in the shared.conf
would
override other conf files as well.  Running that step looks like this
in my
run_metplus_aq.sh script:

*${MET_PLUS}/ush/master_metplus.py -c
${MET_PLUS}/parm/use_cases/grid_to_obs/grid_to_obs.conf -c
${MET_PLUS}/parm/use_cases/grid_to_obs/examples/conus_surface.conf -c
${MET_PLUS_CONF}/pb2nc_aq.conf -c ${MET_PLUS_CONF}/point_stat_aq.conf
-c
${MET_PLUS_TMP}/${field}.conf -c ${MET_PLUS_CONF}/shared.conf -c
${MET_PLUS_TMP}/shared.conf_aq -c ${MET_PLUS_CONF}/system_aq.conf*

Something else that is troubling from that conf file you mentioned is
the
valid times settings:
VALID_BEG = 20210116
VALID_END = 20210116

I set that specifically to 20190810.  I didn't think to look where you
did
because the inputs that I assigned to my main run script look like
this:

/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/scripts/run_metplus_aq.sh
*r112_v1 archive verification LAM 20190810*

the input date in this case is 20190810 but somehow is not being
used..  I
don't know how this could be happening.  Thanks for pointing this out.

Edward Strobach
EMC/NCEP/NWS/ IMSG
Coding Logic:  if, then, else

On Tue, Apr 13, 2021 at 2:20 PM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:

> Hi Edward.
>
> I see that you are having trouble getting matched pairs from Point-
Stat.
>
> Let's start by taking a look at exactly where the log messages are
pointing
> us. Looking at one of your log files:
>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210413144310
>
> Looking at the first call of point_stat, I see the following output
> indicating that all of the 21,468 observations were rejected due to
the
> valid_time, which is why there are no matched pairs for this case:
>
> > DEBUG 3: Number of matched pairs   = 0
> > DEBUG 3: Observations processed    = 21468
> > DEBUG 3: Rejected: station id      = 0
> > DEBUG 3: Rejected: obs type        = 0
> > DEBUG 3: Rejected: valid time      = 21468
> > DEBUG 3: Rejected: bad obs value   = 0
> > DEBUG 3: Rejected: off the grid    = 0
> > DEBUG 3: Rejected: topography      = 0
> > DEBUG 3: Rejected: level mismatch  = 0
> > DEBUG 3: Rejected: quality marker  = 0
> > DEBUG 3: Rejected: message type    = 0
> > DEBUG 3: Rejected: masking region  = 0
> > DEBUG 3: Rejected: bad fcst value  = 0
> > DEBUG 3: Rejected: bad climo mean  = 0
> > DEBUG 3: Rejected: bad climo stdev = 0
> > DEBUG 3: Rejected: duplicates      = 0
>
>
> This is similar to the situation in the Rejected values that you had
pasted
> in, except some of those observations were rejected because they
were off
> of the grid, which is why there are no matched pairs for this case:
>
> > DEBUG 3: Observations processed = 59531
> > DEBUG 3: Rejected: valid time      = 57003
> > DEBUG 3: Rejected: off the grid    = 2528
>
> I wasn't sure which log file that came from, so I just used your
most
> recent log file to get the example above.
>
> You'll want to make sure that when you run Point-Stat, the forecast
> valid time matches up correctly with the valid time of point
observations.
> I may not be the best person to help with that, but I did find
something
> that may be a part of the problem.
>
> Here is the call to point_stat for the case with the 21,468
observations
> rejected due to valid time:
> point_stat -v 7 \
>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/\
> aqm.20190809/aqm.t12z.metcro2d.12.nc \
>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r112_v1/pm/
> prepbufr.pm.20190810.nc
>
>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
> \
> -outdir
>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/stat/pm/r112_v1
>
> I can see that
> the
>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
> configuration file is being used.
>
> In that configuration file I see:
>
> > obs_window = {
> >    beg = 0;
> >    end = 0;
> > }
>
>
> whereas in many of the other Point-Stat configuration files in that
> directory I see:
>
> > obs_window = {
> >    beg = ${OBS_WINDOW_BEGIN};
> >    end = ${OBS_WINDOW_END};
> > }
>
>
> In
> your
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
> file, I see the following set:
>
> > OBS_WINDOW_BEGIN = -2700
> > OBS_WINDOW_END = 2700
>
>
> and I was wondering if you are intending for METplus to pass on
these
> values in the run to Point-Stat?  If so, you would need to change
the
> values of zero to ${OBS_WINDOW_BEGIN} and ${OBS_WINDOW_END},
respectively.
> Perhaps that is the cause of the zero matched pairs due to valid
time?
>
> I hope this helps!
>
> Julie
>
>
> On Tue, Apr 13, 2021 at 9:27 AM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Apr 13 09:27:33 2021: Request 99563 was acted upon.
> > Transaction: Ticket created by edward.strobach at noaa.gov
> >        Queue: met_help
> >      Subject: zero matched pairs when processing the AQ inline
data
> >        Owner: Nobody
> >   Requestors: edward.strobach at noaa.gov
> >       Status: new
> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99563 >
> >
> >
> > Good morning,
> >
> > I'm attempting to process the data from here: "
> >
> >
>
*/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/aqm.20190810*
> > "
> >
> > The files follow the "aqm.t12z.metcro2d.HH.nc" format and are
written
> in a
> > format that METplus can handle.  The processing steps include
pb2nc,
> > pointstat, and statanalysis.  The issue occurs at pointstat.  The
pb2nc
> > step runs just fine.
> >
> > Looking in the netcdf files that I created above confirms that the
> fields I
> > need are there:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *>>> Dataset('aqm.t12z.metcro2d.02.nc
> > <http://aqm.t12z.metcro2d.02.nc>')<class
'netCDF4._netCDF4.Dataset'>root
> > group (NETCDF4 data model, file format HDF5):    FileOrigins: NA
> > MET_version: V9.1    hemisphere: N    Projection: Lambert
Conformal
> nx:
> > 191    ny: 97 grid_points    dimensions(sizes): lat(97), lon(191)
> > variables(dimensions): float32 LAT(lat,lon), float32 LON(lat,lon),
> float32
> > WDIR10(lat,lon), float32 WSPD10(lat,lon), float32 TMP(lat,lon),
float32
> > DPT(lat,lon), float32 OZCON(lat,lon), float32 PMTF(lat,lon)
groups: *
> >
> > You can also see that the meta data is not overly descriptive as
in the
> > past.  I'm not sure if that is problematic or not.  I do know that
> whatever
> > is not specified in the file is filled by -9999 or -999 values.
> >
> > Looking at the log file I find the following for every region, for
both
> > PM2.5 and ozone:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *DEBUG 2: Processing OZCONA8 versus COPO/A8, for observation type
AIRNOW,
> > over region WEST, for interpolation method BILIN(4), using 0
matched
> > pairs.DEBUG 3: Number of matched pairs   = 0DEBUG 3: Observations
> processed
> >    = 59531DEBUG 3: Rejected: station id      = 0DEBUG 3: Rejected:
obs
> type
> >        = 0DEBUG 3: Rejected: valid time      = 57003DEBUG 3:
Rejected:
> bad
> > obs value   = 0DEBUG 3: Rejected: off the grid    = 2528DEBUG 3:
> Rejected:
> > topography      = 0DEBUG 3: Rejected: level mismatch  = 0DEBUG 3:
> Rejected:
> > quality marker  = 0DEBUG 3: Rejected: message type    = 0DEBUG 3:
> Rejected:
> > masking region  = 0DEBUG 3: Rejected: bad fcst value  = 0DEBUG 3:
> Rejected:
> > bad climo mean  = 0DEBUG 3: Rejected: bad climo stdev = 0DEBUG 3:
> Rejected:
> > duplicates      = 0*
> >
> > There's no errors to report, rather there are just no matched
pairs.
> Since
> > there are only 24 hours being processed for this data, then only
24 hours
> > worth of data are being added to the prebufr*nc files.  You can
also see
> in
> > the log file that the appropriate files are being selected
according to
> > date:
> >
> >
> >
> > *DEBUG 1: Forecast File:
> >
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/aqm.20190810/
> > aqm.t12z.metcro2d.10.nc
> > <http://aqm.t12z.metcro2d.10.nc>DEBUG 1: Observation File:
> >
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r112_v1/aq/
> > prepbufr.aqm.20190810.nc
> > <http://prepbufr.aqm.20190810.nc>DEBUG 7: check_nc_data_2d()
count:
> > valid=18527, zero=0, missing=0, 18527 == 18527*
> >
> > I'm having a hard time figuring out why there are no matched
pairs.  A
> > couple of other things to note related to an earlier statement
about meta
> > data..
> >
> > Recall that I said the files weren't overly descriptive.  You can
see
> that
> > the -9999 is being filed for many of the definitions that are not
> > specifically being set in the created netcdf files:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *DEBUG 4: Lambert Conformal Grid Data:DEBUG 4:    hemisphere:
NDEBUG 4:
> > scale_lat_1: -9999DEBUG 4:   scale_lat_2: -9999DEBUG 4:
lat_pin:
> > -9999DEBUG 4:       lon_pin: 9999DEBUG 4:         x_pin:
-9999DEBUG 4:
> >     y_pin: -9999DEBUG 4:    lon_orient: 9999DEBUG 4:
d_km:
> > -9999DEBUG 4:          r_km: -9999DEBUG 4:            nx: 191DEBUG
4:
> >      ny: 97DEBUG 4:     so2_angle: 0DEBUG 4:*
> >
> > I'm not sure if that is playing a role here.  Printing out the
ozone
> field
> > from the netcdf file you can see that other information like
long_name,
> > level, etc. are given.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *>>> Dataset('aqm.t12z.metcro2d.02.nc
> > <http://aqm.t12z.metcro2d.02.nc>').variables['OZCON']<class
> > 'netCDF4._netCDF4.Variable'>float32 OZCON(lat, lon)    long_name:
OZCON
> > standard_name: Surface Ozone    units: ppb    level: A8
init_time:
> > 20190810_120000    init_time_ut: 1565438400    valid_time:
> 20190810_140000
> >   valid_time_ut: 1565445600unlimited dimensions: current shape =
(97,
> > 191)filling on, default _FillValue of 9.969209968386869e+36 used*
> >
> > Is there something I'm not thinking about here?  I can provide
more
> details
> > if needed.  Hopefully this description is a good start.  I do
recall
> having
> > a similar issue in the past but couldn't locate the exchange going
over
> > this problem.
> >
> >
> >
> >
> > Edward Strobach
> > EMC/NCEP/NWS/ IMSG
> > Coding Logic:  if, then, else
> >
> >
>
> --
> Julie Prestopnik (she/her)
> Software Engineer
> National Center for Atmospheric Research
> Research Applications Laboratory
> Email: jpresto at ucar.edu
>
> My working day may not be your working day.  Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

------------------------------------------------
Subject: zero matched pairs when processing the AQ inline data
From: Julie Prestopnik
Time: Tue Apr 13 16:16:45 2021

Hi Edward.

It's interesting that you note the settings for OBS_WINDOW* in the
>
>
/*gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
> file* are set to 2700.  I'm not sure what's setting that.
>
Perhaps the metplus_final.conf file I referred to did not go with the
run
for the log I referenced:
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210413144310

Because I can see in that log file, just above the call to point_stat,
the
values for some of the fields are listed and indeed 86400 is used for
OBS_WINDOW_BEGIN:

> 04/13 14:43:12.448 metplus.PointStat (command_builder.py:256) DEBUG:
> export CLIMO_MEAN_FILE=""; export CLIMO_STDEV_FILE=""; export
FCST_FIELD="{
> name\
> =\"PMTF\"; level=\"L1\"; },{ name=\"OZCON\"; level=\"A8\"; }";
export
>
MET_TMP_DIR="/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CM\
> AQ/tmp"; export MODEL="R112_V1"; export OBS_FIELD="{
name=\"COPOPM\";
> level=\"A1\"; convert(x) = x * 10^9; },{ name=\"COPO\";
level=\"A8\";
> convert(x\
> ) = x * 10^9; }"; export OBS_WINDOW_BEGIN="-86400"; export
> OBS_WINDOW_END="86400"; export OUTPUT_PREFIX=""; export
> POINT_STAT_GRID="[\"FULL\"]"; expo\
> rt POINT_STAT_MESSAGE_TYPE="[\"ONLYSF\"]"; export
POINT_STAT_POLY="[]";
> export POINT_STAT_STATION_ID="[]"; export REGRID_TO_GRID="\"G104\"";
export
> V\
> ERIF_MASK="";

In a call like the following:

> *${MET_PLUS}/ush/master_metplus.py -c
> ${MET_PLUS}/parm/use_cases/grid_to_obs/grid_to_obs.conf -c
> ${MET_PLUS}/parm/use_cases/grid_to_obs/examples/conus_surface.conf
-c
> ${MET_PLUS_CONF}/pb2nc_aq.conf -c
${MET_PLUS_CONF}/point_stat_aq.conf -c
> ${MET_PLUS_TMP}/${field}.conf -c ${MET_PLUS_CONF}/shared.conf -c
> ${MET_PLUS_TMP}/shared.conf_aq -c ${MET_PLUS_CONF}/system_aq.conf*

Anything in ${MET_PLUS_CONF}/system_aq.conf would override anything in
any
previously listed config file.  The order in which you list your
METplus
wrapper config files matters, as it sounds like you are aware of. Each
subsequent config file on the command line will override any values
defined
in an earlier config file.

The issue may be the same with the VALID_BEG and VALID_END in that the
metplus_final.conf file I referred to did not go with the run for the
log I
referenced.

You could start a clean run and check the time stamp on
the metplus_final.conf file, save that off along with the
corresponding
metplus log file, check the values and let us know if you see anything
unusual.

Let us know what your questions are as you proceed working on getting
the
valid times to line up.

Julie

On Tue, Apr 13, 2021 at 1:17 PM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99563 >
>
> Thank you for your help.
>
> I see from my metplus_final_pb2nc_pointstat.conf located in my
>
>
"/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/conf/r112_v1/aq"
> directory are references to the time window for both the pb2nc and
> pointstat step:
>
>
> *PB2NC_WINDOW_BEGIN = -86400PB2NC_WINDOW_END = 86400*
>
> and
>
>
> *OBS_WINDOW_BEGIN = -86400OBS_WINDOW_END = 86400*
>
> It's interesting that you note the settings for OBS_WINDOW* in the
>
>
/*gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
> file* are set to 2700.  I'm not sure what's setting that.  I thought
it was
> being set based on what's in the shared.conf found in my
run_metplus_aq.sh
> script located here:
>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/scripts.
>
> I thought that the OBS_WINDOW* definitions set in the shared.conf
would
> override other conf files as well.  Running that step looks like
this in my
> run_metplus_aq.sh script:
>
> *${MET_PLUS}/ush/master_metplus.py -c
> ${MET_PLUS}/parm/use_cases/grid_to_obs/grid_to_obs.conf -c
> ${MET_PLUS}/parm/use_cases/grid_to_obs/examples/conus_surface.conf
-c
> ${MET_PLUS_CONF}/pb2nc_aq.conf -c
${MET_PLUS_CONF}/point_stat_aq.conf -c
> ${MET_PLUS_TMP}/${field}.conf -c ${MET_PLUS_CONF}/shared.conf -c
> ${MET_PLUS_TMP}/shared.conf_aq -c ${MET_PLUS_CONF}/system_aq.conf*
>
> Something else that is troubling from that conf file you mentioned
is the
> valid times settings:
> VALID_BEG = 20210116
> VALID_END = 20210116
>
> I set that specifically to 20190810.  I didn't think to look where
you did
> because the inputs that I assigned to my main run script look like
this:
>
>
>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/scripts/run_metplus_aq.sh
> *r112_v1 archive verification LAM 20190810*
>
> the input date in this case is 20190810 but somehow is not being
used..  I
> don't know how this could be happening.  Thanks for pointing this
out.
>
>
>
> Edward Strobach
> EMC/NCEP/NWS/ IMSG
> Coding Logic:  if, then, else
>
>
> On Tue, Apr 13, 2021 at 2:20 PM Julie Prestopnik via RT
<met_help at ucar.edu
> >
> wrote:
>
> > Hi Edward.
> >
> > I see that you are having trouble getting matched pairs from
Point-Stat.
> >
> > Let's start by taking a look at exactly where the log messages are
> pointing
> > us. Looking at one of your log files:
> >
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210413144310
> >
> > Looking at the first call of point_stat, I see the following
output
> > indicating that all of the 21,468 observations were rejected due
to the
> > valid_time, which is why there are no matched pairs for this case:
> >
> > > DEBUG 3: Number of matched pairs   = 0
> > > DEBUG 3: Observations processed    = 21468
> > > DEBUG 3: Rejected: station id      = 0
> > > DEBUG 3: Rejected: obs type        = 0
> > > DEBUG 3: Rejected: valid time      = 21468
> > > DEBUG 3: Rejected: bad obs value   = 0
> > > DEBUG 3: Rejected: off the grid    = 0
> > > DEBUG 3: Rejected: topography      = 0
> > > DEBUG 3: Rejected: level mismatch  = 0
> > > DEBUG 3: Rejected: quality marker  = 0
> > > DEBUG 3: Rejected: message type    = 0
> > > DEBUG 3: Rejected: masking region  = 0
> > > DEBUG 3: Rejected: bad fcst value  = 0
> > > DEBUG 3: Rejected: bad climo mean  = 0
> > > DEBUG 3: Rejected: bad climo stdev = 0
> > > DEBUG 3: Rejected: duplicates      = 0
> >
> >
> > This is similar to the situation in the Rejected values that you
had
> pasted
> > in, except some of those observations were rejected because they
were off
> > of the grid, which is why there are no matched pairs for this
case:
> >
> > > DEBUG 3: Observations processed = 59531
> > > DEBUG 3: Rejected: valid time      = 57003
> > > DEBUG 3: Rejected: off the grid    = 2528
> >
> > I wasn't sure which log file that came from, so I just used your
most
> > recent log file to get the example above.
> >
> > You'll want to make sure that when you run Point-Stat, the
forecast
> > valid time matches up correctly with the valid time of point
> observations.
> > I may not be the best person to help with that, but I did find
something
> > that may be a part of the problem.
> >
> > Here is the call to point_stat for the case with the 21,468
observations
> > rejected due to valid time:
> > point_stat -v 7 \
> >
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/\
> > aqm.20190809/aqm.t12z.metcro2d.12.nc \
> >
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r112_v1/pm/
> > prepbufr.pm.20190810.nc
> >
> >
>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
> > \
> > -outdir
> >
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/stat/pm/r112_v1
> >
> > I can see that
> > the
> >
>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
> > configuration file is being used.
> >
> > In that configuration file I see:
> >
> > > obs_window = {
> > >    beg = 0;
> > >    end = 0;
> > > }
> >
> >
> > whereas in many of the other Point-Stat configuration files in
that
> > directory I see:
> >
> > > obs_window = {
> > >    beg = ${OBS_WINDOW_BEGIN};
> > >    end = ${OBS_WINDOW_END};
> > > }
> >
> >
> > In
> > your
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
> > file, I see the following set:
> >
> > > OBS_WINDOW_BEGIN = -2700
> > > OBS_WINDOW_END = 2700
> >
> >
> > and I was wondering if you are intending for METplus to pass on
these
> > values in the run to Point-Stat?  If so, you would need to change
the
> > values of zero to ${OBS_WINDOW_BEGIN} and ${OBS_WINDOW_END},
> respectively.
> > Perhaps that is the cause of the zero matched pairs due to valid
time?
> >
> > I hope this helps!
> >
> > Julie
> >
> >
> > On Tue, Apr 13, 2021 at 9:27 AM Edward Strobach - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Apr 13 09:27:33 2021: Request 99563 was acted upon.
> > > Transaction: Ticket created by edward.strobach at noaa.gov
> > >        Queue: met_help
> > >      Subject: zero matched pairs when processing the AQ inline
data
> > >        Owner: Nobody
> > >   Requestors: edward.strobach at noaa.gov
> > >       Status: new
> > >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99563
> >
> > >
> > >
> > > Good morning,
> > >
> > > I'm attempting to process the data from here: "
> > >
> > >
> >
>
*/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/aqm.20190810*
> > > "
> > >
> > > The files follow the "aqm.t12z.metcro2d.HH.nc" format and are
written
> > in a
> > > format that METplus can handle.  The processing steps include
pb2nc,
> > > pointstat, and statanalysis.  The issue occurs at pointstat.
The pb2nc
> > > step runs just fine.
> > >
> > > Looking in the netcdf files that I created above confirms that
the
> > fields I
> > > need are there:
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *>>> Dataset('aqm.t12z.metcro2d.02.nc
> > > <http://aqm.t12z.metcro2d.02.nc>')<class
> 'netCDF4._netCDF4.Dataset'>root
> > > group (NETCDF4 data model, file format HDF5):    FileOrigins: NA
> > > MET_version: V9.1    hemisphere: N    Projection: Lambert
Conformal
> > nx:
> > > 191    ny: 97 grid_points    dimensions(sizes): lat(97),
lon(191)
> > > variables(dimensions): float32 LAT(lat,lon), float32
LON(lat,lon),
> > float32
> > > WDIR10(lat,lon), float32 WSPD10(lat,lon), float32 TMP(lat,lon),
float32
> > > DPT(lat,lon), float32 OZCON(lat,lon), float32 PMTF(lat,lon)
groups:
> *
> > >
> > > You can also see that the meta data is not overly descriptive as
in the
> > > past.  I'm not sure if that is problematic or not.  I do know
that
> > whatever
> > > is not specified in the file is filled by -9999 or -999 values.
> > >
> > > Looking at the log file I find the following for every region,
for both
> > > PM2.5 and ozone:
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *DEBUG 2: Processing OZCONA8 versus COPO/A8, for observation
type
> AIRNOW,
> > > over region WEST, for interpolation method BILIN(4), using 0
matched
> > > pairs.DEBUG 3: Number of matched pairs   = 0DEBUG 3:
Observations
> > processed
> > >    = 59531DEBUG 3: Rejected: station id      = 0DEBUG 3:
Rejected: obs
> > type
> > >        = 0DEBUG 3: Rejected: valid time      = 57003DEBUG 3:
Rejected:
> > bad
> > > obs value   = 0DEBUG 3: Rejected: off the grid    = 2528DEBUG 3:
> > Rejected:
> > > topography      = 0DEBUG 3: Rejected: level mismatch  = 0DEBUG
3:
> > Rejected:
> > > quality marker  = 0DEBUG 3: Rejected: message type    = 0DEBUG
3:
> > Rejected:
> > > masking region  = 0DEBUG 3: Rejected: bad fcst value  = 0DEBUG
3:
> > Rejected:
> > > bad climo mean  = 0DEBUG 3: Rejected: bad climo stdev = 0DEBUG
3:
> > Rejected:
> > > duplicates      = 0*
> > >
> > > There's no errors to report, rather there are just no matched
pairs.
> > Since
> > > there are only 24 hours being processed for this data, then only
24
> hours
> > > worth of data are being added to the prebufr*nc files.  You can
also
> see
> > in
> > > the log file that the appropriate files are being selected
according to
> > > date:
> > >
> > >
> > >
> > > *DEBUG 1: Forecast File:
> > >
> > >
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/aqm.20190810/
> > > aqm.t12z.metcro2d.10.nc
> > > <http://aqm.t12z.metcro2d.10.nc>DEBUG 1: Observation File:
> > >
> > >
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r112_v1/aq/
> > > prepbufr.aqm.20190810.nc
> > > <http://prepbufr.aqm.20190810.nc>DEBUG 7: check_nc_data_2d()
count:
> > > valid=18527, zero=0, missing=0, 18527 == 18527*
> > >
> > > I'm having a hard time figuring out why there are no matched
pairs.  A
> > > couple of other things to note related to an earlier statement
about
> meta
> > > data..
> > >
> > > Recall that I said the files weren't overly descriptive.  You
can see
> > that
> > > the -9999 is being filed for many of the definitions that are
not
> > > specifically being set in the created netcdf files:
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *DEBUG 4: Lambert Conformal Grid Data:DEBUG 4:    hemisphere:
NDEBUG 4:
> > > scale_lat_1: -9999DEBUG 4:   scale_lat_2: -9999DEBUG 4:
lat_pin:
> > > -9999DEBUG 4:       lon_pin: 9999DEBUG 4:         x_pin:
-9999DEBUG 4:
> > >     y_pin: -9999DEBUG 4:    lon_orient: 9999DEBUG 4:
d_km:
> > > -9999DEBUG 4:          r_km: -9999DEBUG 4:            nx:
191DEBUG 4:
> > >      ny: 97DEBUG 4:     so2_angle: 0DEBUG 4:*
> > >
> > > I'm not sure if that is playing a role here.  Printing out the
ozone
> > field
> > > from the netcdf file you can see that other information like
long_name,
> > > level, etc. are given.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *>>> Dataset('aqm.t12z.metcro2d.02.nc
> > > <http://aqm.t12z.metcro2d.02.nc>').variables['OZCON']<class
> > > 'netCDF4._netCDF4.Variable'>float32 OZCON(lat, lon)
long_name: OZCON
> > > standard_name: Surface Ozone    units: ppb    level: A8
init_time:
> > > 20190810_120000    init_time_ut: 1565438400    valid_time:
> > 20190810_140000
> > >   valid_time_ut: 1565445600unlimited dimensions: current shape =
(97,
> > > 191)filling on, default _FillValue of 9.969209968386869e+36
used*
> > >
> > > Is there something I'm not thinking about here?  I can provide
more
> > details
> > > if needed.  Hopefully this description is a good start.  I do
recall
> > having
> > > a similar issue in the past but couldn't locate the exchange
going over
> > > this problem.
> > >
> > >
> > >
> > >
> > > Edward Strobach
> > > EMC/NCEP/NWS/ IMSG
> > > Coding Logic:  if, then, else
> > >
> > >
> >
> > --
> > Julie Prestopnik (she/her)
> > Software Engineer
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > Email: jpresto at ucar.edu
> >
> > My working day may not be your working day.  Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.
> >
> >
>
>

--
Julie Prestopnik (she/her)
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Email: jpresto at ucar.edu

My working day may not be your working day.  Please do not feel
obliged to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: zero matched pairs when processing the AQ inline data
From: Edward Strobach - NOAA Affiliate
Time: Tue Apr 13 18:09:12 2021

Thanks, that's what I'll try to do.

Edward Strobach
EMC/NCEP/NWS/ IMSG
Coding Logic:  if, then, else

On Tue, Apr 13, 2021 at 6:16 PM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:

> Hi Edward.
>
> It's interesting that you note the settings for OBS_WINDOW* in the
> >
> >
>
/*gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
> > file* are set to 2700.  I'm not sure what's setting that.
> >
> Perhaps the metplus_final.conf file I referred to did not go with
the run
> for the log I referenced:
>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210413144310
>
> Because I can see in that log file, just above the call to
point_stat, the
> values for some of the fields are listed and indeed 86400 is used
for
> OBS_WINDOW_BEGIN:
>
> > 04/13 14:43:12.448 metplus.PointStat (command_builder.py:256)
DEBUG:
> > export CLIMO_MEAN_FILE=""; export CLIMO_STDEV_FILE=""; export
> FCST_FIELD="{
> > name\
> > =\"PMTF\"; level=\"L1\"; },{ name=\"OZCON\"; level=\"A8\"; }";
export
> >
>
MET_TMP_DIR="/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CM\
> > AQ/tmp"; export MODEL="R112_V1"; export OBS_FIELD="{
name=\"COPOPM\";
> > level=\"A1\"; convert(x) = x * 10^9; },{ name=\"COPO\";
level=\"A8\";
> > convert(x\
> > ) = x * 10^9; }"; export OBS_WINDOW_BEGIN="-86400"; export
> > OBS_WINDOW_END="86400"; export OUTPUT_PREFIX=""; export
> > POINT_STAT_GRID="[\"FULL\"]"; expo\
> > rt POINT_STAT_MESSAGE_TYPE="[\"ONLYSF\"]"; export
POINT_STAT_POLY="[]";
> > export POINT_STAT_STATION_ID="[]"; export
REGRID_TO_GRID="\"G104\"";
> export
> > V\
> > ERIF_MASK="";
>
>
> In a call like the following:
>
> > *${MET_PLUS}/ush/master_metplus.py -c
> > ${MET_PLUS}/parm/use_cases/grid_to_obs/grid_to_obs.conf -c
> > ${MET_PLUS}/parm/use_cases/grid_to_obs/examples/conus_surface.conf
-c
> > ${MET_PLUS_CONF}/pb2nc_aq.conf -c
${MET_PLUS_CONF}/point_stat_aq.conf -c
> > ${MET_PLUS_TMP}/${field}.conf -c ${MET_PLUS_CONF}/shared.conf -c
> > ${MET_PLUS_TMP}/shared.conf_aq -c ${MET_PLUS_CONF}/system_aq.conf*
>
>
> Anything in ${MET_PLUS_CONF}/system_aq.conf would override anything
in any
> previously listed config file.  The order in which you list your
METplus
> wrapper config files matters, as it sounds like you are aware of.
Each
> subsequent config file on the command line will override any values
defined
> in an earlier config file.
>
> The issue may be the same with the VALID_BEG and VALID_END in that
the
> metplus_final.conf file I referred to did not go with the run for
the log I
> referenced.
>
> You could start a clean run and check the time stamp on
> the metplus_final.conf file, save that off along with the
corresponding
> metplus log file, check the values and let us know if you see
anything
> unusual.
>
> Let us know what your questions are as you proceed working on
getting the
> valid times to line up.
>
> Julie
>
>
> On Tue, Apr 13, 2021 at 1:17 PM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99563 >
> >
> > Thank you for your help.
> >
> > I see from my metplus_final_pb2nc_pointstat.conf located in my
> >
> >
>
"/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/conf/r112_v1/aq"
> > directory are references to the time window for both the pb2nc and
> > pointstat step:
> >
> >
> > *PB2NC_WINDOW_BEGIN = -86400PB2NC_WINDOW_END = 86400*
> >
> > and
> >
> >
> > *OBS_WINDOW_BEGIN = -86400OBS_WINDOW_END = 86400*
> >
> > It's interesting that you note the settings for OBS_WINDOW* in the
> >
> >
>
/*gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
> > file* are set to 2700.  I'm not sure what's setting that.  I
thought it
> was
> > being set based on what's in the shared.conf found in my
> run_metplus_aq.sh
> > script located here:
> >
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/scripts.
> >
> > I thought that the OBS_WINDOW* definitions set in the shared.conf
would
> > override other conf files as well.  Running that step looks like
this in
> my
> > run_metplus_aq.sh script:
> >
> > *${MET_PLUS}/ush/master_metplus.py -c
> > ${MET_PLUS}/parm/use_cases/grid_to_obs/grid_to_obs.conf -c
> > ${MET_PLUS}/parm/use_cases/grid_to_obs/examples/conus_surface.conf
-c
> > ${MET_PLUS_CONF}/pb2nc_aq.conf -c
${MET_PLUS_CONF}/point_stat_aq.conf -c
> > ${MET_PLUS_TMP}/${field}.conf -c ${MET_PLUS_CONF}/shared.conf -c
> > ${MET_PLUS_TMP}/shared.conf_aq -c ${MET_PLUS_CONF}/system_aq.conf*
> >
> > Something else that is troubling from that conf file you mentioned
is the
> > valid times settings:
> > VALID_BEG = 20210116
> > VALID_END = 20210116
> >
> > I set that specifically to 20190810.  I didn't think to look where
you
> did
> > because the inputs that I assigned to my main run script look like
this:
> >
> >
> >
>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/scripts/run_metplus_aq.sh
> > *r112_v1 archive verification LAM 20190810*
> >
> > the input date in this case is 20190810 but somehow is not being
used..
> I
> > don't know how this could be happening.  Thanks for pointing this
out.
> >
> >
> >
> > Edward Strobach
> > EMC/NCEP/NWS/ IMSG
> > Coding Logic:  if, then, else
> >
> >
> > On Tue, Apr 13, 2021 at 2:20 PM Julie Prestopnik via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > > Hi Edward.
> > >
> > > I see that you are having trouble getting matched pairs from
> Point-Stat.
> > >
> > > Let's start by taking a look at exactly where the log messages
are
> > pointing
> > > us. Looking at one of your log files:
> > >
> > >
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210413144310
> > >
> > > Looking at the first call of point_stat, I see the following
output
> > > indicating that all of the 21,468 observations were rejected due
to the
> > > valid_time, which is why there are no matched pairs for this
case:
> > >
> > > > DEBUG 3: Number of matched pairs   = 0
> > > > DEBUG 3: Observations processed    = 21468
> > > > DEBUG 3: Rejected: station id      = 0
> > > > DEBUG 3: Rejected: obs type        = 0
> > > > DEBUG 3: Rejected: valid time      = 21468
> > > > DEBUG 3: Rejected: bad obs value   = 0
> > > > DEBUG 3: Rejected: off the grid    = 0
> > > > DEBUG 3: Rejected: topography      = 0
> > > > DEBUG 3: Rejected: level mismatch  = 0
> > > > DEBUG 3: Rejected: quality marker  = 0
> > > > DEBUG 3: Rejected: message type    = 0
> > > > DEBUG 3: Rejected: masking region  = 0
> > > > DEBUG 3: Rejected: bad fcst value  = 0
> > > > DEBUG 3: Rejected: bad climo mean  = 0
> > > > DEBUG 3: Rejected: bad climo stdev = 0
> > > > DEBUG 3: Rejected: duplicates      = 0
> > >
> > >
> > > This is similar to the situation in the Rejected values that you
had
> > pasted
> > > in, except some of those observations were rejected because they
were
> off
> > > of the grid, which is why there are no matched pairs for this
case:
> > >
> > > > DEBUG 3: Observations processed = 59531
> > > > DEBUG 3: Rejected: valid time      = 57003
> > > > DEBUG 3: Rejected: off the grid    = 2528
> > >
> > > I wasn't sure which log file that came from, so I just used your
most
> > > recent log file to get the example above.
> > >
> > > You'll want to make sure that when you run Point-Stat, the
forecast
> > > valid time matches up correctly with the valid time of point
> > observations.
> > > I may not be the best person to help with that, but I did find
> something
> > > that may be a part of the problem.
> > >
> > > Here is the call to point_stat for the case with the 21,468
> observations
> > > rejected due to valid time:
> > > point_stat -v 7 \
> > >
> > >
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/\
> > > aqm.20190809/aqm.t12z.metcro2d.12.nc \
> > >
> > >
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r112_v1/pm/
> > > prepbufr.pm.20190810.nc
> > >
> > >
> >
>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
> > > \
> > > -outdir
> > >
> > >
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/stat/pm/r112_v1
> > >
> > > I can see that
> > > the
> > >
> >
>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
> > > configuration file is being used.
> > >
> > > In that configuration file I see:
> > >
> > > > obs_window = {
> > > >    beg = 0;
> > > >    end = 0;
> > > > }
> > >
> > >
> > > whereas in many of the other Point-Stat configuration files in
that
> > > directory I see:
> > >
> > > > obs_window = {
> > > >    beg = ${OBS_WINDOW_BEGIN};
> > > >    end = ${OBS_WINDOW_END};
> > > > }
> > >
> > >
> > > In
> > > your
> > >
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
> > > file, I see the following set:
> > >
> > > > OBS_WINDOW_BEGIN = -2700
> > > > OBS_WINDOW_END = 2700
> > >
> > >
> > > and I was wondering if you are intending for METplus to pass on
these
> > > values in the run to Point-Stat?  If so, you would need to
change the
> > > values of zero to ${OBS_WINDOW_BEGIN} and ${OBS_WINDOW_END},
> > respectively.
> > > Perhaps that is the cause of the zero matched pairs due to valid
time?
> > >
> > > I hope this helps!
> > >
> > > Julie
> > >
> > >
> > > On Tue, Apr 13, 2021 at 9:27 AM Edward Strobach - NOAA Affiliate
via
> RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Tue Apr 13 09:27:33 2021: Request 99563 was acted upon.
> > > > Transaction: Ticket created by edward.strobach at noaa.gov
> > > >        Queue: met_help
> > > >      Subject: zero matched pairs when processing the AQ inline
data
> > > >        Owner: Nobody
> > > >   Requestors: edward.strobach at noaa.gov
> > > >       Status: new
> > > >  Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99563
> > >
> > > >
> > > >
> > > > Good morning,
> > > >
> > > > I'm attempting to process the data from here: "
> > > >
> > > >
> > >
> >
>
*/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/aqm.20190810*
> > > > "
> > > >
> > > > The files follow the "aqm.t12z.metcro2d.HH.nc" format and are
> written
> > > in a
> > > > format that METplus can handle.  The processing steps include
pb2nc,
> > > > pointstat, and statanalysis.  The issue occurs at pointstat.
The
> pb2nc
> > > > step runs just fine.
> > > >
> > > > Looking in the netcdf files that I created above confirms that
the
> > > fields I
> > > > need are there:
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *>>> Dataset('aqm.t12z.metcro2d.02.nc
> > > > <http://aqm.t12z.metcro2d.02.nc>')<class
> > 'netCDF4._netCDF4.Dataset'>root
> > > > group (NETCDF4 data model, file format HDF5):    FileOrigins:
NA
> > > > MET_version: V9.1    hemisphere: N    Projection: Lambert
Conformal
> > > nx:
> > > > 191    ny: 97 grid_points    dimensions(sizes): lat(97),
lon(191)
> > > > variables(dimensions): float32 LAT(lat,lon), float32
LON(lat,lon),
> > > float32
> > > > WDIR10(lat,lon), float32 WSPD10(lat,lon), float32
TMP(lat,lon),
> float32
> > > > DPT(lat,lon), float32 OZCON(lat,lon), float32 PMTF(lat,lon)
> groups:
> > *
> > > >
> > > > You can also see that the meta data is not overly descriptive
as in
> the
> > > > past.  I'm not sure if that is problematic or not.  I do know
that
> > > whatever
> > > > is not specified in the file is filled by -9999 or -999
values.
> > > >
> > > > Looking at the log file I find the following for every region,
for
> both
> > > > PM2.5 and ozone:
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *DEBUG 2: Processing OZCONA8 versus COPO/A8, for observation
type
> > AIRNOW,
> > > > over region WEST, for interpolation method BILIN(4), using 0
matched
> > > > pairs.DEBUG 3: Number of matched pairs   = 0DEBUG 3:
Observations
> > > processed
> > > >    = 59531DEBUG 3: Rejected: station id      = 0DEBUG 3:
Rejected:
> obs
> > > type
> > > >        = 0DEBUG 3: Rejected: valid time      = 57003DEBUG 3:
> Rejected:
> > > bad
> > > > obs value   = 0DEBUG 3: Rejected: off the grid    = 2528DEBUG
3:
> > > Rejected:
> > > > topography      = 0DEBUG 3: Rejected: level mismatch  = 0DEBUG
3:
> > > Rejected:
> > > > quality marker  = 0DEBUG 3: Rejected: message type    = 0DEBUG
3:
> > > Rejected:
> > > > masking region  = 0DEBUG 3: Rejected: bad fcst value  = 0DEBUG
3:
> > > Rejected:
> > > > bad climo mean  = 0DEBUG 3: Rejected: bad climo stdev = 0DEBUG
3:
> > > Rejected:
> > > > duplicates      = 0*
> > > >
> > > > There's no errors to report, rather there are just no matched
pairs.
> > > Since
> > > > there are only 24 hours being processed for this data, then
only 24
> > hours
> > > > worth of data are being added to the prebufr*nc files.  You
can also
> > see
> > > in
> > > > the log file that the appropriate files are being selected
according
> to
> > > > date:
> > > >
> > > >
> > > >
> > > > *DEBUG 1: Forecast File:
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/aqm.20190810/
> > > > aqm.t12z.metcro2d.10.nc
> > > > <http://aqm.t12z.metcro2d.10.nc>DEBUG 1: Observation File:
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r112_v1/aq/
> > > > prepbufr.aqm.20190810.nc
> > > > <http://prepbufr.aqm.20190810.nc>DEBUG 7: check_nc_data_2d()
count:
> > > > valid=18527, zero=0, missing=0, 18527 == 18527*
> > > >
> > > > I'm having a hard time figuring out why there are no matched
pairs.
> A
> > > > couple of other things to note related to an earlier statement
about
> > meta
> > > > data..
> > > >
> > > > Recall that I said the files weren't overly descriptive.  You
can see
> > > that
> > > > the -9999 is being filed for many of the definitions that are
not
> > > > specifically being set in the created netcdf files:
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *DEBUG 4: Lambert Conformal Grid Data:DEBUG 4:    hemisphere:
NDEBUG
> 4:
> > > > scale_lat_1: -9999DEBUG 4:   scale_lat_2: -9999DEBUG 4:
>  lat_pin:
> > > > -9999DEBUG 4:       lon_pin: 9999DEBUG 4:         x_pin:
-9999DEBUG
> 4:
> > > >     y_pin: -9999DEBUG 4:    lon_orient: 9999DEBUG 4:
d_km:
> > > > -9999DEBUG 4:          r_km: -9999DEBUG 4:            nx:
191DEBUG 4:
> > > >      ny: 97DEBUG 4:     so2_angle: 0DEBUG 4:*
> > > >
> > > > I'm not sure if that is playing a role here.  Printing out the
ozone
> > > field
> > > > from the netcdf file you can see that other information like
> long_name,
> > > > level, etc. are given.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *>>> Dataset('aqm.t12z.metcro2d.02.nc
> > > > <http://aqm.t12z.metcro2d.02.nc>').variables['OZCON']<class
> > > > 'netCDF4._netCDF4.Variable'>float32 OZCON(lat, lon)
long_name:
> OZCON
> > > > standard_name: Surface Ozone    units: ppb    level: A8
init_time:
> > > > 20190810_120000    init_time_ut: 1565438400    valid_time:
> > > 20190810_140000
> > > >   valid_time_ut: 1565445600unlimited dimensions: current shape
= (97,
> > > > 191)filling on, default _FillValue of 9.969209968386869e+36
used*
> > > >
> > > > Is there something I'm not thinking about here?  I can provide
more
> > > details
> > > > if needed.  Hopefully this description is a good start.  I do
recall
> > > having
> > > > a similar issue in the past but couldn't locate the exchange
going
> over
> > > > this problem.
> > > >
> > > >
> > > >
> > > >
> > > > Edward Strobach
> > > > EMC/NCEP/NWS/ IMSG
> > > > Coding Logic:  if, then, else
> > > >
> > > >
> > >
> > > --
> > > Julie Prestopnik (she/her)
> > > Software Engineer
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > Email: jpresto at ucar.edu
> > >
> > > My working day may not be your working day.  Please do not feel
obliged
> > to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
> --
> Julie Prestopnik (she/her)
> Software Engineer
> National Center for Atmospheric Research
> Research Applications Laboratory
> Email: jpresto at ucar.edu
>
> My working day may not be your working day.  Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

------------------------------------------------
Subject: zero matched pairs when processing the AQ inline data
From: Edward Strobach - NOAA Affiliate
Time: Wed Apr 14 11:40:57 2021

Hello,

Latest tests indicate, at least from what I can see, that the issue is
not
necessarily the valid time.  I'm running 31 jobs corresponding to 31
days
in Aug. 2019.  For the 8/22/2019 example you can see that netcdf files
are
created for both ozone and pm for the pb2nc step.  You can see that
here:
*DEBUG 1: Creating NetCDF File:
 /gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r111_v1/pm/prepbufr.pm.20190822.nc
<http://prepbufr.pm.20190822.nc>*

I believe that the following file "
*/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/Archive/201908/Obs/hourly.20190823/aqm.t12z.anowpm.pb.tm024*"
is grabbed in this case because of the 24 hour OBS_WINDOW that is set
(i.e.
-86400 and +86400 centered on 8/23/2019).

After pb2nc is completed the same file is then used for pointstat:
*04/14 17:23:06.994 metplus.PointStat (command_builder.py:548) DEBUG:
Found
file:
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r111_v1/pm/prepbufr.pm.20190822.n*
c

It starts by identifying this file:
*/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r111_v1/aqm.20190821/aqm.t12z.metcro2d.12.nc
<http://aqm.t12z.metcro2d.12.nc>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r111_v1/pm/prepbufr.pm.20190822.nc
<http://prepbufr.pm.20190822.nc>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
-outdir
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/stat/pm/r111_v1*

There are some warning messages related to valid_time but I don't know
what
they mean; it seems more related to a switch in interpolation based on
condition that is met:

*WARNING: RegridInfo::validate() -> Resetting the regridding method
from
"BILIN" to "NEAREST" since the regridding width is 1.*

Eventually as we step through valid times we arrive to the next day's
files:

*DEBUG 1: Forecast File:
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r111_v1/aqm.20190822/aqm.t12z.metcro2d.02.nc
<http://aqm.t12z.metcro2d.02.nc>DEBUG 1: Observation File:
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r111_v1/pm/prepbufr.pm.20190822.nc
<http://prepbufr.pm.20190822.nc>*

Now I'm certain that the valid_time is set correctly.  This is what
I've
always used in the past. The only difference is that I'm using a
different
model data source but I attempt to follow a netcdf writing template
that is
compliant with MET standards.  One other difference is that I'm
processing
24 hours of forecast data while in the past I processed 48 to 72 hours
of
forecast data.

I'm attaching an example conf file that is used for the
pointstat/pb2nc
process.  This shows the correct settings that I wanted to use.

Edward Strobach
EMC/NCEP/NWS/ IMSG
Coding Logic:  if, then, else

On Tue, Apr 13, 2021 at 8:08 PM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:

> Thanks, that's what I'll try to do.
>
>
> Edward Strobach
> EMC/NCEP/NWS/ IMSG
> Coding Logic:  if, then, else
>
>
> On Tue, Apr 13, 2021 at 6:16 PM Julie Prestopnik via RT
<met_help at ucar.edu>
> wrote:
>
>> Hi Edward.
>>
>> It's interesting that you note the settings for OBS_WINDOW* in the
>> >
>> >
>>
/*gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
>> > file* are set to 2700.  I'm not sure what's setting that.
>> >
>> Perhaps the metplus_final.conf file I referred to did not go with
the run
>> for the log I referenced:
>>
>>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210413144310
>>
>> Because I can see in that log file, just above the call to
point_stat, the
>> values for some of the fields are listed and indeed 86400 is used
for
>> OBS_WINDOW_BEGIN:
>>
>> > 04/13 14:43:12.448 metplus.PointStat (command_builder.py:256)
DEBUG:
>> > export CLIMO_MEAN_FILE=""; export CLIMO_STDEV_FILE=""; export
>> FCST_FIELD="{
>> > name\
>> > =\"PMTF\"; level=\"L1\"; },{ name=\"OZCON\"; level=\"A8\"; }";
export
>> >
>>
MET_TMP_DIR="/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CM\
>> > AQ/tmp"; export MODEL="R112_V1"; export OBS_FIELD="{
name=\"COPOPM\";
>> > level=\"A1\"; convert(x) = x * 10^9; },{ name=\"COPO\";
level=\"A8\";
>> > convert(x\
>> > ) = x * 10^9; }"; export OBS_WINDOW_BEGIN="-86400"; export
>> > OBS_WINDOW_END="86400"; export OUTPUT_PREFIX=""; export
>> > POINT_STAT_GRID="[\"FULL\"]"; expo\
>> > rt POINT_STAT_MESSAGE_TYPE="[\"ONLYSF\"]"; export
POINT_STAT_POLY="[]";
>> > export POINT_STAT_STATION_ID="[]"; export
REGRID_TO_GRID="\"G104\"";
>> export
>> > V\
>> > ERIF_MASK="";
>>
>>
>> In a call like the following:
>>
>> > *${MET_PLUS}/ush/master_metplus.py -c
>> > ${MET_PLUS}/parm/use_cases/grid_to_obs/grid_to_obs.conf -c
>> >
${MET_PLUS}/parm/use_cases/grid_to_obs/examples/conus_surface.conf -c
>> > ${MET_PLUS_CONF}/pb2nc_aq.conf -c
${MET_PLUS_CONF}/point_stat_aq.conf -c
>> > ${MET_PLUS_TMP}/${field}.conf -c ${MET_PLUS_CONF}/shared.conf -c
>> > ${MET_PLUS_TMP}/shared.conf_aq -c
${MET_PLUS_CONF}/system_aq.conf*
>>
>>
>> Anything in ${MET_PLUS_CONF}/system_aq.conf would override anything
in any
>> previously listed config file.  The order in which you list your
METplus
>> wrapper config files matters, as it sounds like you are aware of.
Each
>> subsequent config file on the command line will override any values
>> defined
>> in an earlier config file.
>>
>> The issue may be the same with the VALID_BEG and VALID_END in that
the
>> metplus_final.conf file I referred to did not go with the run for
the log
>> I
>> referenced.
>>
>> You could start a clean run and check the time stamp on
>> the metplus_final.conf file, save that off along with the
corresponding
>> metplus log file, check the values and let us know if you see
anything
>> unusual.
>>
>> Let us know what your questions are as you proceed working on
getting the
>> valid times to line up.
>>
>> Julie
>>
>>
>> On Tue, Apr 13, 2021 at 1:17 PM Edward Strobach - NOAA Affiliate
via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99563 >
>> >
>> > Thank you for your help.
>> >
>> > I see from my metplus_final_pb2nc_pointstat.conf located in my
>> >
>> >
>>
"/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/conf/r112_v1/aq"
>> > directory are references to the time window for both the pb2nc
and
>> > pointstat step:
>> >
>> >
>> > *PB2NC_WINDOW_BEGIN = -86400PB2NC_WINDOW_END = 86400*
>> >
>> > and
>> >
>> >
>> > *OBS_WINDOW_BEGIN = -86400OBS_WINDOW_END = 86400*
>> >
>> > It's interesting that you note the settings for OBS_WINDOW* in
the
>> >
>> >
>>
/*gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
>> > file* are set to 2700.  I'm not sure what's setting that.  I
thought it
>> was
>> > being set based on what's in the shared.conf found in my
>> run_metplus_aq.sh
>> > script located here:
>> >
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/scripts.
>> >
>> > I thought that the OBS_WINDOW* definitions set in the shared.conf
would
>> > override other conf files as well.  Running that step looks like
this
>> in my
>> > run_metplus_aq.sh script:
>> >
>> > *${MET_PLUS}/ush/master_metplus.py -c
>> > ${MET_PLUS}/parm/use_cases/grid_to_obs/grid_to_obs.conf -c
>> >
${MET_PLUS}/parm/use_cases/grid_to_obs/examples/conus_surface.conf -c
>> > ${MET_PLUS_CONF}/pb2nc_aq.conf -c
${MET_PLUS_CONF}/point_stat_aq.conf -c
>> > ${MET_PLUS_TMP}/${field}.conf -c ${MET_PLUS_CONF}/shared.conf -c
>> > ${MET_PLUS_TMP}/shared.conf_aq -c
${MET_PLUS_CONF}/system_aq.conf*
>> >
>> > Something else that is troubling from that conf file you
mentioned is
>> the
>> > valid times settings:
>> > VALID_BEG = 20210116
>> > VALID_END = 20210116
>> >
>> > I set that specifically to 20190810.  I didn't think to look
where you
>> did
>> > because the inputs that I assigned to my main run script look
like this:
>> >
>> >
>> >
>>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/scripts/run_metplus_aq.sh
>> > *r112_v1 archive verification LAM 20190810*
>> >
>> > the input date in this case is 20190810 but somehow is not being
>> used..  I
>> > don't know how this could be happening.  Thanks for pointing this
out.
>> >
>> >
>> >
>> > Edward Strobach
>> > EMC/NCEP/NWS/ IMSG
>> > Coding Logic:  if, then, else
>> >
>> >
>> > On Tue, Apr 13, 2021 at 2:20 PM Julie Prestopnik via RT <
>> met_help at ucar.edu
>> > >
>> > wrote:
>> >
>> > > Hi Edward.
>> > >
>> > > I see that you are having trouble getting matched pairs from
>> Point-Stat.
>> > >
>> > > Let's start by taking a look at exactly where the log messages
are
>> > pointing
>> > > us. Looking at one of your log files:
>> > >
>> > >
>> >
>>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210413144310
>> > >
>> > > Looking at the first call of point_stat, I see the following
output
>> > > indicating that all of the 21,468 observations were rejected
due to
>> the
>> > > valid_time, which is why there are no matched pairs for this
case:
>> > >
>> > > > DEBUG 3: Number of matched pairs   = 0
>> > > > DEBUG 3: Observations processed    = 21468
>> > > > DEBUG 3: Rejected: station id      = 0
>> > > > DEBUG 3: Rejected: obs type        = 0
>> > > > DEBUG 3: Rejected: valid time      = 21468
>> > > > DEBUG 3: Rejected: bad obs value   = 0
>> > > > DEBUG 3: Rejected: off the grid    = 0
>> > > > DEBUG 3: Rejected: topography      = 0
>> > > > DEBUG 3: Rejected: level mismatch  = 0
>> > > > DEBUG 3: Rejected: quality marker  = 0
>> > > > DEBUG 3: Rejected: message type    = 0
>> > > > DEBUG 3: Rejected: masking region  = 0
>> > > > DEBUG 3: Rejected: bad fcst value  = 0
>> > > > DEBUG 3: Rejected: bad climo mean  = 0
>> > > > DEBUG 3: Rejected: bad climo stdev = 0
>> > > > DEBUG 3: Rejected: duplicates      = 0
>> > >
>> > >
>> > > This is similar to the situation in the Rejected values that
you had
>> > pasted
>> > > in, except some of those observations were rejected because
they were
>> off
>> > > of the grid, which is why there are no matched pairs for this
case:
>> > >
>> > > > DEBUG 3: Observations processed = 59531
>> > > > DEBUG 3: Rejected: valid time      = 57003
>> > > > DEBUG 3: Rejected: off the grid    = 2528
>> > >
>> > > I wasn't sure which log file that came from, so I just used
your most
>> > > recent log file to get the example above.
>> > >
>> > > You'll want to make sure that when you run Point-Stat, the
forecast
>> > > valid time matches up correctly with the valid time of point
>> > observations.
>> > > I may not be the best person to help with that, but I did find
>> something
>> > > that may be a part of the problem.
>> > >
>> > > Here is the call to point_stat for the case with the 21,468
>> observations
>> > > rejected due to valid time:
>> > > point_stat -v 7 \
>> > >
>> > >
>> >
>>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/\
>> > > aqm.20190809/aqm.t12z.metcro2d.12.nc \
>> > >
>> > >
>> >
>>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r112_v1/pm/
>> > > prepbufr.pm.20190810.nc
>> > >
>> > >
>> >
>>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
>> > > \
>> > > -outdir
>> > >
>> > >
>> >
>>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/stat/pm/r112_v1
>> > >
>> > > I can see that
>> > > the
>> > >
>> >
>>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
>> > > configuration file is being used.
>> > >
>> > > In that configuration file I see:
>> > >
>> > > > obs_window = {
>> > > >    beg = 0;
>> > > >    end = 0;
>> > > > }
>> > >
>> > >
>> > > whereas in many of the other Point-Stat configuration files in
that
>> > > directory I see:
>> > >
>> > > > obs_window = {
>> > > >    beg = ${OBS_WINDOW_BEGIN};
>> > > >    end = ${OBS_WINDOW_END};
>> > > > }
>> > >
>> > >
>> > > In
>> > > your
>> > >
>> >
>>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
>> > > file, I see the following set:
>> > >
>> > > > OBS_WINDOW_BEGIN = -2700
>> > > > OBS_WINDOW_END = 2700
>> > >
>> > >
>> > > and I was wondering if you are intending for METplus to pass on
these
>> > > values in the run to Point-Stat?  If so, you would need to
change the
>> > > values of zero to ${OBS_WINDOW_BEGIN} and ${OBS_WINDOW_END},
>> > respectively.
>> > > Perhaps that is the cause of the zero matched pairs due to
valid time?
>> > >
>> > > I hope this helps!
>> > >
>> > > Julie
>> > >
>> > >
>> > > On Tue, Apr 13, 2021 at 9:27 AM Edward Strobach - NOAA
Affiliate via
>> RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > Tue Apr 13 09:27:33 2021: Request 99563 was acted upon.
>> > > > Transaction: Ticket created by edward.strobach at noaa.gov
>> > > >        Queue: met_help
>> > > >      Subject: zero matched pairs when processing the AQ
inline data
>> > > >        Owner: Nobody
>> > > >   Requestors: edward.strobach at noaa.gov
>> > > >       Status: new
>> > > >  Ticket <URL:
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99563
>> > >
>> > > >
>> > > >
>> > > > Good morning,
>> > > >
>> > > > I'm attempting to process the data from here: "
>> > > >
>> > > >
>> > >
>> >
>>
*/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/aqm.20190810*
>> > > > "
>> > > >
>> > > > The files follow the "aqm.t12z.metcro2d.HH.nc" format and are
>> written
>> > > in a
>> > > > format that METplus can handle.  The processing steps include
pb2nc,
>> > > > pointstat, and statanalysis.  The issue occurs at pointstat.
The
>> pb2nc
>> > > > step runs just fine.
>> > > >
>> > > > Looking in the netcdf files that I created above confirms
that the
>> > > fields I
>> > > > need are there:
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > *>>> Dataset('aqm.t12z.metcro2d.02.nc
>> > > > <http://aqm.t12z.metcro2d.02.nc>')<class
>> > 'netCDF4._netCDF4.Dataset'>root
>> > > > group (NETCDF4 data model, file format HDF5):    FileOrigins:
NA
>> > > > MET_version: V9.1    hemisphere: N    Projection: Lambert
Conformal
>> > > nx:
>> > > > 191    ny: 97 grid_points    dimensions(sizes): lat(97),
lon(191)
>> > > > variables(dimensions): float32 LAT(lat,lon), float32
LON(lat,lon),
>> > > float32
>> > > > WDIR10(lat,lon), float32 WSPD10(lat,lon), float32
TMP(lat,lon),
>> float32
>> > > > DPT(lat,lon), float32 OZCON(lat,lon), float32 PMTF(lat,lon)
>> groups:
>> > *
>> > > >
>> > > > You can also see that the meta data is not overly descriptive
as in
>> the
>> > > > past.  I'm not sure if that is problematic or not.  I do know
that
>> > > whatever
>> > > > is not specified in the file is filled by -9999 or -999
values.
>> > > >
>> > > > Looking at the log file I find the following for every
region, for
>> both
>> > > > PM2.5 and ozone:
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > *DEBUG 2: Processing OZCONA8 versus COPO/A8, for observation
type
>> > AIRNOW,
>> > > > over region WEST, for interpolation method BILIN(4), using 0
matched
>> > > > pairs.DEBUG 3: Number of matched pairs   = 0DEBUG 3:
Observations
>> > > processed
>> > > >    = 59531DEBUG 3: Rejected: station id      = 0DEBUG 3:
Rejected:
>> obs
>> > > type
>> > > >        = 0DEBUG 3: Rejected: valid time      = 57003DEBUG 3:
>> Rejected:
>> > > bad
>> > > > obs value   = 0DEBUG 3: Rejected: off the grid    = 2528DEBUG
3:
>> > > Rejected:
>> > > > topography      = 0DEBUG 3: Rejected: level mismatch  =
0DEBUG 3:
>> > > Rejected:
>> > > > quality marker  = 0DEBUG 3: Rejected: message type    =
0DEBUG 3:
>> > > Rejected:
>> > > > masking region  = 0DEBUG 3: Rejected: bad fcst value  =
0DEBUG 3:
>> > > Rejected:
>> > > > bad climo mean  = 0DEBUG 3: Rejected: bad climo stdev =
0DEBUG 3:
>> > > Rejected:
>> > > > duplicates      = 0*
>> > > >
>> > > > There's no errors to report, rather there are just no matched
pairs.
>> > > Since
>> > > > there are only 24 hours being processed for this data, then
only 24
>> > hours
>> > > > worth of data are being added to the prebufr*nc files.  You
can also
>> > see
>> > > in
>> > > > the log file that the appropriate files are being selected
>> according to
>> > > > date:
>> > > >
>> > > >
>> > > >
>> > > > *DEBUG 1: Forecast File:
>> > > >
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/aqm.20190810/
>> > > > aqm.t12z.metcro2d.10.nc
>> > > > <http://aqm.t12z.metcro2d.10.nc>DEBUG 1: Observation File:
>> > > >
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r112_v1/aq/
>> > > > prepbufr.aqm.20190810.nc
>> > > > <http://prepbufr.aqm.20190810.nc>DEBUG 7: check_nc_data_2d()
count:
>> > > > valid=18527, zero=0, missing=0, 18527 == 18527*
>> > > >
>> > > > I'm having a hard time figuring out why there are no matched
>> pairs.  A
>> > > > couple of other things to note related to an earlier
statement about
>> > meta
>> > > > data..
>> > > >
>> > > > Recall that I said the files weren't overly descriptive.  You
can
>> see
>> > > that
>> > > > the -9999 is being filed for many of the definitions that are
not
>> > > > specifically being set in the created netcdf files:
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > *DEBUG 4: Lambert Conformal Grid Data:DEBUG 4:    hemisphere:
>> NDEBUG 4:
>> > > > scale_lat_1: -9999DEBUG 4:   scale_lat_2: -9999DEBUG 4:
>>  lat_pin:
>> > > > -9999DEBUG 4:       lon_pin: 9999DEBUG 4:         x_pin:
-9999DEBUG
>> 4:
>> > > >     y_pin: -9999DEBUG 4:    lon_orient: 9999DEBUG 4:
d_km:
>> > > > -9999DEBUG 4:          r_km: -9999DEBUG 4:            nx:
191DEBUG
>> 4:
>> > > >      ny: 97DEBUG 4:     so2_angle: 0DEBUG 4:*
>> > > >
>> > > > I'm not sure if that is playing a role here.  Printing out
the ozone
>> > > field
>> > > > from the netcdf file you can see that other information like
>> long_name,
>> > > > level, etc. are given.
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > *>>> Dataset('aqm.t12z.metcro2d.02.nc
>> > > > <http://aqm.t12z.metcro2d.02.nc>').variables['OZCON']<class
>> > > > 'netCDF4._netCDF4.Variable'>float32 OZCON(lat, lon)
long_name:
>> OZCON
>> > > > standard_name: Surface Ozone    units: ppb    level: A8
>> init_time:
>> > > > 20190810_120000    init_time_ut: 1565438400    valid_time:
>> > > 20190810_140000
>> > > >   valid_time_ut: 1565445600unlimited dimensions: current
shape =
>> (97,
>> > > > 191)filling on, default _FillValue of 9.969209968386869e+36
used*
>> > > >
>> > > > Is there something I'm not thinking about here?  I can
provide more
>> > > details
>> > > > if needed.  Hopefully this description is a good start.  I do
recall
>> > > having
>> > > > a similar issue in the past but couldn't locate the exchange
going
>> over
>> > > > this problem.
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > Edward Strobach
>> > > > EMC/NCEP/NWS/ IMSG
>> > > > Coding Logic:  if, then, else
>> > > >
>> > > >
>> > >
>> > > --
>> > > Julie Prestopnik (she/her)
>> > > Software Engineer
>> > > National Center for Atmospheric Research
>> > > Research Applications Laboratory
>> > > Email: jpresto at ucar.edu
>> > >
>> > > My working day may not be your working day.  Please do not feel
>> obliged
>> > to
>> > > reply to this email outside of your normal working hours.
>> > >
>> > >
>> >
>> >
>>
>> --
>> Julie Prestopnik (she/her)
>> Software Engineer
>> National Center for Atmospheric Research
>> Research Applications Laboratory
>> Email: jpresto at ucar.edu
>>
>> My working day may not be your working day.  Please do not feel
obliged to
>> reply to this email outside of your normal working hours.
>>
>>

------------------------------------------------
Subject: zero matched pairs when processing the AQ inline data
From: George McCabe
Time: Wed Apr 14 14:28:44 2021

Hi Edward,

>From looking at this my guess is the missing data values in the grid
information could be the cause of no matched pairs. I'm not sure why
that
would be if the NetCDF files can be read by MET, but that seems off to
me.
I don't have access to WCOSS, but I would check that the grid
information
needed by MET is found in the input data.

Another potential reason for the pairs being rejected for being off
the
grid is the grid definitions to set in the use case. It looks like
PB2NC is
set the mask.grid value to G212 while PointStat is set to use FULL for
the
mask.grid and G104 for the regrid.grid value:

PB2NC_GRID = G212
POINT_STAT_GRID = FULL
POINT_STAT_REGRID_TO_GRID = G104

I would try to resolve these values so that the desired grid is used
consistently.

I hope that helps! Let me know if you are still having issues.

Thanks,
George

On Wed, Apr 14, 2021 at 11:40 AM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99563 >
>
> Hello,
>
> Latest tests indicate, at least from what I can see, that the issue
is not
> necessarily the valid time.  I'm running 31 jobs corresponding to 31
days
> in Aug. 2019.  For the 8/22/2019 example you can see that netcdf
files are
> created for both ozone and pm for the pb2nc step.  You can see that
here:
> *DEBUG 1: Creating NetCDF File:
>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r111_v1/pm/
> prepbufr.pm.20190822.nc
> <http://prepbufr.pm.20190822.nc>*
>
> I believe that the following file "
>
>
*/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/Archive/201908/Obs/hourly.20190823/aqm.t12z.anowpm.pb.tm024*"
> is grabbed in this case because of the 24 hour OBS_WINDOW that is
set (i.e.
> -86400 and +86400 centered on 8/23/2019).
>
> After pb2nc is completed the same file is then used for pointstat:
> *04/14 17:23:06.994 metplus.PointStat (command_builder.py:548)
DEBUG: Found
> file:
>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r111_v1/pm/
> prepbufr.pm.20190822.n*
> c
>
> It starts by identifying this file:
>
>
*/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r111_v1/aqm.20190821/
> aqm.t12z.metcro2d.12.nc
> <http://aqm.t12z.metcro2d.12.nc>
>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r111_v1/pm/
> prepbufr.pm.20190822.nc
> <http://prepbufr.pm.20190822.nc>
>
>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
> -outdir
>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/stat/pm/r111_v1*
>
> There are some warning messages related to valid_time but I don't
know what
> they mean; it seems more related to a switch in interpolation based
on
> condition that is met:
>
> *WARNING: RegridInfo::validate() -> Resetting the regridding method
from
> "BILIN" to "NEAREST" since the regridding width is 1.*
>
> Eventually as we step through valid times we arrive to the next
day's
> files:
>
> *DEBUG 1: Forecast File:
>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r111_v1/aqm.20190822/
> aqm.t12z.metcro2d.02.nc
> <http://aqm.t12z.metcro2d.02.nc>DEBUG 1: Observation File:
>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r111_v1/pm/
> prepbufr.pm.20190822.nc
> <http://prepbufr.pm.20190822.nc>*
>
> Now I'm certain that the valid_time is set correctly.  This is what
I've
> always used in the past. The only difference is that I'm using a
different
> model data source but I attempt to follow a netcdf writing template
that is
> compliant with MET standards.  One other difference is that I'm
processing
> 24 hours of forecast data while in the past I processed 48 to 72
hours of
> forecast data.
>
> I'm attaching an example conf file that is used for the
pointstat/pb2nc
> process.  This shows the correct settings that I wanted to use.
>
> Edward Strobach
> EMC/NCEP/NWS/ IMSG
> Coding Logic:  if, then, else
>
>
> On Tue, Apr 13, 2021 at 8:08 PM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
> > Thanks, that's what I'll try to do.
> >
> >
> > Edward Strobach
> > EMC/NCEP/NWS/ IMSG
> > Coding Logic:  if, then, else
> >
> >
> > On Tue, Apr 13, 2021 at 6:16 PM Julie Prestopnik via RT <
> met_help at ucar.edu>
> > wrote:
> >
> >> Hi Edward.
> >>
> >> It's interesting that you note the settings for OBS_WINDOW* in
the
> >> >
> >> >
> >>
>
/*gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
> >> > file* are set to 2700.  I'm not sure what's setting that.
> >> >
> >> Perhaps the metplus_final.conf file I referred to did not go with
the
> run
> >> for the log I referenced:
> >>
> >>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210413144310
> >>
> >> Because I can see in that log file, just above the call to
point_stat,
> the
> >> values for some of the fields are listed and indeed 86400 is used
for
> >> OBS_WINDOW_BEGIN:
> >>
> >> > 04/13 14:43:12.448 metplus.PointStat (command_builder.py:256)
DEBUG:
> >> > export CLIMO_MEAN_FILE=""; export CLIMO_STDEV_FILE=""; export
> >> FCST_FIELD="{
> >> > name\
> >> > =\"PMTF\"; level=\"L1\"; },{ name=\"OZCON\"; level=\"A8\"; }";
export
> >> >
> >>
>
MET_TMP_DIR="/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CM\
> >> > AQ/tmp"; export MODEL="R112_V1"; export OBS_FIELD="{
name=\"COPOPM\";
> >> > level=\"A1\"; convert(x) = x * 10^9; },{ name=\"COPO\";
level=\"A8\";
> >> > convert(x\
> >> > ) = x * 10^9; }"; export OBS_WINDOW_BEGIN="-86400"; export
> >> > OBS_WINDOW_END="86400"; export OUTPUT_PREFIX=""; export
> >> > POINT_STAT_GRID="[\"FULL\"]"; expo\
> >> > rt POINT_STAT_MESSAGE_TYPE="[\"ONLYSF\"]"; export
> POINT_STAT_POLY="[]";
> >> > export POINT_STAT_STATION_ID="[]"; export
REGRID_TO_GRID="\"G104\"";
> >> export
> >> > V\
> >> > ERIF_MASK="";
> >>
> >>
> >> In a call like the following:
> >>
> >> > *${MET_PLUS}/ush/master_metplus.py -c
> >> > ${MET_PLUS}/parm/use_cases/grid_to_obs/grid_to_obs.conf -c
> >> >
${MET_PLUS}/parm/use_cases/grid_to_obs/examples/conus_surface.conf -c
> >> > ${MET_PLUS_CONF}/pb2nc_aq.conf -c
${MET_PLUS_CONF}/point_stat_aq.conf
> -c
> >> > ${MET_PLUS_TMP}/${field}.conf -c ${MET_PLUS_CONF}/shared.conf
-c
> >> > ${MET_PLUS_TMP}/shared.conf_aq -c
${MET_PLUS_CONF}/system_aq.conf*
> >>
> >>
> >> Anything in ${MET_PLUS_CONF}/system_aq.conf would override
anything in
> any
> >> previously listed config file.  The order in which you list your
METplus
> >> wrapper config files matters, as it sounds like you are aware of.
Each
> >> subsequent config file on the command line will override any
values
> >> defined
> >> in an earlier config file.
> >>
> >> The issue may be the same with the VALID_BEG and VALID_END in
that the
> >> metplus_final.conf file I referred to did not go with the run for
the
> log
> >> I
> >> referenced.
> >>
> >> You could start a clean run and check the time stamp on
> >> the metplus_final.conf file, save that off along with the
corresponding
> >> metplus log file, check the values and let us know if you see
anything
> >> unusual.
> >>
> >> Let us know what your questions are as you proceed working on
getting
> the
> >> valid times to line up.
> >>
> >> Julie
> >>
> >>
> >> On Tue, Apr 13, 2021 at 1:17 PM Edward Strobach - NOAA Affiliate
via RT
> <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99563 >
> >> >
> >> > Thank you for your help.
> >> >
> >> > I see from my metplus_final_pb2nc_pointstat.conf located in my
> >> >
> >> >
> >>
>
"/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/conf/r112_v1/aq"
> >> > directory are references to the time window for both the pb2nc
and
> >> > pointstat step:
> >> >
> >> >
> >> > *PB2NC_WINDOW_BEGIN = -86400PB2NC_WINDOW_END = 86400*
> >> >
> >> > and
> >> >
> >> >
> >> > *OBS_WINDOW_BEGIN = -86400OBS_WINDOW_END = 86400*
> >> >
> >> > It's interesting that you note the settings for OBS_WINDOW* in
the
> >> >
> >> >
> >>
>
/*gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
> >> > file* are set to 2700.  I'm not sure what's setting that.  I
thought
> it
> >> was
> >> > being set based on what's in the shared.conf found in my
> >> run_metplus_aq.sh
> >> > script located here:
> >> >
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/scripts.
> >> >
> >> > I thought that the OBS_WINDOW* definitions set in the
shared.conf
> would
> >> > override other conf files as well.  Running that step looks
like this
> >> in my
> >> > run_metplus_aq.sh script:
> >> >
> >> > *${MET_PLUS}/ush/master_metplus.py -c
> >> > ${MET_PLUS}/parm/use_cases/grid_to_obs/grid_to_obs.conf -c
> >> >
${MET_PLUS}/parm/use_cases/grid_to_obs/examples/conus_surface.conf -c
> >> > ${MET_PLUS_CONF}/pb2nc_aq.conf -c
${MET_PLUS_CONF}/point_stat_aq.conf
> -c
> >> > ${MET_PLUS_TMP}/${field}.conf -c ${MET_PLUS_CONF}/shared.conf
-c
> >> > ${MET_PLUS_TMP}/shared.conf_aq -c
${MET_PLUS_CONF}/system_aq.conf*
> >> >
> >> > Something else that is troubling from that conf file you
mentioned is
> >> the
> >> > valid times settings:
> >> > VALID_BEG = 20210116
> >> > VALID_END = 20210116
> >> >
> >> > I set that specifically to 20190810.  I didn't think to look
where you
> >> did
> >> > because the inputs that I assigned to my main run script look
like
> this:
> >> >
> >> >
> >> >
> >>
>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/scripts/run_metplus_aq.sh
> >> > *r112_v1 archive verification LAM 20190810*
> >> >
> >> > the input date in this case is 20190810 but somehow is not
being
> >> used..  I
> >> > don't know how this could be happening.  Thanks for pointing
this out.
> >> >
> >> >
> >> >
> >> > Edward Strobach
> >> > EMC/NCEP/NWS/ IMSG
> >> > Coding Logic:  if, then, else
> >> >
> >> >
> >> > On Tue, Apr 13, 2021 at 2:20 PM Julie Prestopnik via RT <
> >> met_help at ucar.edu
> >> > >
> >> > wrote:
> >> >
> >> > > Hi Edward.
> >> > >
> >> > > I see that you are having trouble getting matched pairs from
> >> Point-Stat.
> >> > >
> >> > > Let's start by taking a look at exactly where the log
messages are
> >> > pointing
> >> > > us. Looking at one of your log files:
> >> > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210413144310
> >> > >
> >> > > Looking at the first call of point_stat, I see the following
output
> >> > > indicating that all of the 21,468 observations were rejected
due to
> >> the
> >> > > valid_time, which is why there are no matched pairs for this
case:
> >> > >
> >> > > > DEBUG 3: Number of matched pairs   = 0
> >> > > > DEBUG 3: Observations processed    = 21468
> >> > > > DEBUG 3: Rejected: station id      = 0
> >> > > > DEBUG 3: Rejected: obs type        = 0
> >> > > > DEBUG 3: Rejected: valid time      = 21468
> >> > > > DEBUG 3: Rejected: bad obs value   = 0
> >> > > > DEBUG 3: Rejected: off the grid    = 0
> >> > > > DEBUG 3: Rejected: topography      = 0
> >> > > > DEBUG 3: Rejected: level mismatch  = 0
> >> > > > DEBUG 3: Rejected: quality marker  = 0
> >> > > > DEBUG 3: Rejected: message type    = 0
> >> > > > DEBUG 3: Rejected: masking region  = 0
> >> > > > DEBUG 3: Rejected: bad fcst value  = 0
> >> > > > DEBUG 3: Rejected: bad climo mean  = 0
> >> > > > DEBUG 3: Rejected: bad climo stdev = 0
> >> > > > DEBUG 3: Rejected: duplicates      = 0
> >> > >
> >> > >
> >> > > This is similar to the situation in the Rejected values that
you had
> >> > pasted
> >> > > in, except some of those observations were rejected because
they
> were
> >> off
> >> > > of the grid, which is why there are no matched pairs for this
case:
> >> > >
> >> > > > DEBUG 3: Observations processed = 59531
> >> > > > DEBUG 3: Rejected: valid time      = 57003
> >> > > > DEBUG 3: Rejected: off the grid    = 2528
> >> > >
> >> > > I wasn't sure which log file that came from, so I just used
your
> most
> >> > > recent log file to get the example above.
> >> > >
> >> > > You'll want to make sure that when you run Point-Stat, the
forecast
> >> > > valid time matches up correctly with the valid time of point
> >> > observations.
> >> > > I may not be the best person to help with that, but I did
find
> >> something
> >> > > that may be a part of the problem.
> >> > >
> >> > > Here is the call to point_stat for the case with the 21,468
> >> observations
> >> > > rejected due to valid time:
> >> > > point_stat -v 7 \
> >> > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/\
> >> > > aqm.20190809/aqm.t12z.metcro2d.12.nc \
> >> > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r112_v1/pm/
> >> > > prepbufr.pm.20190810.nc
> >> > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
> >> > > \
> >> > > -outdir
> >> > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/stat/pm/r112_v1
> >> > >
> >> > > I can see that
> >> > > the
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
> >> > > configuration file is being used.
> >> > >
> >> > > In that configuration file I see:
> >> > >
> >> > > > obs_window = {
> >> > > >    beg = 0;
> >> > > >    end = 0;
> >> > > > }
> >> > >
> >> > >
> >> > > whereas in many of the other Point-Stat configuration files
in that
> >> > > directory I see:
> >> > >
> >> > > > obs_window = {
> >> > > >    beg = ${OBS_WINDOW_BEGIN};
> >> > > >    end = ${OBS_WINDOW_END};
> >> > > > }
> >> > >
> >> > >
> >> > > In
> >> > > your
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
> >> > > file, I see the following set:
> >> > >
> >> > > > OBS_WINDOW_BEGIN = -2700
> >> > > > OBS_WINDOW_END = 2700
> >> > >
> >> > >
> >> > > and I was wondering if you are intending for METplus to pass
on
> these
> >> > > values in the run to Point-Stat?  If so, you would need to
change
> the
> >> > > values of zero to ${OBS_WINDOW_BEGIN} and ${OBS_WINDOW_END},
> >> > respectively.
> >> > > Perhaps that is the cause of the zero matched pairs due to
valid
> time?
> >> > >
> >> > > I hope this helps!
> >> > >
> >> > > Julie
> >> > >
> >> > >
> >> > > On Tue, Apr 13, 2021 at 9:27 AM Edward Strobach - NOAA
Affiliate via
> >> RT <
> >> > > met_help at ucar.edu> wrote:
> >> > >
> >> > > >
> >> > > > Tue Apr 13 09:27:33 2021: Request 99563 was acted upon.
> >> > > > Transaction: Ticket created by edward.strobach at noaa.gov
> >> > > >        Queue: met_help
> >> > > >      Subject: zero matched pairs when processing the AQ
inline
> data
> >> > > >        Owner: Nobody
> >> > > >   Requestors: edward.strobach at noaa.gov
> >> > > >       Status: new
> >> > > >  Ticket <URL:
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99563
> >> > >
> >> > > >
> >> > > >
> >> > > > Good morning,
> >> > > >
> >> > > > I'm attempting to process the data from here: "
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
>
*/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/aqm.20190810*
> >> > > > "
> >> > > >
> >> > > > The files follow the "aqm.t12z.metcro2d.HH.nc" format and
are
> >> written
> >> > > in a
> >> > > > format that METplus can handle.  The processing steps
include
> pb2nc,
> >> > > > pointstat, and statanalysis.  The issue occurs at
pointstat.  The
> >> pb2nc
> >> > > > step runs just fine.
> >> > > >
> >> > > > Looking in the netcdf files that I created above confirms
that the
> >> > > fields I
> >> > > > need are there:
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > *>>> Dataset('aqm.t12z.metcro2d.02.nc
> >> > > > <http://aqm.t12z.metcro2d.02.nc>')<class
> >> > 'netCDF4._netCDF4.Dataset'>root
> >> > > > group (NETCDF4 data model, file format HDF5):
FileOrigins: NA
> >> > > > MET_version: V9.1    hemisphere: N    Projection: Lambert
> Conformal
> >> > > nx:
> >> > > > 191    ny: 97 grid_points    dimensions(sizes): lat(97),
lon(191)
> >> > > > variables(dimensions): float32 LAT(lat,lon), float32
LON(lat,lon),
> >> > > float32
> >> > > > WDIR10(lat,lon), float32 WSPD10(lat,lon), float32
TMP(lat,lon),
> >> float32
> >> > > > DPT(lat,lon), float32 OZCON(lat,lon), float32 PMTF(lat,lon)
> >> groups:
> >> > *
> >> > > >
> >> > > > You can also see that the meta data is not overly
descriptive as
> in
> >> the
> >> > > > past.  I'm not sure if that is problematic or not.  I do
know that
> >> > > whatever
> >> > > > is not specified in the file is filled by -9999 or -999
values.
> >> > > >
> >> > > > Looking at the log file I find the following for every
region, for
> >> both
> >> > > > PM2.5 and ozone:
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > *DEBUG 2: Processing OZCONA8 versus COPO/A8, for
observation type
> >> > AIRNOW,
> >> > > > over region WEST, for interpolation method BILIN(4), using
0
> matched
> >> > > > pairs.DEBUG 3: Number of matched pairs   = 0DEBUG 3:
Observations
> >> > > processed
> >> > > >    = 59531DEBUG 3: Rejected: station id      = 0DEBUG 3:
Rejected:
> >> obs
> >> > > type
> >> > > >        = 0DEBUG 3: Rejected: valid time      = 57003DEBUG
3:
> >> Rejected:
> >> > > bad
> >> > > > obs value   = 0DEBUG 3: Rejected: off the grid    =
2528DEBUG 3:
> >> > > Rejected:
> >> > > > topography      = 0DEBUG 3: Rejected: level mismatch  =
0DEBUG 3:
> >> > > Rejected:
> >> > > > quality marker  = 0DEBUG 3: Rejected: message type    =
0DEBUG 3:
> >> > > Rejected:
> >> > > > masking region  = 0DEBUG 3: Rejected: bad fcst value  =
0DEBUG 3:
> >> > > Rejected:
> >> > > > bad climo mean  = 0DEBUG 3: Rejected: bad climo stdev =
0DEBUG 3:
> >> > > Rejected:
> >> > > > duplicates      = 0*
> >> > > >
> >> > > > There's no errors to report, rather there are just no
matched
> pairs.
> >> > > Since
> >> > > > there are only 24 hours being processed for this data, then
only
> 24
> >> > hours
> >> > > > worth of data are being added to the prebufr*nc files.  You
can
> also
> >> > see
> >> > > in
> >> > > > the log file that the appropriate files are being selected
> >> according to
> >> > > > date:
> >> > > >
> >> > > >
> >> > > >
> >> > > > *DEBUG 1: Forecast File:
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/aqm.20190810/
> >> > > > aqm.t12z.metcro2d.10.nc
> >> > > > <http://aqm.t12z.metcro2d.10.nc>DEBUG 1: Observation File:
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r112_v1/aq/
> >> > > > prepbufr.aqm.20190810.nc
> >> > > > <http://prepbufr.aqm.20190810.nc>DEBUG 7:
check_nc_data_2d()
> count:
> >> > > > valid=18527, zero=0, missing=0, 18527 == 18527*
> >> > > >
> >> > > > I'm having a hard time figuring out why there are no
matched
> >> pairs.  A
> >> > > > couple of other things to note related to an earlier
statement
> about
> >> > meta
> >> > > > data..
> >> > > >
> >> > > > Recall that I said the files weren't overly descriptive.
You can
> >> see
> >> > > that
> >> > > > the -9999 is being filed for many of the definitions that
are not
> >> > > > specifically being set in the created netcdf files:
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > *DEBUG 4: Lambert Conformal Grid Data:DEBUG 4:
hemisphere:
> >> NDEBUG 4:
> >> > > > scale_lat_1: -9999DEBUG 4:   scale_lat_2: -9999DEBUG 4:
> >>  lat_pin:
> >> > > > -9999DEBUG 4:       lon_pin: 9999DEBUG 4:         x_pin:
> -9999DEBUG
> >> 4:
> >> > > >     y_pin: -9999DEBUG 4:    lon_orient: 9999DEBUG 4:
> d_km:
> >> > > > -9999DEBUG 4:          r_km: -9999DEBUG 4:            nx:
191DEBUG
> >> 4:
> >> > > >      ny: 97DEBUG 4:     so2_angle: 0DEBUG 4:*
> >> > > >
> >> > > > I'm not sure if that is playing a role here.  Printing out
the
> ozone
> >> > > field
> >> > > > from the netcdf file you can see that other information
like
> >> long_name,
> >> > > > level, etc. are given.
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > *>>> Dataset('aqm.t12z.metcro2d.02.nc
> >> > > > <http://aqm.t12z.metcro2d.02.nc>').variables['OZCON']<class
> >> > > > 'netCDF4._netCDF4.Variable'>float32 OZCON(lat, lon)
long_name:
> >> OZCON
> >> > > > standard_name: Surface Ozone    units: ppb    level: A8
> >> init_time:
> >> > > > 20190810_120000    init_time_ut: 1565438400    valid_time:
> >> > > 20190810_140000
> >> > > >   valid_time_ut: 1565445600unlimited dimensions: current
shape =
> >> (97,
> >> > > > 191)filling on, default _FillValue of 9.969209968386869e+36
used*
> >> > > >
> >> > > > Is there something I'm not thinking about here?  I can
provide
> more
> >> > > details
> >> > > > if needed.  Hopefully this description is a good start.  I
do
> recall
> >> > > having
> >> > > > a similar issue in the past but couldn't locate the
exchange going
> >> over
> >> > > > this problem.
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > Edward Strobach
> >> > > > EMC/NCEP/NWS/ IMSG
> >> > > > Coding Logic:  if, then, else
> >> > > >
> >> > > >
> >> > >
> >> > > --
> >> > > Julie Prestopnik (she/her)
> >> > > Software Engineer
> >> > > National Center for Atmospheric Research
> >> > > Research Applications Laboratory
> >> > > Email: jpresto at ucar.edu
> >> > >
> >> > > My working day may not be your working day.  Please do not
feel
> >> obliged
> >> > to
> >> > > reply to this email outside of your normal working hours.
> >> > >
> >> > >
> >> >
> >> >
> >>
> >> --
> >> Julie Prestopnik (she/her)
> >> Software Engineer
> >> National Center for Atmospheric Research
> >> Research Applications Laboratory
> >> Email: jpresto at ucar.edu
> >>
> >> My working day may not be your working day.  Please do not feel
obliged
> to
> >> reply to this email outside of your normal working hours.
> >>
> >>
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: zero matched pairs when processing the AQ inline data
From: Edward Strobach - NOAA Affiliate
Time: Wed Apr 14 19:19:10 2021

Thanks, but I've used these settings for quite a long time and it
didn't
result in these issues before.  I just looked at an example where it
did
work and the configure file at the pb2nc grid set to G212 while
pointstat
was FULL.  I'm going to have to look at other possible explanations.
I did
note originally
that the metadata information is not complete in these files compared
to
cases that work.  For instance I have this for the cases that are
failing:

*>>> Dataset('aqm.t12z.metcro2d.02.nc
<http://aqm.t12z.metcro2d.02.nc>')<class
'netCDF4._netCDF4.Dataset'>root
group (NETCDF4 data model, file format HDF5):    FileOrigins: NA
MET_version: V9.1    hemisphere: N    Projection: Lambert Conformal
nx:
191    ny: 97 grid_points    dimensions(sizes): lat(97), lon(191)
variables(dimensions): float32 LAT(lat,lon), float32 LON(lat,lon),
float32
WDIR10(lat,lon), float32 WSPD10(lat,lon), float32 TMP(lat,lon),
float32
DPT(lat,lon), float32 OZCON(lat,lon), float32 PMTF(lat,lon)    groups:
*

Because not everything was specified in the metadata structure I was
getting filler values for those definitions. See the printout below:

*DEBUG 4: Lambert Conformal Grid Data:DEBUG 4:    hemisphere: NDEBUG
4:
scale_lat_1: -9999DEBUG 4:   scale_lat_2: -9999DEBUG 4:       lat_pin:
-9999DEBUG 4:       lon_pin: 9999DEBUG 4:         x_pin: -9999DEBUG 4:
    y_pin: -9999DEBUG 4:    lon_orient: 9999DEBUG 4:          d_km:
-9999DEBUG 4:          r_km: -9999DEBUG 4:            nx: 191DEBUG 4:
     ny: 97DEBUG 4:     so2_angle: 0DEBUG 4:*

In the case that worked I had all these quantities specified so there
were
no filler values.  I'm not sure, but I think this could be an issue. I
will
try to look for these values but I can't guarantee that I'll find them
all.

Edward Strobach
EMC/NCEP/NWS/ IMSG
Coding Logic:  if, then, else

On Wed, Apr 14, 2021 at 4:28 PM George McCabe via RT
<met_help at ucar.edu>
wrote:

> Hi Edward,
>
> From looking at this my guess is the missing data values in the grid
> information could be the cause of no matched pairs. I'm not sure why
that
> would be if the NetCDF files can be read by MET, but that seems off
to me.
> I don't have access to WCOSS, but I would check that the grid
information
> needed by MET is found in the input data.
>
> Another potential reason for the pairs being rejected for being off
the
> grid is the grid definitions to set in the use case. It looks like
PB2NC is
> set the mask.grid value to G212 while PointStat is set to use FULL
for the
> mask.grid and G104 for the regrid.grid value:
>
> PB2NC_GRID = G212
> POINT_STAT_GRID = FULL
> POINT_STAT_REGRID_TO_GRID = G104
>
> I would try to resolve these values so that the desired grid is used
> consistently.
>
> I hope that helps! Let me know if you are still having issues.
>
> Thanks,
> George
>
> On Wed, Apr 14, 2021 at 11:40 AM Edward Strobach - NOAA Affiliate
via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99563 >
> >
> > Hello,
> >
> > Latest tests indicate, at least from what I can see, that the
issue is
> not
> > necessarily the valid time.  I'm running 31 jobs corresponding to
31 days
> > in Aug. 2019.  For the 8/22/2019 example you can see that netcdf
files
> are
> > created for both ozone and pm for the pb2nc step.  You can see
that here:
> > *DEBUG 1: Creating NetCDF File:
> >
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r111_v1/pm/
> > prepbufr.pm.20190822.nc
> > <http://prepbufr.pm.20190822.nc>*
> >
> > I believe that the following file "
> >
> >
>
*/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/Archive/201908/Obs/hourly.20190823/aqm.t12z.anowpm.pb.tm024*"
> > is grabbed in this case because of the 24 hour OBS_WINDOW that is
set
> (i.e.
> > -86400 and +86400 centered on 8/23/2019).
> >
> > After pb2nc is completed the same file is then used for pointstat:
> > *04/14 17:23:06.994 metplus.PointStat (command_builder.py:548)
DEBUG:
> Found
> > file:
> >
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r111_v1/pm/
> > prepbufr.pm.20190822.n*
> > c
> >
> > It starts by identifying this file:
> >
> >
>
*/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r111_v1/aqm.20190821/
> > aqm.t12z.metcro2d.12.nc
> > <http://aqm.t12z.metcro2d.12.nc>
> >
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r111_v1/pm/
> > prepbufr.pm.20190822.nc
> > <http://prepbufr.pm.20190822.nc>
> >
> >
>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
> > -outdir
> >
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/stat/pm/r111_v1*
> >
> > There are some warning messages related to valid_time but I don't
know
> what
> > they mean; it seems more related to a switch in interpolation
based on
> > condition that is met:
> >
> > *WARNING: RegridInfo::validate() -> Resetting the regridding
method from
> > "BILIN" to "NEAREST" since the regridding width is 1.*
> >
> > Eventually as we step through valid times we arrive to the next
day's
> > files:
> >
> > *DEBUG 1: Forecast File:
> >
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r111_v1/aqm.20190822/
> > aqm.t12z.metcro2d.02.nc
> > <http://aqm.t12z.metcro2d.02.nc>DEBUG 1: Observation File:
> >
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r111_v1/pm/
> > prepbufr.pm.20190822.nc
> > <http://prepbufr.pm.20190822.nc>*
> >
> > Now I'm certain that the valid_time is set correctly.  This is
what I've
> > always used in the past. The only difference is that I'm using a
> different
> > model data source but I attempt to follow a netcdf writing
template that
> is
> > compliant with MET standards.  One other difference is that I'm
> processing
> > 24 hours of forecast data while in the past I processed 48 to 72
hours of
> > forecast data.
> >
> > I'm attaching an example conf file that is used for the
pointstat/pb2nc
> > process.  This shows the correct settings that I wanted to use.
> >
> > Edward Strobach
> > EMC/NCEP/NWS/ IMSG
> > Coding Logic:  if, then, else
> >
> >
> > On Tue, Apr 13, 2021 at 8:08 PM Edward Strobach - NOAA Affiliate <
> > edward.strobach at noaa.gov> wrote:
> >
> > > Thanks, that's what I'll try to do.
> > >
> > >
> > > Edward Strobach
> > > EMC/NCEP/NWS/ IMSG
> > > Coding Logic:  if, then, else
> > >
> > >
> > > On Tue, Apr 13, 2021 at 6:16 PM Julie Prestopnik via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > >> Hi Edward.
> > >>
> > >> It's interesting that you note the settings for OBS_WINDOW* in
the
> > >> >
> > >> >
> > >>
> >
>
/*gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
> > >> > file* are set to 2700.  I'm not sure what's setting that.
> > >> >
> > >> Perhaps the metplus_final.conf file I referred to did not go
with the
> > run
> > >> for the log I referenced:
> > >>
> > >>
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210413144310
> > >>
> > >> Because I can see in that log file, just above the call to
point_stat,
> > the
> > >> values for some of the fields are listed and indeed 86400 is
used for
> > >> OBS_WINDOW_BEGIN:
> > >>
> > >> > 04/13 14:43:12.448 metplus.PointStat (command_builder.py:256)
DEBUG:
> > >> > export CLIMO_MEAN_FILE=""; export CLIMO_STDEV_FILE=""; export
> > >> FCST_FIELD="{
> > >> > name\
> > >> > =\"PMTF\"; level=\"L1\"; },{ name=\"OZCON\"; level=\"A8\";
}";
> export
> > >> >
> > >>
> >
>
MET_TMP_DIR="/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CM\
> > >> > AQ/tmp"; export MODEL="R112_V1"; export OBS_FIELD="{
> name=\"COPOPM\";
> > >> > level=\"A1\"; convert(x) = x * 10^9; },{ name=\"COPO\";
> level=\"A8\";
> > >> > convert(x\
> > >> > ) = x * 10^9; }"; export OBS_WINDOW_BEGIN="-86400"; export
> > >> > OBS_WINDOW_END="86400"; export OUTPUT_PREFIX=""; export
> > >> > POINT_STAT_GRID="[\"FULL\"]"; expo\
> > >> > rt POINT_STAT_MESSAGE_TYPE="[\"ONLYSF\"]"; export
> > POINT_STAT_POLY="[]";
> > >> > export POINT_STAT_STATION_ID="[]"; export
REGRID_TO_GRID="\"G104\"";
> > >> export
> > >> > V\
> > >> > ERIF_MASK="";
> > >>
> > >>
> > >> In a call like the following:
> > >>
> > >> > *${MET_PLUS}/ush/master_metplus.py -c
> > >> > ${MET_PLUS}/parm/use_cases/grid_to_obs/grid_to_obs.conf -c
> > >> >
${MET_PLUS}/parm/use_cases/grid_to_obs/examples/conus_surface.conf
> -c
> > >> > ${MET_PLUS_CONF}/pb2nc_aq.conf -c
> ${MET_PLUS_CONF}/point_stat_aq.conf
> > -c
> > >> > ${MET_PLUS_TMP}/${field}.conf -c ${MET_PLUS_CONF}/shared.conf
-c
> > >> > ${MET_PLUS_TMP}/shared.conf_aq -c
${MET_PLUS_CONF}/system_aq.conf*
> > >>
> > >>
> > >> Anything in ${MET_PLUS_CONF}/system_aq.conf would override
anything in
> > any
> > >> previously listed config file.  The order in which you list
your
> METplus
> > >> wrapper config files matters, as it sounds like you are aware
of. Each
> > >> subsequent config file on the command line will override any
values
> > >> defined
> > >> in an earlier config file.
> > >>
> > >> The issue may be the same with the VALID_BEG and VALID_END in
that the
> > >> metplus_final.conf file I referred to did not go with the run
for the
> > log
> > >> I
> > >> referenced.
> > >>
> > >> You could start a clean run and check the time stamp on
> > >> the metplus_final.conf file, save that off along with the
> corresponding
> > >> metplus log file, check the values and let us know if you see
anything
> > >> unusual.
> > >>
> > >> Let us know what your questions are as you proceed working on
getting
> > the
> > >> valid times to line up.
> > >>
> > >> Julie
> > >>
> > >>
> > >> On Tue, Apr 13, 2021 at 1:17 PM Edward Strobach - NOAA
Affiliate via
> RT
> > <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >> >
> > >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99563
>
> > >> >
> > >> > Thank you for your help.
> > >> >
> > >> > I see from my metplus_final_pb2nc_pointstat.conf located in
my
> > >> >
> > >> >
> > >>
> >
>
"/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/conf/r112_v1/aq"
> > >> > directory are references to the time window for both the
pb2nc and
> > >> > pointstat step:
> > >> >
> > >> >
> > >> > *PB2NC_WINDOW_BEGIN = -86400PB2NC_WINDOW_END = 86400*
> > >> >
> > >> > and
> > >> >
> > >> >
> > >> > *OBS_WINDOW_BEGIN = -86400OBS_WINDOW_END = 86400*
> > >> >
> > >> > It's interesting that you note the settings for OBS_WINDOW*
in the
> > >> >
> > >> >
> > >>
> >
>
/*gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
> > >> > file* are set to 2700.  I'm not sure what's setting that.  I
thought
> > it
> > >> was
> > >> > being set based on what's in the shared.conf found in my
> > >> run_metplus_aq.sh
> > >> > script located here:
> > >> >
>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/scripts.
> > >> >
> > >> > I thought that the OBS_WINDOW* definitions set in the
shared.conf
> > would
> > >> > override other conf files as well.  Running that step looks
like
> this
> > >> in my
> > >> > run_metplus_aq.sh script:
> > >> >
> > >> > *${MET_PLUS}/ush/master_metplus.py -c
> > >> > ${MET_PLUS}/parm/use_cases/grid_to_obs/grid_to_obs.conf -c
> > >> >
${MET_PLUS}/parm/use_cases/grid_to_obs/examples/conus_surface.conf
> -c
> > >> > ${MET_PLUS_CONF}/pb2nc_aq.conf -c
> ${MET_PLUS_CONF}/point_stat_aq.conf
> > -c
> > >> > ${MET_PLUS_TMP}/${field}.conf -c ${MET_PLUS_CONF}/shared.conf
-c
> > >> > ${MET_PLUS_TMP}/shared.conf_aq -c
${MET_PLUS_CONF}/system_aq.conf*
> > >> >
> > >> > Something else that is troubling from that conf file you
mentioned
> is
> > >> the
> > >> > valid times settings:
> > >> > VALID_BEG = 20210116
> > >> > VALID_END = 20210116
> > >> >
> > >> > I set that specifically to 20190810.  I didn't think to look
where
> you
> > >> did
> > >> > because the inputs that I assigned to my main run script look
like
> > this:
> > >> >
> > >> >
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/scripts/run_metplus_aq.sh
> > >> > *r112_v1 archive verification LAM 20190810*
> > >> >
> > >> > the input date in this case is 20190810 but somehow is not
being
> > >> used..  I
> > >> > don't know how this could be happening.  Thanks for pointing
this
> out.
> > >> >
> > >> >
> > >> >
> > >> > Edward Strobach
> > >> > EMC/NCEP/NWS/ IMSG
> > >> > Coding Logic:  if, then, else
> > >> >
> > >> >
> > >> > On Tue, Apr 13, 2021 at 2:20 PM Julie Prestopnik via RT <
> > >> met_help at ucar.edu
> > >> > >
> > >> > wrote:
> > >> >
> > >> > > Hi Edward.
> > >> > >
> > >> > > I see that you are having trouble getting matched pairs
from
> > >> Point-Stat.
> > >> > >
> > >> > > Let's start by taking a look at exactly where the log
messages are
> > >> > pointing
> > >> > > us. Looking at one of your log files:
> > >> > >
> > >> > >
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210413144310
> > >> > >
> > >> > > Looking at the first call of point_stat, I see the
following
> output
> > >> > > indicating that all of the 21,468 observations were
rejected due
> to
> > >> the
> > >> > > valid_time, which is why there are no matched pairs for
this case:
> > >> > >
> > >> > > > DEBUG 3: Number of matched pairs   = 0
> > >> > > > DEBUG 3: Observations processed    = 21468
> > >> > > > DEBUG 3: Rejected: station id      = 0
> > >> > > > DEBUG 3: Rejected: obs type        = 0
> > >> > > > DEBUG 3: Rejected: valid time      = 21468
> > >> > > > DEBUG 3: Rejected: bad obs value   = 0
> > >> > > > DEBUG 3: Rejected: off the grid    = 0
> > >> > > > DEBUG 3: Rejected: topography      = 0
> > >> > > > DEBUG 3: Rejected: level mismatch  = 0
> > >> > > > DEBUG 3: Rejected: quality marker  = 0
> > >> > > > DEBUG 3: Rejected: message type    = 0
> > >> > > > DEBUG 3: Rejected: masking region  = 0
> > >> > > > DEBUG 3: Rejected: bad fcst value  = 0
> > >> > > > DEBUG 3: Rejected: bad climo mean  = 0
> > >> > > > DEBUG 3: Rejected: bad climo stdev = 0
> > >> > > > DEBUG 3: Rejected: duplicates      = 0
> > >> > >
> > >> > >
> > >> > > This is similar to the situation in the Rejected values
that you
> had
> > >> > pasted
> > >> > > in, except some of those observations were rejected because
they
> > were
> > >> off
> > >> > > of the grid, which is why there are no matched pairs for
this
> case:
> > >> > >
> > >> > > > DEBUG 3: Observations processed = 59531
> > >> > > > DEBUG 3: Rejected: valid time      = 57003
> > >> > > > DEBUG 3: Rejected: off the grid    = 2528
> > >> > >
> > >> > > I wasn't sure which log file that came from, so I just used
your
> > most
> > >> > > recent log file to get the example above.
> > >> > >
> > >> > > You'll want to make sure that when you run Point-Stat, the
> forecast
> > >> > > valid time matches up correctly with the valid time of
point
> > >> > observations.
> > >> > > I may not be the best person to help with that, but I did
find
> > >> something
> > >> > > that may be a part of the problem.
> > >> > >
> > >> > > Here is the call to point_stat for the case with the 21,468
> > >> observations
> > >> > > rejected due to valid time:
> > >> > > point_stat -v 7 \
> > >> > >
> > >> > >
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/\
> > >> > > aqm.20190809/aqm.t12z.metcro2d.12.nc \
> > >> > >
> > >> > >
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r112_v1/pm/
> > >> > > prepbufr.pm.20190810.nc
> > >> > >
> > >> > >
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
> > >> > > \
> > >> > > -outdir
> > >> > >
> > >> > >
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/stat/pm/r112_v1
> > >> > >
> > >> > > I can see that
> > >> > > the
> > >> > >
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
> > >> > > configuration file is being used.
> > >> > >
> > >> > > In that configuration file I see:
> > >> > >
> > >> > > > obs_window = {
> > >> > > >    beg = 0;
> > >> > > >    end = 0;
> > >> > > > }
> > >> > >
> > >> > >
> > >> > > whereas in many of the other Point-Stat configuration files
in
> that
> > >> > > directory I see:
> > >> > >
> > >> > > > obs_window = {
> > >> > > >    beg = ${OBS_WINDOW_BEGIN};
> > >> > > >    end = ${OBS_WINDOW_END};
> > >> > > > }
> > >> > >
> > >> > >
> > >> > > In
> > >> > > your
> > >> > >
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
> > >> > > file, I see the following set:
> > >> > >
> > >> > > > OBS_WINDOW_BEGIN = -2700
> > >> > > > OBS_WINDOW_END = 2700
> > >> > >
> > >> > >
> > >> > > and I was wondering if you are intending for METplus to
pass on
> > these
> > >> > > values in the run to Point-Stat?  If so, you would need to
change
> > the
> > >> > > values of zero to ${OBS_WINDOW_BEGIN} and
${OBS_WINDOW_END},
> > >> > respectively.
> > >> > > Perhaps that is the cause of the zero matched pairs due to
valid
> > time?
> > >> > >
> > >> > > I hope this helps!
> > >> > >
> > >> > > Julie
> > >> > >
> > >> > >
> > >> > > On Tue, Apr 13, 2021 at 9:27 AM Edward Strobach - NOAA
Affiliate
> via
> > >> RT <
> > >> > > met_help at ucar.edu> wrote:
> > >> > >
> > >> > > >
> > >> > > > Tue Apr 13 09:27:33 2021: Request 99563 was acted upon.
> > >> > > > Transaction: Ticket created by edward.strobach at noaa.gov
> > >> > > >        Queue: met_help
> > >> > > >      Subject: zero matched pairs when processing the AQ
inline
> > data
> > >> > > >        Owner: Nobody
> > >> > > >   Requestors: edward.strobach at noaa.gov
> > >> > > >       Status: new
> > >> > > >  Ticket <URL:
> > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99563
> > >> > >
> > >> > > >
> > >> > > >
> > >> > > > Good morning,
> > >> > > >
> > >> > > > I'm attempting to process the data from here: "
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
*/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/aqm.20190810*
> > >> > > > "
> > >> > > >
> > >> > > > The files follow the "aqm.t12z.metcro2d.HH.nc" format and
are
> > >> written
> > >> > > in a
> > >> > > > format that METplus can handle.  The processing steps
include
> > pb2nc,
> > >> > > > pointstat, and statanalysis.  The issue occurs at
pointstat.
> The
> > >> pb2nc
> > >> > > > step runs just fine.
> > >> > > >
> > >> > > > Looking in the netcdf files that I created above confirms
that
> the
> > >> > > fields I
> > >> > > > need are there:
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > *>>> Dataset('aqm.t12z.metcro2d.02.nc
> > >> > > > <http://aqm.t12z.metcro2d.02.nc>')<class
> > >> > 'netCDF4._netCDF4.Dataset'>root
> > >> > > > group (NETCDF4 data model, file format HDF5):
FileOrigins: NA
> > >> > > > MET_version: V9.1    hemisphere: N    Projection: Lambert
> > Conformal
> > >> > > nx:
> > >> > > > 191    ny: 97 grid_points    dimensions(sizes): lat(97),
> lon(191)
> > >> > > > variables(dimensions): float32 LAT(lat,lon), float32
> LON(lat,lon),
> > >> > > float32
> > >> > > > WDIR10(lat,lon), float32 WSPD10(lat,lon), float32
TMP(lat,lon),
> > >> float32
> > >> > > > DPT(lat,lon), float32 OZCON(lat,lon), float32
PMTF(lat,lon)
> > >> groups:
> > >> > *
> > >> > > >
> > >> > > > You can also see that the meta data is not overly
descriptive as
> > in
> > >> the
> > >> > > > past.  I'm not sure if that is problematic or not.  I do
know
> that
> > >> > > whatever
> > >> > > > is not specified in the file is filled by -9999 or -999
values.
> > >> > > >
> > >> > > > Looking at the log file I find the following for every
region,
> for
> > >> both
> > >> > > > PM2.5 and ozone:
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > *DEBUG 2: Processing OZCONA8 versus COPO/A8, for
observation
> type
> > >> > AIRNOW,
> > >> > > > over region WEST, for interpolation method BILIN(4),
using 0
> > matched
> > >> > > > pairs.DEBUG 3: Number of matched pairs   = 0DEBUG 3:
> Observations
> > >> > > processed
> > >> > > >    = 59531DEBUG 3: Rejected: station id      = 0DEBUG 3:
> Rejected:
> > >> obs
> > >> > > type
> > >> > > >        = 0DEBUG 3: Rejected: valid time      = 57003DEBUG
3:
> > >> Rejected:
> > >> > > bad
> > >> > > > obs value   = 0DEBUG 3: Rejected: off the grid    =
2528DEBUG 3:
> > >> > > Rejected:
> > >> > > > topography      = 0DEBUG 3: Rejected: level mismatch  =
0DEBUG
> 3:
> > >> > > Rejected:
> > >> > > > quality marker  = 0DEBUG 3: Rejected: message type    =
0DEBUG
> 3:
> > >> > > Rejected:
> > >> > > > masking region  = 0DEBUG 3: Rejected: bad fcst value  =
0DEBUG
> 3:
> > >> > > Rejected:
> > >> > > > bad climo mean  = 0DEBUG 3: Rejected: bad climo stdev =
0DEBUG
> 3:
> > >> > > Rejected:
> > >> > > > duplicates      = 0*
> > >> > > >
> > >> > > > There's no errors to report, rather there are just no
matched
> > pairs.
> > >> > > Since
> > >> > > > there are only 24 hours being processed for this data,
then only
> > 24
> > >> > hours
> > >> > > > worth of data are being added to the prebufr*nc files.
You can
> > also
> > >> > see
> > >> > > in
> > >> > > > the log file that the appropriate files are being
selected
> > >> according to
> > >> > > > date:
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > *DEBUG 1: Forecast File:
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/aqm.20190810/
> > >> > > > aqm.t12z.metcro2d.10.nc
> > >> > > > <http://aqm.t12z.metcro2d.10.nc>DEBUG 1: Observation
File:
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r112_v1/aq/
> > >> > > > prepbufr.aqm.20190810.nc
> > >> > > > <http://prepbufr.aqm.20190810.nc>DEBUG 7:
check_nc_data_2d()
> > count:
> > >> > > > valid=18527, zero=0, missing=0, 18527 == 18527*
> > >> > > >
> > >> > > > I'm having a hard time figuring out why there are no
matched
> > >> pairs.  A
> > >> > > > couple of other things to note related to an earlier
statement
> > about
> > >> > meta
> > >> > > > data..
> > >> > > >
> > >> > > > Recall that I said the files weren't overly descriptive.
You
> can
> > >> see
> > >> > > that
> > >> > > > the -9999 is being filed for many of the definitions that
are
> not
> > >> > > > specifically being set in the created netcdf files:
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > *DEBUG 4: Lambert Conformal Grid Data:DEBUG 4:
hemisphere:
> > >> NDEBUG 4:
> > >> > > > scale_lat_1: -9999DEBUG 4:   scale_lat_2: -9999DEBUG 4:
> > >>  lat_pin:
> > >> > > > -9999DEBUG 4:       lon_pin: 9999DEBUG 4:         x_pin:
> > -9999DEBUG
> > >> 4:
> > >> > > >     y_pin: -9999DEBUG 4:    lon_orient: 9999DEBUG 4:
> > d_km:
> > >> > > > -9999DEBUG 4:          r_km: -9999DEBUG 4:            nx:
> 191DEBUG
> > >> 4:
> > >> > > >      ny: 97DEBUG 4:     so2_angle: 0DEBUG 4:*
> > >> > > >
> > >> > > > I'm not sure if that is playing a role here.  Printing
out the
> > ozone
> > >> > > field
> > >> > > > from the netcdf file you can see that other information
like
> > >> long_name,
> > >> > > > level, etc. are given.
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > *>>> Dataset('aqm.t12z.metcro2d.02.nc
> > >> > > >
<http://aqm.t12z.metcro2d.02.nc>').variables['OZCON']<class
> > >> > > > 'netCDF4._netCDF4.Variable'>float32 OZCON(lat, lon)
> long_name:
> > >> OZCON
> > >> > > > standard_name: Surface Ozone    units: ppb    level: A8
> > >> init_time:
> > >> > > > 20190810_120000    init_time_ut: 1565438400
valid_time:
> > >> > > 20190810_140000
> > >> > > >   valid_time_ut: 1565445600unlimited dimensions: current
shape =
> > >> (97,
> > >> > > > 191)filling on, default _FillValue of
9.969209968386869e+36
> used*
> > >> > > >
> > >> > > > Is there something I'm not thinking about here?  I can
provide
> > more
> > >> > > details
> > >> > > > if needed.  Hopefully this description is a good start.
I do
> > recall
> > >> > > having
> > >> > > > a similar issue in the past but couldn't locate the
exchange
> going
> > >> over
> > >> > > > this problem.
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > Edward Strobach
> > >> > > > EMC/NCEP/NWS/ IMSG
> > >> > > > Coding Logic:  if, then, else
> > >> > > >
> > >> > > >
> > >> > >
> > >> > > --
> > >> > > Julie Prestopnik (she/her)
> > >> > > Software Engineer
> > >> > > National Center for Atmospheric Research
> > >> > > Research Applications Laboratory
> > >> > > Email: jpresto at ucar.edu
> > >> > >
> > >> > > My working day may not be your working day.  Please do not
feel
> > >> obliged
> > >> > to
> > >> > > reply to this email outside of your normal working hours.
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >>
> > >> --
> > >> Julie Prestopnik (she/her)
> > >> Software Engineer
> > >> National Center for Atmospheric Research
> > >> Research Applications Laboratory
> > >> Email: jpresto at ucar.edu
> > >>
> > >> My working day may not be your working day.  Please do not feel
> obliged
> > to
> > >> reply to this email outside of your normal working hours.
> > >>
> > >>
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

------------------------------------------------
Subject: zero matched pairs when processing the AQ inline data
From: Edward Strobach - NOAA Affiliate
Time: Wed Apr 14 19:43:09 2021

That was it!  I had to populate the metadata further.  I don't know if
all
the definitions had to be filled but now I'm getting the stats.

Edward Strobach
EMC/NCEP/NWS/ IMSG
Coding Logic:  if, then, else

On Wed, Apr 14, 2021 at 9:18 PM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:

> Thanks, but I've used these settings for quite a long time and it
didn't
> result in these issues before.  I just looked at an example where it
did
> work and the configure file at the pb2nc grid set to G212 while
pointstat
> was FULL.  I'm going to have to look at other possible explanations.
I did
> note originally
> that the metadata information is not complete in these files
compared to
> cases that work.  For instance I have this for the cases that are
failing:
>
>
>
>
>
>
>
>
>
>
>
>
> *>>> Dataset('aqm.t12z.metcro2d.02.nc
> <http://aqm.t12z.metcro2d.02.nc>')<class
'netCDF4._netCDF4.Dataset'>root
> group (NETCDF4 data model, file format HDF5):    FileOrigins: NA
> MET_version: V9.1    hemisphere: N    Projection: Lambert Conformal
nx:
> 191    ny: 97 grid_points    dimensions(sizes): lat(97), lon(191)
> variables(dimensions): float32 LAT(lat,lon), float32 LON(lat,lon),
float32
> WDIR10(lat,lon), float32 WSPD10(lat,lon), float32 TMP(lat,lon),
float32
> DPT(lat,lon), float32 OZCON(lat,lon), float32 PMTF(lat,lon)
groups: *
>
> Because not everything was specified in the metadata structure I was
> getting filler values for those definitions. See the printout below:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *DEBUG 4: Lambert Conformal Grid Data:DEBUG 4:    hemisphere: NDEBUG
4:
> scale_lat_1: -9999DEBUG 4:   scale_lat_2: -9999DEBUG 4:
lat_pin:
> -9999DEBUG 4:       lon_pin: 9999DEBUG 4:         x_pin: -9999DEBUG
4:
>     y_pin: -9999DEBUG 4:    lon_orient: 9999DEBUG 4:          d_km:
> -9999DEBUG 4:          r_km: -9999DEBUG 4:            nx: 191DEBUG
4:
>      ny: 97DEBUG 4:     so2_angle: 0DEBUG 4:*
>
> In the case that worked I had all these quantities specified so
there were
> no filler values.  I'm not sure, but I think this could be an issue.
I will
> try to look for these values but I can't guarantee that I'll find
them all.
>
> Edward Strobach
> EMC/NCEP/NWS/ IMSG
> Coding Logic:  if, then, else
>
>
> On Wed, Apr 14, 2021 at 4:28 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
>> Hi Edward,
>>
>> From looking at this my guess is the missing data values in the
grid
>> information could be the cause of no matched pairs. I'm not sure
why that
>> would be if the NetCDF files can be read by MET, but that seems off
to me.
>> I don't have access to WCOSS, but I would check that the grid
information
>> needed by MET is found in the input data.
>>
>> Another potential reason for the pairs being rejected for being off
the
>> grid is the grid definitions to set in the use case. It looks like
PB2NC
>> is
>> set the mask.grid value to G212 while PointStat is set to use FULL
for the
>> mask.grid and G104 for the regrid.grid value:
>>
>> PB2NC_GRID = G212
>> POINT_STAT_GRID = FULL
>> POINT_STAT_REGRID_TO_GRID = G104
>>
>> I would try to resolve these values so that the desired grid is
used
>> consistently.
>>
>> I hope that helps! Let me know if you are still having issues.
>>
>> Thanks,
>> George
>>
>> On Wed, Apr 14, 2021 at 11:40 AM Edward Strobach - NOAA Affiliate
via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99563 >
>> >
>> > Hello,
>> >
>> > Latest tests indicate, at least from what I can see, that the
issue is
>> not
>> > necessarily the valid time.  I'm running 31 jobs corresponding to
31
>> days
>> > in Aug. 2019.  For the 8/22/2019 example you can see that netcdf
files
>> are
>> > created for both ozone and pm for the pb2nc step.  You can see
that
>> here:
>> > *DEBUG 1: Creating NetCDF File:
>> >
>> >
>>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r111_v1/pm/
>> > prepbufr.pm.20190822.nc
>> > <http://prepbufr.pm.20190822.nc>*
>> >
>> > I believe that the following file "
>> >
>> >
>>
*/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/Archive/201908/Obs/hourly.20190823/aqm.t12z.anowpm.pb.tm024*"
>> > is grabbed in this case because of the 24 hour OBS_WINDOW that is
set
>> (i.e.
>> > -86400 and +86400 centered on 8/23/2019).
>> >
>> > After pb2nc is completed the same file is then used for
pointstat:
>> > *04/14 17:23:06.994 metplus.PointStat (command_builder.py:548)
DEBUG:
>> Found
>> > file:
>> >
>> >
>>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r111_v1/pm/
>> > prepbufr.pm.20190822.n*
>> > c
>> >
>> > It starts by identifying this file:
>> >
>> >
>>
*/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r111_v1/aqm.20190821/
>> > aqm.t12z.metcro2d.12.nc
>> > <http://aqm.t12z.metcro2d.12.nc>
>> >
>> >
>>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r111_v1/pm/
>> > prepbufr.pm.20190822.nc
>> > <http://prepbufr.pm.20190822.nc>
>> >
>> >
>>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
>> > -outdir
>> >
>> >
>>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/stat/pm/r111_v1*
>> >
>> > There are some warning messages related to valid_time but I don't
know
>> what
>> > they mean; it seems more related to a switch in interpolation
based on
>> > condition that is met:
>> >
>> > *WARNING: RegridInfo::validate() -> Resetting the regridding
method from
>> > "BILIN" to "NEAREST" since the regridding width is 1.*
>> >
>> > Eventually as we step through valid times we arrive to the next
day's
>> > files:
>> >
>> > *DEBUG 1: Forecast File:
>> >
>> >
>>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r111_v1/aqm.20190822/
>> > aqm.t12z.metcro2d.02.nc
>> > <http://aqm.t12z.metcro2d.02.nc>DEBUG 1: Observation File:
>> >
>> >
>>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r111_v1/pm/
>> > prepbufr.pm.20190822.nc
>> > <http://prepbufr.pm.20190822.nc>*
>> >
>> > Now I'm certain that the valid_time is set correctly.  This is
what I've
>> > always used in the past. The only difference is that I'm using a
>> different
>> > model data source but I attempt to follow a netcdf writing
template
>> that is
>> > compliant with MET standards.  One other difference is that I'm
>> processing
>> > 24 hours of forecast data while in the past I processed 48 to 72
hours
>> of
>> > forecast data.
>> >
>> > I'm attaching an example conf file that is used for the
pointstat/pb2nc
>> > process.  This shows the correct settings that I wanted to use.
>> >
>> > Edward Strobach
>> > EMC/NCEP/NWS/ IMSG
>> > Coding Logic:  if, then, else
>> >
>> >
>> > On Tue, Apr 13, 2021 at 8:08 PM Edward Strobach - NOAA Affiliate
<
>> > edward.strobach at noaa.gov> wrote:
>> >
>> > > Thanks, that's what I'll try to do.
>> > >
>> > >
>> > > Edward Strobach
>> > > EMC/NCEP/NWS/ IMSG
>> > > Coding Logic:  if, then, else
>> > >
>> > >
>> > > On Tue, Apr 13, 2021 at 6:16 PM Julie Prestopnik via RT <
>> > met_help at ucar.edu>
>> > > wrote:
>> > >
>> > >> Hi Edward.
>> > >>
>> > >> It's interesting that you note the settings for OBS_WINDOW* in
the
>> > >> >
>> > >> >
>> > >>
>> >
>>
/*gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
>> > >> > file* are set to 2700.  I'm not sure what's setting that.
>> > >> >
>> > >> Perhaps the metplus_final.conf file I referred to did not go
with the
>> > run
>> > >> for the log I referenced:
>> > >>
>> > >>
>> >
>>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210413144310
>> > >>
>> > >> Because I can see in that log file, just above the call to
>> point_stat,
>> > the
>> > >> values for some of the fields are listed and indeed 86400 is
used for
>> > >> OBS_WINDOW_BEGIN:
>> > >>
>> > >> > 04/13 14:43:12.448 metplus.PointStat
(command_builder.py:256)
>> DEBUG:
>> > >> > export CLIMO_MEAN_FILE=""; export CLIMO_STDEV_FILE="";
export
>> > >> FCST_FIELD="{
>> > >> > name\
>> > >> > =\"PMTF\"; level=\"L1\"; },{ name=\"OZCON\"; level=\"A8\";
}";
>> export
>> > >> >
>> > >>
>> >
>>
MET_TMP_DIR="/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CM\
>> > >> > AQ/tmp"; export MODEL="R112_V1"; export OBS_FIELD="{
>> name=\"COPOPM\";
>> > >> > level=\"A1\"; convert(x) = x * 10^9; },{ name=\"COPO\";
>> level=\"A8\";
>> > >> > convert(x\
>> > >> > ) = x * 10^9; }"; export OBS_WINDOW_BEGIN="-86400"; export
>> > >> > OBS_WINDOW_END="86400"; export OUTPUT_PREFIX=""; export
>> > >> > POINT_STAT_GRID="[\"FULL\"]"; expo\
>> > >> > rt POINT_STAT_MESSAGE_TYPE="[\"ONLYSF\"]"; export
>> > POINT_STAT_POLY="[]";
>> > >> > export POINT_STAT_STATION_ID="[]"; export
>> REGRID_TO_GRID="\"G104\"";
>> > >> export
>> > >> > V\
>> > >> > ERIF_MASK="";
>> > >>
>> > >>
>> > >> In a call like the following:
>> > >>
>> > >> > *${MET_PLUS}/ush/master_metplus.py -c
>> > >> > ${MET_PLUS}/parm/use_cases/grid_to_obs/grid_to_obs.conf -c
>> > >> >
${MET_PLUS}/parm/use_cases/grid_to_obs/examples/conus_surface.conf
>> -c
>> > >> > ${MET_PLUS_CONF}/pb2nc_aq.conf -c
>> ${MET_PLUS_CONF}/point_stat_aq.conf
>> > -c
>> > >> > ${MET_PLUS_TMP}/${field}.conf -c
${MET_PLUS_CONF}/shared.conf -c
>> > >> > ${MET_PLUS_TMP}/shared.conf_aq -c
${MET_PLUS_CONF}/system_aq.conf*
>> > >>
>> > >>
>> > >> Anything in ${MET_PLUS_CONF}/system_aq.conf would override
anything
>> in
>> > any
>> > >> previously listed config file.  The order in which you list
your
>> METplus
>> > >> wrapper config files matters, as it sounds like you are aware
of.
>> Each
>> > >> subsequent config file on the command line will override any
values
>> > >> defined
>> > >> in an earlier config file.
>> > >>
>> > >> The issue may be the same with the VALID_BEG and VALID_END in
that
>> the
>> > >> metplus_final.conf file I referred to did not go with the run
for the
>> > log
>> > >> I
>> > >> referenced.
>> > >>
>> > >> You could start a clean run and check the time stamp on
>> > >> the metplus_final.conf file, save that off along with the
>> corresponding
>> > >> metplus log file, check the values and let us know if you see
>> anything
>> > >> unusual.
>> > >>
>> > >> Let us know what your questions are as you proceed working on
getting
>> > the
>> > >> valid times to line up.
>> > >>
>> > >> Julie
>> > >>
>> > >>
>> > >> On Tue, Apr 13, 2021 at 1:17 PM Edward Strobach - NOAA
Affiliate via
>> RT
>> > <
>> > >> met_help at ucar.edu> wrote:
>> > >>
>> > >> >
>> > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99563 >
>> > >> >
>> > >> > Thank you for your help.
>> > >> >
>> > >> > I see from my metplus_final_pb2nc_pointstat.conf located in
my
>> > >> >
>> > >> >
>> > >>
>> >
>>
"/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/conf/r112_v1/aq"
>> > >> > directory are references to the time window for both the
pb2nc and
>> > >> > pointstat step:
>> > >> >
>> > >> >
>> > >> > *PB2NC_WINDOW_BEGIN = -86400PB2NC_WINDOW_END = 86400*
>> > >> >
>> > >> > and
>> > >> >
>> > >> >
>> > >> > *OBS_WINDOW_BEGIN = -86400OBS_WINDOW_END = 86400*
>> > >> >
>> > >> > It's interesting that you note the settings for OBS_WINDOW*
in the
>> > >> >
>> > >> >
>> > >>
>> >
>>
/*gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
>> > >> > file* are set to 2700.  I'm not sure what's setting that.  I
>> thought
>> > it
>> > >> was
>> > >> > being set based on what's in the shared.conf found in my
>> > >> run_metplus_aq.sh
>> > >> > script located here:
>> > >> >
>>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/scripts.
>> > >> >
>> > >> > I thought that the OBS_WINDOW* definitions set in the
shared.conf
>> > would
>> > >> > override other conf files as well.  Running that step looks
like
>> this
>> > >> in my
>> > >> > run_metplus_aq.sh script:
>> > >> >
>> > >> > *${MET_PLUS}/ush/master_metplus.py -c
>> > >> > ${MET_PLUS}/parm/use_cases/grid_to_obs/grid_to_obs.conf -c
>> > >> >
${MET_PLUS}/parm/use_cases/grid_to_obs/examples/conus_surface.conf
>> -c
>> > >> > ${MET_PLUS_CONF}/pb2nc_aq.conf -c
>> ${MET_PLUS_CONF}/point_stat_aq.conf
>> > -c
>> > >> > ${MET_PLUS_TMP}/${field}.conf -c
${MET_PLUS_CONF}/shared.conf -c
>> > >> > ${MET_PLUS_TMP}/shared.conf_aq -c
${MET_PLUS_CONF}/system_aq.conf*
>> > >> >
>> > >> > Something else that is troubling from that conf file you
mentioned
>> is
>> > >> the
>> > >> > valid times settings:
>> > >> > VALID_BEG = 20210116
>> > >> > VALID_END = 20210116
>> > >> >
>> > >> > I set that specifically to 20190810.  I didn't think to look
where
>> you
>> > >> did
>> > >> > because the inputs that I assigned to my main run script
look like
>> > this:
>> > >> >
>> > >> >
>> > >> >
>> > >>
>> >
>>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/scripts/run_metplus_aq.sh
>> > >> > *r112_v1 archive verification LAM 20190810*
>> > >> >
>> > >> > the input date in this case is 20190810 but somehow is not
being
>> > >> used..  I
>> > >> > don't know how this could be happening.  Thanks for pointing
this
>> out.
>> > >> >
>> > >> >
>> > >> >
>> > >> > Edward Strobach
>> > >> > EMC/NCEP/NWS/ IMSG
>> > >> > Coding Logic:  if, then, else
>> > >> >
>> > >> >
>> > >> > On Tue, Apr 13, 2021 at 2:20 PM Julie Prestopnik via RT <
>> > >> met_help at ucar.edu
>> > >> > >
>> > >> > wrote:
>> > >> >
>> > >> > > Hi Edward.
>> > >> > >
>> > >> > > I see that you are having trouble getting matched pairs
from
>> > >> Point-Stat.
>> > >> > >
>> > >> > > Let's start by taking a look at exactly where the log
messages
>> are
>> > >> > pointing
>> > >> > > us. Looking at one of your log files:
>> > >> > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210413144310
>> > >> > >
>> > >> > > Looking at the first call of point_stat, I see the
following
>> output
>> > >> > > indicating that all of the 21,468 observations were
rejected due
>> to
>> > >> the
>> > >> > > valid_time, which is why there are no matched pairs for
this
>> case:
>> > >> > >
>> > >> > > > DEBUG 3: Number of matched pairs   = 0
>> > >> > > > DEBUG 3: Observations processed    = 21468
>> > >> > > > DEBUG 3: Rejected: station id      = 0
>> > >> > > > DEBUG 3: Rejected: obs type        = 0
>> > >> > > > DEBUG 3: Rejected: valid time      = 21468
>> > >> > > > DEBUG 3: Rejected: bad obs value   = 0
>> > >> > > > DEBUG 3: Rejected: off the grid    = 0
>> > >> > > > DEBUG 3: Rejected: topography      = 0
>> > >> > > > DEBUG 3: Rejected: level mismatch  = 0
>> > >> > > > DEBUG 3: Rejected: quality marker  = 0
>> > >> > > > DEBUG 3: Rejected: message type    = 0
>> > >> > > > DEBUG 3: Rejected: masking region  = 0
>> > >> > > > DEBUG 3: Rejected: bad fcst value  = 0
>> > >> > > > DEBUG 3: Rejected: bad climo mean  = 0
>> > >> > > > DEBUG 3: Rejected: bad climo stdev = 0
>> > >> > > > DEBUG 3: Rejected: duplicates      = 0
>> > >> > >
>> > >> > >
>> > >> > > This is similar to the situation in the Rejected values
that you
>> had
>> > >> > pasted
>> > >> > > in, except some of those observations were rejected
because they
>> > were
>> > >> off
>> > >> > > of the grid, which is why there are no matched pairs for
this
>> case:
>> > >> > >
>> > >> > > > DEBUG 3: Observations processed = 59531
>> > >> > > > DEBUG 3: Rejected: valid time      = 57003
>> > >> > > > DEBUG 3: Rejected: off the grid    = 2528
>> > >> > >
>> > >> > > I wasn't sure which log file that came from, so I just
used your
>> > most
>> > >> > > recent log file to get the example above.
>> > >> > >
>> > >> > > You'll want to make sure that when you run Point-Stat, the
>> forecast
>> > >> > > valid time matches up correctly with the valid time of
point
>> > >> > observations.
>> > >> > > I may not be the best person to help with that, but I did
find
>> > >> something
>> > >> > > that may be a part of the problem.
>> > >> > >
>> > >> > > Here is the call to point_stat for the case with the
21,468
>> > >> observations
>> > >> > > rejected due to valid time:
>> > >> > > point_stat -v 7 \
>> > >> > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/\
>> > >> > > aqm.20190809/aqm.t12z.metcro2d.12.nc \
>> > >> > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r112_v1/pm/
>> > >> > > prepbufr.pm.20190810.nc
>> > >> > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
>> > >> > > \
>> > >> > > -outdir
>> > >> > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/stat/pm/r112_v1
>> > >> > >
>> > >> > > I can see that
>> > >> > > the
>> > >> > >
>> > >> >
>> > >>
>> >
>>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
>> > >> > > configuration file is being used.
>> > >> > >
>> > >> > > In that configuration file I see:
>> > >> > >
>> > >> > > > obs_window = {
>> > >> > > >    beg = 0;
>> > >> > > >    end = 0;
>> > >> > > > }
>> > >> > >
>> > >> > >
>> > >> > > whereas in many of the other Point-Stat configuration
files in
>> that
>> > >> > > directory I see:
>> > >> > >
>> > >> > > > obs_window = {
>> > >> > > >    beg = ${OBS_WINDOW_BEGIN};
>> > >> > > >    end = ${OBS_WINDOW_END};
>> > >> > > > }
>> > >> > >
>> > >> > >
>> > >> > > In
>> > >> > > your
>> > >> > >
>> > >> >
>> > >>
>> >
>>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
>> > >> > > file, I see the following set:
>> > >> > >
>> > >> > > > OBS_WINDOW_BEGIN = -2700
>> > >> > > > OBS_WINDOW_END = 2700
>> > >> > >
>> > >> > >
>> > >> > > and I was wondering if you are intending for METplus to
pass on
>> > these
>> > >> > > values in the run to Point-Stat?  If so, you would need to
change
>> > the
>> > >> > > values of zero to ${OBS_WINDOW_BEGIN} and
${OBS_WINDOW_END},
>> > >> > respectively.
>> > >> > > Perhaps that is the cause of the zero matched pairs due to
valid
>> > time?
>> > >> > >
>> > >> > > I hope this helps!
>> > >> > >
>> > >> > > Julie
>> > >> > >
>> > >> > >
>> > >> > > On Tue, Apr 13, 2021 at 9:27 AM Edward Strobach - NOAA
Affiliate
>> via
>> > >> RT <
>> > >> > > met_help at ucar.edu> wrote:
>> > >> > >
>> > >> > > >
>> > >> > > > Tue Apr 13 09:27:33 2021: Request 99563 was acted upon.
>> > >> > > > Transaction: Ticket created by edward.strobach at noaa.gov
>> > >> > > >        Queue: met_help
>> > >> > > >      Subject: zero matched pairs when processing the AQ
inline
>> > data
>> > >> > > >        Owner: Nobody
>> > >> > > >   Requestors: edward.strobach at noaa.gov
>> > >> > > >       Status: new
>> > >> > > >  Ticket <URL:
>> > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99563
>> > >> > >
>> > >> > > >
>> > >> > > >
>> > >> > > > Good morning,
>> > >> > > >
>> > >> > > > I'm attempting to process the data from here: "
>> > >> > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
*/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/aqm.20190810*
>> > >> > > > "
>> > >> > > >
>> > >> > > > The files follow the "aqm.t12z.metcro2d.HH.nc" format
and are
>> > >> written
>> > >> > > in a
>> > >> > > > format that METplus can handle.  The processing steps
include
>> > pb2nc,
>> > >> > > > pointstat, and statanalysis.  The issue occurs at
pointstat.
>> The
>> > >> pb2nc
>> > >> > > > step runs just fine.
>> > >> > > >
>> > >> > > > Looking in the netcdf files that I created above
confirms that
>> the
>> > >> > > fields I
>> > >> > > > need are there:
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > > *>>> Dataset('aqm.t12z.metcro2d.02.nc
>> > >> > > > <http://aqm.t12z.metcro2d.02.nc>')<class
>> > >> > 'netCDF4._netCDF4.Dataset'>root
>> > >> > > > group (NETCDF4 data model, file format HDF5):
FileOrigins:
>> NA
>> > >> > > > MET_version: V9.1    hemisphere: N    Projection:
Lambert
>> > Conformal
>> > >> > > nx:
>> > >> > > > 191    ny: 97 grid_points    dimensions(sizes): lat(97),
>> lon(191)
>> > >> > > > variables(dimensions): float32 LAT(lat,lon), float32
>> LON(lat,lon),
>> > >> > > float32
>> > >> > > > WDIR10(lat,lon), float32 WSPD10(lat,lon), float32
TMP(lat,lon),
>> > >> float32
>> > >> > > > DPT(lat,lon), float32 OZCON(lat,lon), float32
PMTF(lat,lon)
>> > >> groups:
>> > >> > *
>> > >> > > >
>> > >> > > > You can also see that the meta data is not overly
descriptive
>> as
>> > in
>> > >> the
>> > >> > > > past.  I'm not sure if that is problematic or not.  I do
know
>> that
>> > >> > > whatever
>> > >> > > > is not specified in the file is filled by -9999 or -999
values.
>> > >> > > >
>> > >> > > > Looking at the log file I find the following for every
region,
>> for
>> > >> both
>> > >> > > > PM2.5 and ozone:
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > > *DEBUG 2: Processing OZCONA8 versus COPO/A8, for
observation
>> type
>> > >> > AIRNOW,
>> > >> > > > over region WEST, for interpolation method BILIN(4),
using 0
>> > matched
>> > >> > > > pairs.DEBUG 3: Number of matched pairs   = 0DEBUG 3:
>> Observations
>> > >> > > processed
>> > >> > > >    = 59531DEBUG 3: Rejected: station id      = 0DEBUG 3:
>> Rejected:
>> > >> obs
>> > >> > > type
>> > >> > > >        = 0DEBUG 3: Rejected: valid time      =
57003DEBUG 3:
>> > >> Rejected:
>> > >> > > bad
>> > >> > > > obs value   = 0DEBUG 3: Rejected: off the grid    =
2528DEBUG
>> 3:
>> > >> > > Rejected:
>> > >> > > > topography      = 0DEBUG 3: Rejected: level mismatch  =
0DEBUG
>> 3:
>> > >> > > Rejected:
>> > >> > > > quality marker  = 0DEBUG 3: Rejected: message type    =
0DEBUG
>> 3:
>> > >> > > Rejected:
>> > >> > > > masking region  = 0DEBUG 3: Rejected: bad fcst value  =
0DEBUG
>> 3:
>> > >> > > Rejected:
>> > >> > > > bad climo mean  = 0DEBUG 3: Rejected: bad climo stdev =
0DEBUG
>> 3:
>> > >> > > Rejected:
>> > >> > > > duplicates      = 0*
>> > >> > > >
>> > >> > > > There's no errors to report, rather there are just no
matched
>> > pairs.
>> > >> > > Since
>> > >> > > > there are only 24 hours being processed for this data,
then
>> only
>> > 24
>> > >> > hours
>> > >> > > > worth of data are being added to the prebufr*nc files.
You can
>> > also
>> > >> > see
>> > >> > > in
>> > >> > > > the log file that the appropriate files are being
selected
>> > >> according to
>> > >> > > > date:
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > > *DEBUG 1: Forecast File:
>> > >> > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/aqm.20190810/
>> > >> > > > aqm.t12z.metcro2d.10.nc
>> > >> > > > <http://aqm.t12z.metcro2d.10.nc>DEBUG 1: Observation
File:
>> > >> > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r112_v1/aq/
>> > >> > > > prepbufr.aqm.20190810.nc
>> > >> > > > <http://prepbufr.aqm.20190810.nc>DEBUG 7:
check_nc_data_2d()
>> > count:
>> > >> > > > valid=18527, zero=0, missing=0, 18527 == 18527*
>> > >> > > >
>> > >> > > > I'm having a hard time figuring out why there are no
matched
>> > >> pairs.  A
>> > >> > > > couple of other things to note related to an earlier
statement
>> > about
>> > >> > meta
>> > >> > > > data..
>> > >> > > >
>> > >> > > > Recall that I said the files weren't overly descriptive.
You
>> can
>> > >> see
>> > >> > > that
>> > >> > > > the -9999 is being filed for many of the definitions
that are
>> not
>> > >> > > > specifically being set in the created netcdf files:
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > > *DEBUG 4: Lambert Conformal Grid Data:DEBUG 4:
hemisphere:
>> > >> NDEBUG 4:
>> > >> > > > scale_lat_1: -9999DEBUG 4:   scale_lat_2: -9999DEBUG 4:
>> > >>  lat_pin:
>> > >> > > > -9999DEBUG 4:       lon_pin: 9999DEBUG 4:         x_pin:
>> > -9999DEBUG
>> > >> 4:
>> > >> > > >     y_pin: -9999DEBUG 4:    lon_orient: 9999DEBUG 4:
>> > d_km:
>> > >> > > > -9999DEBUG 4:          r_km: -9999DEBUG 4:
nx:
>> 191DEBUG
>> > >> 4:
>> > >> > > >      ny: 97DEBUG 4:     so2_angle: 0DEBUG 4:*
>> > >> > > >
>> > >> > > > I'm not sure if that is playing a role here.  Printing
out the
>> > ozone
>> > >> > > field
>> > >> > > > from the netcdf file you can see that other information
like
>> > >> long_name,
>> > >> > > > level, etc. are given.
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > > *>>> Dataset('aqm.t12z.metcro2d.02.nc
>> > >> > > >
<http://aqm.t12z.metcro2d.02.nc>').variables['OZCON']<class
>> > >> > > > 'netCDF4._netCDF4.Variable'>float32 OZCON(lat, lon)
>> long_name:
>> > >> OZCON
>> > >> > > > standard_name: Surface Ozone    units: ppb    level: A8
>> > >> init_time:
>> > >> > > > 20190810_120000    init_time_ut: 1565438400
valid_time:
>> > >> > > 20190810_140000
>> > >> > > >   valid_time_ut: 1565445600unlimited dimensions: current
shape
>> =
>> > >> (97,
>> > >> > > > 191)filling on, default _FillValue of
9.969209968386869e+36
>> used*
>> > >> > > >
>> > >> > > > Is there something I'm not thinking about here?  I can
provide
>> > more
>> > >> > > details
>> > >> > > > if needed.  Hopefully this description is a good start.
I do
>> > recall
>> > >> > > having
>> > >> > > > a similar issue in the past but couldn't locate the
exchange
>> going
>> > >> over
>> > >> > > > this problem.
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > >
>> > >> > > > Edward Strobach
>> > >> > > > EMC/NCEP/NWS/ IMSG
>> > >> > > > Coding Logic:  if, then, else
>> > >> > > >
>> > >> > > >
>> > >> > >
>> > >> > > --
>> > >> > > Julie Prestopnik (she/her)
>> > >> > > Software Engineer
>> > >> > > National Center for Atmospheric Research
>> > >> > > Research Applications Laboratory
>> > >> > > Email: jpresto at ucar.edu
>> > >> > >
>> > >> > > My working day may not be your working day.  Please do not
feel
>> > >> obliged
>> > >> > to
>> > >> > > reply to this email outside of your normal working hours.
>> > >> > >
>> > >> > >
>> > >> >
>> > >> >
>> > >>
>> > >> --
>> > >> Julie Prestopnik (she/her)
>> > >> Software Engineer
>> > >> National Center for Atmospheric Research
>> > >> Research Applications Laboratory
>> > >> Email: jpresto at ucar.edu
>> > >>
>> > >> My working day may not be your working day.  Please do not
feel
>> obliged
>> > to
>> > >> reply to this email outside of your normal working hours.
>> > >>
>> > >>
>> >
>> >
>>
>> --
>> George McCabe - Software Engineer III
>> National Center for Atmospheric Research
>> Research Applications Laboratory
>> 303-497-2768
>> ---
>> My working day may not be your working day. Please do not feel
obliged to
>> reply to this email outside of your normal working hours.
>>
>>

------------------------------------------------
Subject: zero matched pairs when processing the AQ inline data
From: Julie Prestopnik
Time: Thu Apr 15 12:27:04 2021

That's great news!  Thank you for letting us know.  I'll go ahead and
close
this ticket, but please feel free to start a new one with any other
issues.

Julie

On Wed, Apr 14, 2021 at 7:43 PM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99563 >
>
> That was it!  I had to populate the metadata further.  I don't know
if all
> the definitions had to be filled but now I'm getting the stats.
>
>
> Edward Strobach
> EMC/NCEP/NWS/ IMSG
> Coding Logic:  if, then, else
>
>
> On Wed, Apr 14, 2021 at 9:18 PM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
> > Thanks, but I've used these settings for quite a long time and it
didn't
> > result in these issues before.  I just looked at an example where
it did
> > work and the configure file at the pb2nc grid set to G212 while
pointstat
> > was FULL.  I'm going to have to look at other possible
explanations.  I
> did
> > note originally
> > that the metadata information is not complete in these files
compared to
> > cases that work.  For instance I have this for the cases that are
> failing:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *>>> Dataset('aqm.t12z.metcro2d.02.nc
> > <http://aqm.t12z.metcro2d.02.nc>')<class
'netCDF4._netCDF4.Dataset'>root
> > group (NETCDF4 data model, file format HDF5):    FileOrigins: NA
> > MET_version: V9.1    hemisphere: N    Projection: Lambert
Conformal
> nx:
> > 191    ny: 97 grid_points    dimensions(sizes): lat(97), lon(191)
> > variables(dimensions): float32 LAT(lat,lon), float32 LON(lat,lon),
> float32
> > WDIR10(lat,lon), float32 WSPD10(lat,lon), float32 TMP(lat,lon),
float32
> > DPT(lat,lon), float32 OZCON(lat,lon), float32 PMTF(lat,lon)
groups: *
> >
> > Because not everything was specified in the metadata structure I
was
> > getting filler values for those definitions. See the printout
below:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *DEBUG 4: Lambert Conformal Grid Data:DEBUG 4:    hemisphere:
NDEBUG 4:
> > scale_lat_1: -9999DEBUG 4:   scale_lat_2: -9999DEBUG 4:
lat_pin:
> > -9999DEBUG 4:       lon_pin: 9999DEBUG 4:         x_pin:
-9999DEBUG 4:
> >     y_pin: -9999DEBUG 4:    lon_orient: 9999DEBUG 4:
d_km:
> > -9999DEBUG 4:          r_km: -9999DEBUG 4:            nx: 191DEBUG
4:
> >      ny: 97DEBUG 4:     so2_angle: 0DEBUG 4:*
> >
> > In the case that worked I had all these quantities specified so
there
> were
> > no filler values.  I'm not sure, but I think this could be an
issue. I
> will
> > try to look for these values but I can't guarantee that I'll find
them
> all.
> >
> > Edward Strobach
> > EMC/NCEP/NWS/ IMSG
> > Coding Logic:  if, then, else
> >
> >
> > On Wed, Apr 14, 2021 at 4:28 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> >> Hi Edward,
> >>
> >> From looking at this my guess is the missing data values in the
grid
> >> information could be the cause of no matched pairs. I'm not sure
why
> that
> >> would be if the NetCDF files can be read by MET, but that seems
off to
> me.
> >> I don't have access to WCOSS, but I would check that the grid
> information
> >> needed by MET is found in the input data.
> >>
> >> Another potential reason for the pairs being rejected for being
off the
> >> grid is the grid definitions to set in the use case. It looks
like PB2NC
> >> is
> >> set the mask.grid value to G212 while PointStat is set to use
FULL for
> the
> >> mask.grid and G104 for the regrid.grid value:
> >>
> >> PB2NC_GRID = G212
> >> POINT_STAT_GRID = FULL
> >> POINT_STAT_REGRID_TO_GRID = G104
> >>
> >> I would try to resolve these values so that the desired grid is
used
> >> consistently.
> >>
> >> I hope that helps! Let me know if you are still having issues.
> >>
> >> Thanks,
> >> George
> >>
> >> On Wed, Apr 14, 2021 at 11:40 AM Edward Strobach - NOAA Affiliate
via
> RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99563 >
> >> >
> >> > Hello,
> >> >
> >> > Latest tests indicate, at least from what I can see, that the
issue is
> >> not
> >> > necessarily the valid time.  I'm running 31 jobs corresponding
to 31
> >> days
> >> > in Aug. 2019.  For the 8/22/2019 example you can see that
netcdf files
> >> are
> >> > created for both ozone and pm for the pb2nc step.  You can see
that
> >> here:
> >> > *DEBUG 1: Creating NetCDF File:
> >> >
> >> >
> >>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r111_v1/pm/
> >> > prepbufr.pm.20190822.nc
> >> > <http://prepbufr.pm.20190822.nc>*
> >> >
> >> > I believe that the following file "
> >> >
> >> >
> >>
>
*/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/Archive/201908/Obs/hourly.20190823/aqm.t12z.anowpm.pb.tm024*"
> >> > is grabbed in this case because of the 24 hour OBS_WINDOW that
is set
> >> (i.e.
> >> > -86400 and +86400 centered on 8/23/2019).
> >> >
> >> > After pb2nc is completed the same file is then used for
pointstat:
> >> > *04/14 17:23:06.994 metplus.PointStat (command_builder.py:548)
DEBUG:
> >> Found
> >> > file:
> >> >
> >> >
> >>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r111_v1/pm/
> >> > prepbufr.pm.20190822.n*
> >> > c
> >> >
> >> > It starts by identifying this file:
> >> >
> >> >
> >>
>
*/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r111_v1/aqm.20190821/
> >> > aqm.t12z.metcro2d.12.nc
> >> > <http://aqm.t12z.metcro2d.12.nc>
> >> >
> >> >
> >>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r111_v1/pm/
> >> > prepbufr.pm.20190822.nc
> >> > <http://prepbufr.pm.20190822.nc>
> >> >
> >> >
> >>
>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
> >> > -outdir
> >> >
> >> >
> >>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/stat/pm/r111_v1*
> >> >
> >> > There are some warning messages related to valid_time but I
don't know
> >> what
> >> > they mean; it seems more related to a switch in interpolation
based on
> >> > condition that is met:
> >> >
> >> > *WARNING: RegridInfo::validate() -> Resetting the regridding
method
> from
> >> > "BILIN" to "NEAREST" since the regridding width is 1.*
> >> >
> >> > Eventually as we step through valid times we arrive to the next
day's
> >> > files:
> >> >
> >> > *DEBUG 1: Forecast File:
> >> >
> >> >
> >>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r111_v1/aqm.20190822/
> >> > aqm.t12z.metcro2d.02.nc
> >> > <http://aqm.t12z.metcro2d.02.nc>DEBUG 1: Observation File:
> >> >
> >> >
> >>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r111_v1/pm/
> >> > prepbufr.pm.20190822.nc
> >> > <http://prepbufr.pm.20190822.nc>*
> >> >
> >> > Now I'm certain that the valid_time is set correctly.  This is
what
> I've
> >> > always used in the past. The only difference is that I'm using
a
> >> different
> >> > model data source but I attempt to follow a netcdf writing
template
> >> that is
> >> > compliant with MET standards.  One other difference is that I'm
> >> processing
> >> > 24 hours of forecast data while in the past I processed 48 to
72 hours
> >> of
> >> > forecast data.
> >> >
> >> > I'm attaching an example conf file that is used for the
> pointstat/pb2nc
> >> > process.  This shows the correct settings that I wanted to use.
> >> >
> >> > Edward Strobach
> >> > EMC/NCEP/NWS/ IMSG
> >> > Coding Logic:  if, then, else
> >> >
> >> >
> >> > On Tue, Apr 13, 2021 at 8:08 PM Edward Strobach - NOAA
Affiliate <
> >> > edward.strobach at noaa.gov> wrote:
> >> >
> >> > > Thanks, that's what I'll try to do.
> >> > >
> >> > >
> >> > > Edward Strobach
> >> > > EMC/NCEP/NWS/ IMSG
> >> > > Coding Logic:  if, then, else
> >> > >
> >> > >
> >> > > On Tue, Apr 13, 2021 at 6:16 PM Julie Prestopnik via RT <
> >> > met_help at ucar.edu>
> >> > > wrote:
> >> > >
> >> > >> Hi Edward.
> >> > >>
> >> > >> It's interesting that you note the settings for OBS_WINDOW*
in the
> >> > >> >
> >> > >> >
> >> > >>
> >> >
> >>
>
/*gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
> >> > >> > file* are set to 2700.  I'm not sure what's setting that.
> >> > >> >
> >> > >> Perhaps the metplus_final.conf file I referred to did not go
with
> the
> >> > run
> >> > >> for the log I referenced:
> >> > >>
> >> > >>
> >> >
> >>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210413144310
> >> > >>
> >> > >> Because I can see in that log file, just above the call to
> >> point_stat,
> >> > the
> >> > >> values for some of the fields are listed and indeed 86400 is
used
> for
> >> > >> OBS_WINDOW_BEGIN:
> >> > >>
> >> > >> > 04/13 14:43:12.448 metplus.PointStat
(command_builder.py:256)
> >> DEBUG:
> >> > >> > export CLIMO_MEAN_FILE=""; export CLIMO_STDEV_FILE="";
export
> >> > >> FCST_FIELD="{
> >> > >> > name\
> >> > >> > =\"PMTF\"; level=\"L1\"; },{ name=\"OZCON\"; level=\"A8\";
}";
> >> export
> >> > >> >
> >> > >>
> >> >
> >>
>
MET_TMP_DIR="/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CM\
> >> > >> > AQ/tmp"; export MODEL="R112_V1"; export OBS_FIELD="{
> >> name=\"COPOPM\";
> >> > >> > level=\"A1\"; convert(x) = x * 10^9; },{ name=\"COPO\";
> >> level=\"A8\";
> >> > >> > convert(x\
> >> > >> > ) = x * 10^9; }"; export OBS_WINDOW_BEGIN="-86400"; export
> >> > >> > OBS_WINDOW_END="86400"; export OUTPUT_PREFIX=""; export
> >> > >> > POINT_STAT_GRID="[\"FULL\"]"; expo\
> >> > >> > rt POINT_STAT_MESSAGE_TYPE="[\"ONLYSF\"]"; export
> >> > POINT_STAT_POLY="[]";
> >> > >> > export POINT_STAT_STATION_ID="[]"; export
> >> REGRID_TO_GRID="\"G104\"";
> >> > >> export
> >> > >> > V\
> >> > >> > ERIF_MASK="";
> >> > >>
> >> > >>
> >> > >> In a call like the following:
> >> > >>
> >> > >> > *${MET_PLUS}/ush/master_metplus.py -c
> >> > >> > ${MET_PLUS}/parm/use_cases/grid_to_obs/grid_to_obs.conf -c
> >> > >> >
> ${MET_PLUS}/parm/use_cases/grid_to_obs/examples/conus_surface.conf
> >> -c
> >> > >> > ${MET_PLUS_CONF}/pb2nc_aq.conf -c
> >> ${MET_PLUS_CONF}/point_stat_aq.conf
> >> > -c
> >> > >> > ${MET_PLUS_TMP}/${field}.conf -c
${MET_PLUS_CONF}/shared.conf -c
> >> > >> > ${MET_PLUS_TMP}/shared.conf_aq -c
> ${MET_PLUS_CONF}/system_aq.conf*
> >> > >>
> >> > >>
> >> > >> Anything in ${MET_PLUS_CONF}/system_aq.conf would override
anything
> >> in
> >> > any
> >> > >> previously listed config file.  The order in which you list
your
> >> METplus
> >> > >> wrapper config files matters, as it sounds like you are
aware of.
> >> Each
> >> > >> subsequent config file on the command line will override any
values
> >> > >> defined
> >> > >> in an earlier config file.
> >> > >>
> >> > >> The issue may be the same with the VALID_BEG and VALID_END
in that
> >> the
> >> > >> metplus_final.conf file I referred to did not go with the
run for
> the
> >> > log
> >> > >> I
> >> > >> referenced.
> >> > >>
> >> > >> You could start a clean run and check the time stamp on
> >> > >> the metplus_final.conf file, save that off along with the
> >> corresponding
> >> > >> metplus log file, check the values and let us know if you
see
> >> anything
> >> > >> unusual.
> >> > >>
> >> > >> Let us know what your questions are as you proceed working
on
> getting
> >> > the
> >> > >> valid times to line up.
> >> > >>
> >> > >> Julie
> >> > >>
> >> > >>
> >> > >> On Tue, Apr 13, 2021 at 1:17 PM Edward Strobach - NOAA
Affiliate
> via
> >> RT
> >> > <
> >> > >> met_help at ucar.edu> wrote:
> >> > >>
> >> > >> >
> >> > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99563 >
> >> > >> >
> >> > >> > Thank you for your help.
> >> > >> >
> >> > >> > I see from my metplus_final_pb2nc_pointstat.conf located
in my
> >> > >> >
> >> > >> >
> >> > >>
> >> >
> >>
>
"/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/conf/r112_v1/aq"
> >> > >> > directory are references to the time window for both the
pb2nc
> and
> >> > >> > pointstat step:
> >> > >> >
> >> > >> >
> >> > >> > *PB2NC_WINDOW_BEGIN = -86400PB2NC_WINDOW_END = 86400*
> >> > >> >
> >> > >> > and
> >> > >> >
> >> > >> >
> >> > >> > *OBS_WINDOW_BEGIN = -86400OBS_WINDOW_END = 86400*
> >> > >> >
> >> > >> > It's interesting that you note the settings for
OBS_WINDOW* in
> the
> >> > >> >
> >> > >> >
> >> > >>
> >> >
> >>
>
/*gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
> >> > >> > file* are set to 2700.  I'm not sure what's setting that.
I
> >> thought
> >> > it
> >> > >> was
> >> > >> > being set based on what's in the shared.conf found in my
> >> > >> run_metplus_aq.sh
> >> > >> > script located here:
> >> > >> >
> >>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/scripts.
> >> > >> >
> >> > >> > I thought that the OBS_WINDOW* definitions set in the
shared.conf
> >> > would
> >> > >> > override other conf files as well.  Running that step
looks like
> >> this
> >> > >> in my
> >> > >> > run_metplus_aq.sh script:
> >> > >> >
> >> > >> > *${MET_PLUS}/ush/master_metplus.py -c
> >> > >> > ${MET_PLUS}/parm/use_cases/grid_to_obs/grid_to_obs.conf -c
> >> > >> >
> ${MET_PLUS}/parm/use_cases/grid_to_obs/examples/conus_surface.conf
> >> -c
> >> > >> > ${MET_PLUS_CONF}/pb2nc_aq.conf -c
> >> ${MET_PLUS_CONF}/point_stat_aq.conf
> >> > -c
> >> > >> > ${MET_PLUS_TMP}/${field}.conf -c
${MET_PLUS_CONF}/shared.conf -c
> >> > >> > ${MET_PLUS_TMP}/shared.conf_aq -c
> ${MET_PLUS_CONF}/system_aq.conf*
> >> > >> >
> >> > >> > Something else that is troubling from that conf file you
> mentioned
> >> is
> >> > >> the
> >> > >> > valid times settings:
> >> > >> > VALID_BEG = 20210116
> >> > >> > VALID_END = 20210116
> >> > >> >
> >> > >> > I set that specifically to 20190810.  I didn't think to
look
> where
> >> you
> >> > >> did
> >> > >> > because the inputs that I assigned to my main run script
look
> like
> >> > this:
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >>
> >> >
> >>
>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/scripts/run_metplus_aq.sh
> >> > >> > *r112_v1 archive verification LAM 20190810*
> >> > >> >
> >> > >> > the input date in this case is 20190810 but somehow is not
being
> >> > >> used..  I
> >> > >> > don't know how this could be happening.  Thanks for
pointing this
> >> out.
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > Edward Strobach
> >> > >> > EMC/NCEP/NWS/ IMSG
> >> > >> > Coding Logic:  if, then, else
> >> > >> >
> >> > >> >
> >> > >> > On Tue, Apr 13, 2021 at 2:20 PM Julie Prestopnik via RT <
> >> > >> met_help at ucar.edu
> >> > >> > >
> >> > >> > wrote:
> >> > >> >
> >> > >> > > Hi Edward.
> >> > >> > >
> >> > >> > > I see that you are having trouble getting matched pairs
from
> >> > >> Point-Stat.
> >> > >> > >
> >> > >> > > Let's start by taking a look at exactly where the log
messages
> >> are
> >> > >> > pointing
> >> > >> > > us. Looking at one of your log files:
> >> > >> > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210413144310
> >> > >> > >
> >> > >> > > Looking at the first call of point_stat, I see the
following
> >> output
> >> > >> > > indicating that all of the 21,468 observations were
rejected
> due
> >> to
> >> > >> the
> >> > >> > > valid_time, which is why there are no matched pairs for
this
> >> case:
> >> > >> > >
> >> > >> > > > DEBUG 3: Number of matched pairs   = 0
> >> > >> > > > DEBUG 3: Observations processed    = 21468
> >> > >> > > > DEBUG 3: Rejected: station id      = 0
> >> > >> > > > DEBUG 3: Rejected: obs type        = 0
> >> > >> > > > DEBUG 3: Rejected: valid time      = 21468
> >> > >> > > > DEBUG 3: Rejected: bad obs value   = 0
> >> > >> > > > DEBUG 3: Rejected: off the grid    = 0
> >> > >> > > > DEBUG 3: Rejected: topography      = 0
> >> > >> > > > DEBUG 3: Rejected: level mismatch  = 0
> >> > >> > > > DEBUG 3: Rejected: quality marker  = 0
> >> > >> > > > DEBUG 3: Rejected: message type    = 0
> >> > >> > > > DEBUG 3: Rejected: masking region  = 0
> >> > >> > > > DEBUG 3: Rejected: bad fcst value  = 0
> >> > >> > > > DEBUG 3: Rejected: bad climo mean  = 0
> >> > >> > > > DEBUG 3: Rejected: bad climo stdev = 0
> >> > >> > > > DEBUG 3: Rejected: duplicates      = 0
> >> > >> > >
> >> > >> > >
> >> > >> > > This is similar to the situation in the Rejected values
that
> you
> >> had
> >> > >> > pasted
> >> > >> > > in, except some of those observations were rejected
because
> they
> >> > were
> >> > >> off
> >> > >> > > of the grid, which is why there are no matched pairs for
this
> >> case:
> >> > >> > >
> >> > >> > > > DEBUG 3: Observations processed = 59531
> >> > >> > > > DEBUG 3: Rejected: valid time      = 57003
> >> > >> > > > DEBUG 3: Rejected: off the grid    = 2528
> >> > >> > >
> >> > >> > > I wasn't sure which log file that came from, so I just
used
> your
> >> > most
> >> > >> > > recent log file to get the example above.
> >> > >> > >
> >> > >> > > You'll want to make sure that when you run Point-Stat,
the
> >> forecast
> >> > >> > > valid time matches up correctly with the valid time of
point
> >> > >> > observations.
> >> > >> > > I may not be the best person to help with that, but I
did find
> >> > >> something
> >> > >> > > that may be a part of the problem.
> >> > >> > >
> >> > >> > > Here is the call to point_stat for the case with the
21,468
> >> > >> observations
> >> > >> > > rejected due to valid time:
> >> > >> > > point_stat -v 7 \
> >> > >> > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/\
> >> > >> > > aqm.20190809/aqm.t12z.metcro2d.12.nc \
> >> > >> > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r112_v1/pm/
> >> > >> > > prepbufr.pm.20190810.nc
> >> > >> > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
> >> > >> > > \
> >> > >> > > -outdir
> >> > >> > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/stat/pm/r112_v1
> >> > >> > >
> >> > >> > > I can see that
> >> > >> > > the
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
>
/gpfs/dell2/emc/verification/save/Edward.Strobach/MET/METplus/parm/met_config/PointStatConfig_ANOWPM
> >> > >> > > configuration file is being used.
> >> > >> > >
> >> > >> > > In that configuration file I see:
> >> > >> > >
> >> > >> > > > obs_window = {
> >> > >> > > >    beg = 0;
> >> > >> > > >    end = 0;
> >> > >> > > > }
> >> > >> > >
> >> > >> > >
> >> > >> > > whereas in many of the other Point-Stat configuration
files in
> >> that
> >> > >> > > directory I see:
> >> > >> > >
> >> > >> > > > obs_window = {
> >> > >> > > >    beg = ${OBS_WINDOW_BEGIN};
> >> > >> > > >    end = ${OBS_WINDOW_END};
> >> > >> > > > }
> >> > >> > >
> >> > >> > >
> >> > >> > > In
> >> > >> > > your
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/metplus_final.conf
> >> > >> > > file, I see the following set:
> >> > >> > >
> >> > >> > > > OBS_WINDOW_BEGIN = -2700
> >> > >> > > > OBS_WINDOW_END = 2700
> >> > >> > >
> >> > >> > >
> >> > >> > > and I was wondering if you are intending for METplus to
pass on
> >> > these
> >> > >> > > values in the run to Point-Stat?  If so, you would need
to
> change
> >> > the
> >> > >> > > values of zero to ${OBS_WINDOW_BEGIN} and
${OBS_WINDOW_END},
> >> > >> > respectively.
> >> > >> > > Perhaps that is the cause of the zero matched pairs due
to
> valid
> >> > time?
> >> > >> > >
> >> > >> > > I hope this helps!
> >> > >> > >
> >> > >> > > Julie
> >> > >> > >
> >> > >> > >
> >> > >> > > On Tue, Apr 13, 2021 at 9:27 AM Edward Strobach - NOAA
> Affiliate
> >> via
> >> > >> RT <
> >> > >> > > met_help at ucar.edu> wrote:
> >> > >> > >
> >> > >> > > >
> >> > >> > > > Tue Apr 13 09:27:33 2021: Request 99563 was acted
upon.
> >> > >> > > > Transaction: Ticket created by
edward.strobach at noaa.gov
> >> > >> > > >        Queue: met_help
> >> > >> > > >      Subject: zero matched pairs when processing the
AQ
> inline
> >> > data
> >> > >> > > >        Owner: Nobody
> >> > >> > > >   Requestors: edward.strobach at noaa.gov
> >> > >> > > >       Status: new
> >> > >> > > >  Ticket <URL:
> >> > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=99563
> >> > >> > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > Good morning,
> >> > >> > > >
> >> > >> > > > I'm attempting to process the data from here: "
> >> > >> > > >
> >> > >> > > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
>
*/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/aqm.20190810*
> >> > >> > > > "
> >> > >> > > >
> >> > >> > > > The files follow the "aqm.t12z.metcro2d.HH.nc" format
and
> are
> >> > >> written
> >> > >> > > in a
> >> > >> > > > format that METplus can handle.  The processing steps
include
> >> > pb2nc,
> >> > >> > > > pointstat, and statanalysis.  The issue occurs at
pointstat.
> >> The
> >> > >> pb2nc
> >> > >> > > > step runs just fine.
> >> > >> > > >
> >> > >> > > > Looking in the netcdf files that I created above
confirms
> that
> >> the
> >> > >> > > fields I
> >> > >> > > > need are there:
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > *>>> Dataset('aqm.t12z.metcro2d.02.nc
> >> > >> > > > <http://aqm.t12z.metcro2d.02.nc>')<class
> >> > >> > 'netCDF4._netCDF4.Dataset'>root
> >> > >> > > > group (NETCDF4 data model, file format HDF5):
FileOrigins:
> >> NA
> >> > >> > > > MET_version: V9.1    hemisphere: N    Projection:
Lambert
> >> > Conformal
> >> > >> > > nx:
> >> > >> > > > 191    ny: 97 grid_points    dimensions(sizes):
lat(97),
> >> lon(191)
> >> > >> > > > variables(dimensions): float32 LAT(lat,lon), float32
> >> LON(lat,lon),
> >> > >> > > float32
> >> > >> > > > WDIR10(lat,lon), float32 WSPD10(lat,lon), float32
> TMP(lat,lon),
> >> > >> float32
> >> > >> > > > DPT(lat,lon), float32 OZCON(lat,lon), float32
PMTF(lat,lon)
> >> > >> groups:
> >> > >> > *
> >> > >> > > >
> >> > >> > > > You can also see that the meta data is not overly
descriptive
> >> as
> >> > in
> >> > >> the
> >> > >> > > > past.  I'm not sure if that is problematic or not.  I
do know
> >> that
> >> > >> > > whatever
> >> > >> > > > is not specified in the file is filled by -9999 or
-999
> values.
> >> > >> > > >
> >> > >> > > > Looking at the log file I find the following for every
> region,
> >> for
> >> > >> both
> >> > >> > > > PM2.5 and ozone:
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > *DEBUG 2: Processing OZCONA8 versus COPO/A8, for
observation
> >> type
> >> > >> > AIRNOW,
> >> > >> > > > over region WEST, for interpolation method BILIN(4),
using 0
> >> > matched
> >> > >> > > > pairs.DEBUG 3: Number of matched pairs   = 0DEBUG 3:
> >> Observations
> >> > >> > > processed
> >> > >> > > >    = 59531DEBUG 3: Rejected: station id      = 0DEBUG
3:
> >> Rejected:
> >> > >> obs
> >> > >> > > type
> >> > >> > > >        = 0DEBUG 3: Rejected: valid time      =
57003DEBUG 3:
> >> > >> Rejected:
> >> > >> > > bad
> >> > >> > > > obs value   = 0DEBUG 3: Rejected: off the grid    =
2528DEBUG
> >> 3:
> >> > >> > > Rejected:
> >> > >> > > > topography      = 0DEBUG 3: Rejected: level mismatch
=
> 0DEBUG
> >> 3:
> >> > >> > > Rejected:
> >> > >> > > > quality marker  = 0DEBUG 3: Rejected: message type
=
> 0DEBUG
> >> 3:
> >> > >> > > Rejected:
> >> > >> > > > masking region  = 0DEBUG 3: Rejected: bad fcst value
=
> 0DEBUG
> >> 3:
> >> > >> > > Rejected:
> >> > >> > > > bad climo mean  = 0DEBUG 3: Rejected: bad climo stdev
=
> 0DEBUG
> >> 3:
> >> > >> > > Rejected:
> >> > >> > > > duplicates      = 0*
> >> > >> > > >
> >> > >> > > > There's no errors to report, rather there are just no
matched
> >> > pairs.
> >> > >> > > Since
> >> > >> > > > there are only 24 hours being processed for this data,
then
> >> only
> >> > 24
> >> > >> > hours
> >> > >> > > > worth of data are being added to the prebufr*nc files.
You
> can
> >> > also
> >> > >> > see
> >> > >> > > in
> >> > >> > > > the log file that the appropriate files are being
selected
> >> > >> according to
> >> > >> > > > date:
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > *DEBUG 1: Forecast File:
> >> > >> > > >
> >> > >> > > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/CMAQ_Converted_Files/r112_v1/aqm.20190810/
> >> > >> > > > aqm.t12z.metcro2d.10.nc
> >> > >> > > > <http://aqm.t12z.metcro2d.10.nc>DEBUG 1: Observation
File:
> >> > >> > > >
> >> > >> > > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> >
> >>
>
/gpfs/dell2/emc/verification/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r112_v1/aq/
> >> > >> > > > prepbufr.aqm.20190810.nc
> >> > >> > > > <http://prepbufr.aqm.20190810.nc>DEBUG 7:
check_nc_data_2d()
> >> > count:
> >> > >> > > > valid=18527, zero=0, missing=0, 18527 == 18527*
> >> > >> > > >
> >> > >> > > > I'm having a hard time figuring out why there are no
matched
> >> > >> pairs.  A
> >> > >> > > > couple of other things to note related to an earlier
> statement
> >> > about
> >> > >> > meta
> >> > >> > > > data..
> >> > >> > > >
> >> > >> > > > Recall that I said the files weren't overly
descriptive.  You
> >> can
> >> > >> see
> >> > >> > > that
> >> > >> > > > the -9999 is being filed for many of the definitions
that are
> >> not
> >> > >> > > > specifically being set in the created netcdf files:
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > *DEBUG 4: Lambert Conformal Grid Data:DEBUG 4:
hemisphere:
> >> > >> NDEBUG 4:
> >> > >> > > > scale_lat_1: -9999DEBUG 4:   scale_lat_2: -9999DEBUG
4:
> >> > >>  lat_pin:
> >> > >> > > > -9999DEBUG 4:       lon_pin: 9999DEBUG 4:
x_pin:
> >> > -9999DEBUG
> >> > >> 4:
> >> > >> > > >     y_pin: -9999DEBUG 4:    lon_orient: 9999DEBUG 4:
> >> > d_km:
> >> > >> > > > -9999DEBUG 4:          r_km: -9999DEBUG 4:
nx:
> >> 191DEBUG
> >> > >> 4:
> >> > >> > > >      ny: 97DEBUG 4:     so2_angle: 0DEBUG 4:*
> >> > >> > > >
> >> > >> > > > I'm not sure if that is playing a role here.  Printing
out
> the
> >> > ozone
> >> > >> > > field
> >> > >> > > > from the netcdf file you can see that other
information like
> >> > >> long_name,
> >> > >> > > > level, etc. are given.
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > *>>> Dataset('aqm.t12z.metcro2d.02.nc
> >> > >> > > >
<http://aqm.t12z.metcro2d.02.nc>').variables['OZCON']<class
> >> > >> > > > 'netCDF4._netCDF4.Variable'>float32 OZCON(lat, lon)
> >> long_name:
> >> > >> OZCON
> >> > >> > > > standard_name: Surface Ozone    units: ppb    level:
A8
> >> > >> init_time:
> >> > >> > > > 20190810_120000    init_time_ut: 1565438400
valid_time:
> >> > >> > > 20190810_140000
> >> > >> > > >   valid_time_ut: 1565445600unlimited dimensions:
current
> shape
> >> =
> >> > >> (97,
> >> > >> > > > 191)filling on, default _FillValue of
9.969209968386869e+36
> >> used*
> >> > >> > > >
> >> > >> > > > Is there something I'm not thinking about here?  I can
> provide
> >> > more
> >> > >> > > details
> >> > >> > > > if needed.  Hopefully this description is a good
start.  I do
> >> > recall
> >> > >> > > having
> >> > >> > > > a similar issue in the past but couldn't locate the
exchange
> >> going
> >> > >> over
> >> > >> > > > this problem.
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > Edward Strobach
> >> > >> > > > EMC/NCEP/NWS/ IMSG
> >> > >> > > > Coding Logic:  if, then, else
> >> > >> > > >
> >> > >> > > >
> >> > >> > >
> >> > >> > > --
> >> > >> > > Julie Prestopnik (she/her)
> >> > >> > > Software Engineer
> >> > >> > > National Center for Atmospheric Research
> >> > >> > > Research Applications Laboratory
> >> > >> > > Email: jpresto at ucar.edu
> >> > >> > >
> >> > >> > > My working day may not be your working day.  Please do
not feel
> >> > >> obliged
> >> > >> > to
> >> > >> > > reply to this email outside of your normal working
hours.
> >> > >> > >
> >> > >> > >
> >> > >> >
> >> > >> >
> >> > >>
> >> > >> --
> >> > >> Julie Prestopnik (she/her)
> >> > >> Software Engineer
> >> > >> National Center for Atmospheric Research
> >> > >> Research Applications Laboratory
> >> > >> Email: jpresto at ucar.edu
> >> > >>
> >> > >> My working day may not be your working day.  Please do not
feel
> >> obliged
> >> > to
> >> > >> reply to this email outside of your normal working hours.
> >> > >>
> >> > >>
> >> >
> >> >
> >>
> >> --
> >> George McCabe - Software Engineer III
> >> National Center for Atmospheric Research
> >> Research Applications Laboratory
> >> 303-497-2768
> >> ---
> >> My working day may not be your working day. Please do not feel
obliged
> to
> >> reply to this email outside of your normal working hours.
> >>
> >>
>
>

--
Julie Prestopnik (she/her)
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Email: jpresto at ucar.edu

My working day may not be your working day.  Please do not feel
obliged to
reply to this email outside of your normal working hours.

------------------------------------------------