[Met_help] [rt.rap.ucar.edu #98524] History for field name and level strings are not separated when running pointstat for a new model
George McCabe via RT
met_help at ucar.edu
Wed Feb 10 13:51:18 MST 2021
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Good afternoon,
We are trying to generate statistics of the AQ model inline system which
has a different file structure than the offline system. As before, I
generate netcdf files with the fields I want to process which are taken
from files of the inline system. So far, nothing has really changed in my
processing steps, just ingesting data from a different source and grouping
verifiable fields. The netcdf file for a run that works looks like this:
For prod (offiline system that works):
*>>> Dataset('aqm.t12z.metcro2d.05.nc
<http://aqm.t12z.metcro2d.05.nc>')<class 'netCDF4._netCDF4.Dataset'>root
group (NETCDF4 data model, file format HDF5): FileOrigins: NA
MET_version: V9.1 Projection: Lambert Conformal hemisphere: N
accum: 00 scale_lat_1: 33.000000 scale_lat_2: 45.000000 lat_pin:
21.821000 lon_pin: -120.628000 x_pin: 0.000000 y_pin: 0.000000
lon_orient: -97.000000 d_km: 12.000000 r_km: 6371.200000 nx: 442
ny: 265 grid_points dimensions(sizes): lon(442), lat(265)
variables(dimensions): float32 LAT(lat,lon), float32 LON(lat,lon), float32
PMTF(lat,lon), float32 OZCON(lat,lon) groups: *
*For r101_v1 (inline system that doesn't work):*
*Dataset('aqm.t12z.metcro2d.05.nc <http://aqm.t12z.metcro2d.05.nc>')<class
'netCDF4._netCDF4.Dataset'>root group (NETCDF4 data model, file format
HDF5): FileOrigins: NA MET_version: V9.1 hemisphere: N
Projection: Lambert Conformal nx: 191 ny: 97 grid_points
dimensions(sizes): lat(97), lon(191) variables(dimensions): float32
LAT(lat,lon), float32 LON(lat,lon), float32 PBL(lat,lon), float32
WDIR10(lat,lon), float32 WSPD10(lat,lon), float32 TMP(lat,lon), float32
DPT(lat,lon), float32 OZCON(lat,lon), float32 PMTF(lat,lon) groups: *
the meta data is shortened up for the inline system because I don't have
all the information I need, but I don't think that's important here. When
printing out the attributes of one of the inline fields, I see the expected
information:
*Dataset('aqm.t12z.metcro2d.05.nc
<http://aqm.t12z.metcro2d.05.nc>').variables['PMTF']<class
'netCDF4._netCDF4.Variable'>float32 PMTF(lat, lon) long_name: PMTF
standard_name: Surface PM2.5 units: ug/m**3 level: L1 init_time:
20190810_120000 init_time_ut: 1565438400 valid_time: 20190810_170000
valid_time_ut: 1565456400*
What's happening is that the long_name and level are being joined in the
pointstat processing, which is strange to me. I don't know why that would
be. In the log file you can see that they are separate up to a point.
Here's print out confirming one such example:
*02/03 17:26:11.109 metplus.PointStat (command_builder.py:168) DEBUG:
COPYABLE ENVIRONMENT FOR NEXT COMMAND:02/03 17:26:11.109 metplus.PointStat
(command_builder.py:256) DEBUG: export CLIMO_MEAN_FILE=""; export
CLIMO_STDEV_FILE=""; export FCST_FIELD="{ name=\"PMTF\"; level=\"L1\"; },{
name=\"OZCON\"; level=\"A8\"; }"; export
MET_TMP_DIR="/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/tmp";
export MODEL="R101_V1"; export OBS_FIELD="{ name=\"COPOPM\"; level=\"A1\";
convert(x) = x * 10^9; },{*
Below is where the error occurs:
*DEBUG 2: Reading data for PMTFL1.WARNING:WARNING:
process_fcst_climo_files() -> no fields matching PMTFL1 found in file:
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190808/aqm.t12z.metcro2d.36.nc
<http://aqm.t12z.metcro2d.36.nc>WARNING:DEBUG 2:DEBUG 2:
--------------------------------------------------------------------------------DEBUG
2:DEBUG 2: Reading data for OZCONA8.WARNING:WARNING:
process_fcst_climo_files() -> no fields matching OZCONA8 found in file:
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190808/aqm.t12z.metcro2d.36.nc
<http://aqm.t12z.metcro2d.36.nc>WARNING:ERROR :ERROR :
process_fcst_climo_files() -> no requested forecast data found!
Exiting...ERROR :*
You can see that the longname and level are joined, and as result, fails
because that field doesn't exist. You can also clearly see that my
configure file separates these fields:
*FCST_VAR1_NAME = OZCONFCST_VAR1_LEVELS = A1OBS_VAR1_NAME=
COPOOBS_VAR1_LEVELS= A1OBS_VAR1_OPTIONS = convert(x) = x *
10^9;FCST_VAR2_NAME = OZCONFCST_VAR2_LEVELS = A8OBS_VAR2_NAME=
COPOOBS_VAR2_LEVELS= A8*
It's not clear to me why this issue is occurring if everything else appears
correct.
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: field name and level strings are not separated when running pointstat for a new model
From: George McCabe
Time: Wed Feb 03 12:58:32 2021
Hi Edward,
John pulled down the file from your log error message for me and it
looks
like the fields you are requesting are not found in that file:
/usr/local/netcdf/bin/ncdump aqm.t12z.metcro2d.36.nc | grep float
float LAT(lat, lon) ;
float LON(lat, lon) ;
float PBL(lat, lon) ;
float WDIR10(lat, lon) ;
float WSPD10(lat, lon) ;
float TMP(lat, lon) ;
float DPT(lat, lon) ;
That would explain why it can't find that field. Also, it looks like
the
fields in that file are 2D NetCDF, so I think you would want to
specify the
level with the NetCDF notation, so instead of A8, you would use
"(*,*)"
On Wed, Feb 3, 2021 at 10:43 AM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> Wed Feb 03 10:42:39 2021: Request 98524 was acted upon.
> Transaction: Ticket created by edward.strobach at noaa.gov
> Queue: met_help
> Subject: field name and level strings are not separated when
running
> pointstat for a new model
> Owner: Nobody
> Requestors: edward.strobach at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524 >
>
>
> Good afternoon,
>
> We are trying to generate statistics of the AQ model inline system
which
> has a different file structure than the offline system. As before,
I
> generate netcdf files with the fields I want to process which are
taken
> from files of the inline system. So far, nothing has really changed
in my
> processing steps, just ingesting data from a different source and
grouping
> verifiable fields. The netcdf file for a run that works looks like
this:
>
> For prod (offiline system that works):
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *>>> Dataset('aqm.t12z.metcro2d.05.nc
> <http://aqm.t12z.metcro2d.05.nc>')<class
'netCDF4._netCDF4.Dataset'>root
> group (NETCDF4 data model, file format HDF5): FileOrigins: NA
> MET_version: V9.1 Projection: Lambert Conformal hemisphere: N
> accum: 00 scale_lat_1: 33.000000 scale_lat_2: 45.000000
lat_pin:
> 21.821000 lon_pin: -120.628000 x_pin: 0.000000 y_pin:
0.000000
> lon_orient: -97.000000 d_km: 12.000000 r_km: 6371.200000
nx: 442
> ny: 265 grid_points dimensions(sizes): lon(442), lat(265)
> variables(dimensions): float32 LAT(lat,lon), float32 LON(lat,lon),
float32
> PMTF(lat,lon), float32 OZCON(lat,lon) groups: *
>
> *For r101_v1 (inline system that doesn't work):*
>
>
>
>
>
>
>
>
>
>
>
>
> *Dataset('aqm.t12z.metcro2d.05.nc
<http://aqm.t12z.metcro2d.05.nc>')<class
> 'netCDF4._netCDF4.Dataset'>root group (NETCDF4 data model, file
format
> HDF5): FileOrigins: NA MET_version: V9.1 hemisphere: N
> Projection: Lambert Conformal nx: 191 ny: 97 grid_points
> dimensions(sizes): lat(97), lon(191) variables(dimensions):
float32
> LAT(lat,lon), float32 LON(lat,lon), float32 PBL(lat,lon), float32
> WDIR10(lat,lon), float32 WSPD10(lat,lon), float32 TMP(lat,lon),
float32
> DPT(lat,lon), float32 OZCON(lat,lon), float32 PMTF(lat,lon)
groups: *
>
> the meta data is shortened up for the inline system because I don't
have
> all the information I need, but I don't think that's important here.
When
> printing out the attributes of one of the inline fields, I see the
expected
> information:
>
>
>
>
>
>
>
>
>
>
> *Dataset('aqm.t12z.metcro2d.05.nc
> <http://aqm.t12z.metcro2d.05.nc>').variables['PMTF']<class
> 'netCDF4._netCDF4.Variable'>float32 PMTF(lat, lon) long_name:
PMTF
> standard_name: Surface PM2.5 units: ug/m**3 level: L1
init_time:
> 20190810_120000 init_time_ut: 1565438400 valid_time:
20190810_170000
> valid_time_ut: 1565456400*
>
> What's happening is that the long_name and level are being joined in
the
> pointstat processing, which is strange to me. I don't know why that
would
> be. In the log file you can see that they are separate up to a
point.
> Here's print out confirming one such example:
>
>
> *02/03 17:26:11.109 metplus.PointStat (command_builder.py:168)
DEBUG:
> COPYABLE ENVIRONMENT FOR NEXT COMMAND:02/03 17:26:11.109
metplus.PointStat
> (command_builder.py:256) DEBUG: export CLIMO_MEAN_FILE=""; export
> CLIMO_STDEV_FILE=""; export FCST_FIELD="{ name=\"PMTF\";
level=\"L1\"; },{
> name=\"OZCON\"; level=\"A8\"; }"; export
>
>
MET_TMP_DIR="/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/tmp";
> export MODEL="R101_V1"; export OBS_FIELD="{ name=\"COPOPM\";
level=\"A1\";
> convert(x) = x * 10^9; },{*
>
> Below is where the error occurs:
>
>
>
>
>
>
>
>
>
>
>
>
>
> *DEBUG 2: Reading data for PMTFL1.WARNING:WARNING:
> process_fcst_climo_files() -> no fields matching PMTFL1 found in
file:
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190808/
> aqm.t12z.metcro2d.36.nc
> <http://aqm.t12z.metcro2d.36.nc>WARNING:DEBUG 2:DEBUG 2:
>
>
--------------------------------------------------------------------------------DEBUG
> 2:DEBUG 2: Reading data for OZCONA8.WARNING:WARNING:
> process_fcst_climo_files() -> no fields matching OZCONA8 found in
file:
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190808/
> aqm.t12z.metcro2d.36.nc
> <http://aqm.t12z.metcro2d.36.nc>WARNING:ERROR :ERROR :
> process_fcst_climo_files() -> no requested forecast data found!
> Exiting...ERROR :*
>
> You can see that the longname and level are joined, and as result,
fails
> because that field doesn't exist. You can also clearly see that my
> configure file separates these fields:
>
>
>
>
>
>
>
>
>
> *FCST_VAR1_NAME = OZCONFCST_VAR1_LEVELS = A1OBS_VAR1_NAME=
> COPOOBS_VAR1_LEVELS= A1OBS_VAR1_OPTIONS = convert(x) = x *
> 10^9;FCST_VAR2_NAME = OZCONFCST_VAR2_LEVELS = A8OBS_VAR2_NAME=
> COPOOBS_VAR2_LEVELS= A8*
>
> It's not clear to me why this issue is occurring if everything else
appears
> correct.
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: field name and level strings are not separated when running pointstat for a new model
From: Edward Strobach - NOAA Affiliate
Time: Wed Feb 03 13:02:19 2021
Okay, thanks. I'm aware of this but it's only because certain files
don't
contain that field. The forecast runs out to 48 hours, but the
surface
ozone was only saved for the first 24 hours. The metcro files -
forecast
hours 0 through 23 - should be able to be processed. Do all files
have to
be present within the time range encompassing 48 hours to work?
On Wed, Feb 3, 2021 at 2:58 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Edward,
>
> John pulled down the file from your log error message for me and it
looks
> like the fields you are requesting are not found in that file:
>
> /usr/local/netcdf/bin/ncdump aqm.t12z.metcro2d.36.nc | grep float
> float LAT(lat, lon) ;
> float LON(lat, lon) ;
> float PBL(lat, lon) ;
> float WDIR10(lat, lon) ;
> float WSPD10(lat, lon) ;
> float TMP(lat, lon) ;
> float DPT(lat, lon) ;
>
> That would explain why it can't find that field. Also, it looks like
the
> fields in that file are 2D NetCDF, so I think you would want to
specify the
> level with the NetCDF notation, so instead of A8, you would use
"(*,*)"
>
> On Wed, Feb 3, 2021 at 10:43 AM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Wed Feb 03 10:42:39 2021: Request 98524 was acted upon.
> > Transaction: Ticket created by edward.strobach at noaa.gov
> > Queue: met_help
> > Subject: field name and level strings are not separated when
running
> > pointstat for a new model
> > Owner: Nobody
> > Requestors: edward.strobach at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524 >
> >
> >
> > Good afternoon,
> >
> > We are trying to generate statistics of the AQ model inline system
which
> > has a different file structure than the offline system. As
before, I
> > generate netcdf files with the fields I want to process which are
taken
> > from files of the inline system. So far, nothing has really
changed in
> my
> > processing steps, just ingesting data from a different source and
> grouping
> > verifiable fields. The netcdf file for a run that works looks
like this:
> >
> > For prod (offiline system that works):
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *>>> Dataset('aqm.t12z.metcro2d.05.nc
> > <http://aqm.t12z.metcro2d.05.nc>')<class
'netCDF4._netCDF4.Dataset'>root
> > group (NETCDF4 data model, file format HDF5): FileOrigins: NA
> > MET_version: V9.1 Projection: Lambert Conformal hemisphere:
N
> > accum: 00 scale_lat_1: 33.000000 scale_lat_2: 45.000000
lat_pin:
> > 21.821000 lon_pin: -120.628000 x_pin: 0.000000 y_pin:
0.000000
> > lon_orient: -97.000000 d_km: 12.000000 r_km: 6371.200000
nx: 442
> > ny: 265 grid_points dimensions(sizes): lon(442), lat(265)
> > variables(dimensions): float32 LAT(lat,lon), float32 LON(lat,lon),
> float32
> > PMTF(lat,lon), float32 OZCON(lat,lon) groups: *
> >
> > *For r101_v1 (inline system that doesn't work):*
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *Dataset('aqm.t12z.metcro2d.05.nc <http://aqm.t12z.metcro2d.05.nc
> >')<class
> > 'netCDF4._netCDF4.Dataset'>root group (NETCDF4 data model, file
format
> > HDF5): FileOrigins: NA MET_version: V9.1 hemisphere: N
> > Projection: Lambert Conformal nx: 191 ny: 97 grid_points
> > dimensions(sizes): lat(97), lon(191) variables(dimensions):
float32
> > LAT(lat,lon), float32 LON(lat,lon), float32 PBL(lat,lon), float32
> > WDIR10(lat,lon), float32 WSPD10(lat,lon), float32 TMP(lat,lon),
float32
> > DPT(lat,lon), float32 OZCON(lat,lon), float32 PMTF(lat,lon)
groups: *
> >
> > the meta data is shortened up for the inline system because I
don't have
> > all the information I need, but I don't think that's important
here.
> When
> > printing out the attributes of one of the inline fields, I see the
> expected
> > information:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *Dataset('aqm.t12z.metcro2d.05.nc
> > <http://aqm.t12z.metcro2d.05.nc>').variables['PMTF']<class
> > 'netCDF4._netCDF4.Variable'>float32 PMTF(lat, lon) long_name:
PMTF
> > standard_name: Surface PM2.5 units: ug/m**3 level: L1
init_time:
> > 20190810_120000 init_time_ut: 1565438400 valid_time:
> 20190810_170000
> > valid_time_ut: 1565456400*
> >
> > What's happening is that the long_name and level are being joined
in the
> > pointstat processing, which is strange to me. I don't know why
that
> would
> > be. In the log file you can see that they are separate up to a
point.
> > Here's print out confirming one such example:
> >
> >
> > *02/03 17:26:11.109 metplus.PointStat (command_builder.py:168)
DEBUG:
> > COPYABLE ENVIRONMENT FOR NEXT COMMAND:02/03 17:26:11.109
> metplus.PointStat
> > (command_builder.py:256) DEBUG: export CLIMO_MEAN_FILE=""; export
> > CLIMO_STDEV_FILE=""; export FCST_FIELD="{ name=\"PMTF\";
level=\"L1\";
> },{
> > name=\"OZCON\"; level=\"A8\"; }"; export
> >
> >
>
MET_TMP_DIR="/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/tmp";
> > export MODEL="R101_V1"; export OBS_FIELD="{ name=\"COPOPM\";
> level=\"A1\";
> > convert(x) = x * 10^9; },{*
> >
> > Below is where the error occurs:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *DEBUG 2: Reading data for PMTFL1.WARNING:WARNING:
> > process_fcst_climo_files() -> no fields matching PMTFL1 found in
file:
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190808/
> > aqm.t12z.metcro2d.36.nc
> > <http://aqm.t12z.metcro2d.36.nc>WARNING:DEBUG 2:DEBUG 2:
> >
> >
>
--------------------------------------------------------------------------------DEBUG
> > 2:DEBUG 2: Reading data for OZCONA8.WARNING:WARNING:
> > process_fcst_climo_files() -> no fields matching OZCONA8 found in
file:
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190808/
> > aqm.t12z.metcro2d.36.nc
> > <http://aqm.t12z.metcro2d.36.nc>WARNING:ERROR :ERROR :
> > process_fcst_climo_files() -> no requested forecast data found!
> > Exiting...ERROR :*
> >
> > You can see that the longname and level are joined, and as result,
fails
> > because that field doesn't exist. You can also clearly see that
my
> > configure file separates these fields:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *FCST_VAR1_NAME = OZCONFCST_VAR1_LEVELS = A1OBS_VAR1_NAME=
> > COPOOBS_VAR1_LEVELS= A1OBS_VAR1_OPTIONS = convert(x) = x *
> > 10^9;FCST_VAR2_NAME = OZCONFCST_VAR2_LEVELS = A8OBS_VAR2_NAME=
> > COPOOBS_VAR2_LEVELS= A8*
> >
> > It's not clear to me why this issue is occurring if everything
else
> appears
> > correct.
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: field name and level strings are not separated when running pointstat for a new model
From: George McCabe
Time: Wed Feb 03 13:02:43 2021
To clarify, if the level attribute of the field you are trying to read
matches the A8 level you requested, it should be able to find it that
way
instead of changing it to "(*,*)" but the reason for the error is due
to
the field not being found in the file.
On Wed, Feb 3, 2021 at 12:58 PM The RT System itself via RT <
met_help at ucar.edu> wrote:
>
> Wed Feb 03 12:58:33 2021: Request 98524 was acted upon.
> Transaction: Given to mccabe (George McCabe) by RT_System
> Queue: met_help
> Subject: field name and level strings are not separated when
running
> pointstat for a new model
> Owner: mccabe
> Requestors: edward.strobach at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524 >
>
>
> This transaction appears to have no content
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: field name and level strings are not separated when running pointstat for a new model
From: George McCabe
Time: Wed Feb 03 13:07:10 2021
Hi Edward,
What do you mean when you ask if it will work? If running through the
METplus wrappers, it should succeed for the files that have the field
available and fail for the rest. You will get a lot of errors, but it
will
still continue on and try to run the rest.
On Wed, Feb 3, 2021 at 1:02 PM Edward Strobach - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524 >
>
> Okay, thanks. I'm aware of this but it's only because certain files
don't
> contain that field. The forecast runs out to 48 hours, but the
surface
> ozone was only saved for the first 24 hours. The metcro files -
forecast
> hours 0 through 23 - should be able to be processed. Do all files
have to
> be present within the time range encompassing 48 hours to work?
>
> On Wed, Feb 3, 2021 at 2:58 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Edward,
> >
> > John pulled down the file from your log error message for me and
it looks
> > like the fields you are requesting are not found in that file:
> >
> > /usr/local/netcdf/bin/ncdump aqm.t12z.metcro2d.36.nc | grep float
> > float LAT(lat, lon) ;
> > float LON(lat, lon) ;
> > float PBL(lat, lon) ;
> > float WDIR10(lat, lon) ;
> > float WSPD10(lat, lon) ;
> > float TMP(lat, lon) ;
> > float DPT(lat, lon) ;
> >
> > That would explain why it can't find that field. Also, it looks
like the
> > fields in that file are 2D NetCDF, so I think you would want to
specify
> the
> > level with the NetCDF notation, so instead of A8, you would use
"(*,*)"
> >
> > On Wed, Feb 3, 2021 at 10:43 AM Edward Strobach - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Wed Feb 03 10:42:39 2021: Request 98524 was acted upon.
> > > Transaction: Ticket created by edward.strobach at noaa.gov
> > > Queue: met_help
> > > Subject: field name and level strings are not separated
when
> running
> > > pointstat for a new model
> > > Owner: Nobody
> > > Requestors: edward.strobach at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524
> >
> > >
> > >
> > > Good afternoon,
> > >
> > > We are trying to generate statistics of the AQ model inline
system
> which
> > > has a different file structure than the offline system. As
before, I
> > > generate netcdf files with the fields I want to process which
are taken
> > > from files of the inline system. So far, nothing has really
changed in
> > my
> > > processing steps, just ingesting data from a different source
and
> > grouping
> > > verifiable fields. The netcdf file for a run that works looks
like
> this:
> > >
> > > For prod (offiline system that works):
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *>>> Dataset('aqm.t12z.metcro2d.05.nc
> > > <http://aqm.t12z.metcro2d.05.nc>')<class
> 'netCDF4._netCDF4.Dataset'>root
> > > group (NETCDF4 data model, file format HDF5): FileOrigins: NA
> > > MET_version: V9.1 Projection: Lambert Conformal
hemisphere: N
> > > accum: 00 scale_lat_1: 33.000000 scale_lat_2: 45.000000
> lat_pin:
> > > 21.821000 lon_pin: -120.628000 x_pin: 0.000000 y_pin:
0.000000
> > > lon_orient: -97.000000 d_km: 12.000000 r_km: 6371.200000
nx:
> 442
> > > ny: 265 grid_points dimensions(sizes): lon(442), lat(265)
> > > variables(dimensions): float32 LAT(lat,lon), float32
LON(lat,lon),
> > float32
> > > PMTF(lat,lon), float32 OZCON(lat,lon) groups: *
> > >
> > > *For r101_v1 (inline system that doesn't work):*
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *Dataset('aqm.t12z.metcro2d.05.nc
<http://aqm.t12z.metcro2d.05.nc
> > >')<class
> > > 'netCDF4._netCDF4.Dataset'>root group (NETCDF4 data model, file
format
> > > HDF5): FileOrigins: NA MET_version: V9.1 hemisphere: N
> > > Projection: Lambert Conformal nx: 191 ny: 97 grid_points
> > > dimensions(sizes): lat(97), lon(191) variables(dimensions):
float32
> > > LAT(lat,lon), float32 LON(lat,lon), float32 PBL(lat,lon),
float32
> > > WDIR10(lat,lon), float32 WSPD10(lat,lon), float32 TMP(lat,lon),
float32
> > > DPT(lat,lon), float32 OZCON(lat,lon), float32 PMTF(lat,lon)
groups:
> *
> > >
> > > the meta data is shortened up for the inline system because I
don't
> have
> > > all the information I need, but I don't think that's important
here.
> > When
> > > printing out the attributes of one of the inline fields, I see
the
> > expected
> > > information:
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *Dataset('aqm.t12z.metcro2d.05.nc
> > > <http://aqm.t12z.metcro2d.05.nc>').variables['PMTF']<class
> > > 'netCDF4._netCDF4.Variable'>float32 PMTF(lat, lon) long_name:
PMTF
> > > standard_name: Surface PM2.5 units: ug/m**3 level: L1
> init_time:
> > > 20190810_120000 init_time_ut: 1565438400 valid_time:
> > 20190810_170000
> > > valid_time_ut: 1565456400*
> > >
> > > What's happening is that the long_name and level are being
joined in
> the
> > > pointstat processing, which is strange to me. I don't know why
that
> > would
> > > be. In the log file you can see that they are separate up to a
point.
> > > Here's print out confirming one such example:
> > >
> > >
> > > *02/03 17:26:11.109 metplus.PointStat (command_builder.py:168)
DEBUG:
> > > COPYABLE ENVIRONMENT FOR NEXT COMMAND:02/03 17:26:11.109
> > metplus.PointStat
> > > (command_builder.py:256) DEBUG: export CLIMO_MEAN_FILE="";
export
> > > CLIMO_STDEV_FILE=""; export FCST_FIELD="{ name=\"PMTF\";
level=\"L1\";
> > },{
> > > name=\"OZCON\"; level=\"A8\"; }"; export
> > >
> > >
> >
>
MET_TMP_DIR="/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/tmp";
> > > export MODEL="R101_V1"; export OBS_FIELD="{ name=\"COPOPM\";
> > level=\"A1\";
> > > convert(x) = x * 10^9; },{*
> > >
> > > Below is where the error occurs:
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *DEBUG 2: Reading data for PMTFL1.WARNING:WARNING:
> > > process_fcst_climo_files() -> no fields matching PMTFL1 found in
file:
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190808/
> > > aqm.t12z.metcro2d.36.nc
> > > <http://aqm.t12z.metcro2d.36.nc>WARNING:DEBUG 2:DEBUG 2:
> > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > 2:DEBUG 2: Reading data for OZCONA8.WARNING:WARNING:
> > > process_fcst_climo_files() -> no fields matching OZCONA8 found
in file:
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190808/
> > > aqm.t12z.metcro2d.36.nc
> > > <http://aqm.t12z.metcro2d.36.nc>WARNING:ERROR :ERROR :
> > > process_fcst_climo_files() -> no requested forecast data found!
> > > Exiting...ERROR :*
> > >
> > > You can see that the longname and level are joined, and as
result,
> fails
> > > because that field doesn't exist. You can also clearly see that
my
> > > configure file separates these fields:
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *FCST_VAR1_NAME = OZCONFCST_VAR1_LEVELS = A1OBS_VAR1_NAME=
> > > COPOOBS_VAR1_LEVELS= A1OBS_VAR1_OPTIONS = convert(x) = x *
> > > 10^9;FCST_VAR2_NAME = OZCONFCST_VAR2_LEVELS = A8OBS_VAR2_NAME=
> > > COPOOBS_VAR2_LEVELS= A8*
> > >
> > > It's not clear to me why this issue is occurring if everything
else
> > appears
> > > correct.
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: field name and level strings are not separated when running pointstat for a new model
From: Edward Strobach - NOAA Affiliate
Time: Wed Feb 03 13:13:39 2021
I'm trying to understand why this failure happens even though many of
these
files have the PMTF/OZCON fields in them. You won't find OZCON or
PMTF at
forecast hour 36. For the earlier ones you will. My log file was for
20190810, which has PMTF and OZCON for the metcro files. For example,
aqm.t12z.metcro2d.20.nc will have these fields while the
aqm.t12z.metcro2d.36.nc. Of course I will check these files for any
issues
before emailing you. That's what I did originally when I submitted
the
ticket.
On Wed, Feb 3, 2021 at 3:07 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Edward,
>
> What do you mean when you ask if it will work? If running through
the
> METplus wrappers, it should succeed for the files that have the
field
> available and fail for the rest. You will get a lot of errors, but
it will
> still continue on and try to run the rest.
>
> On Wed, Feb 3, 2021 at 1:02 PM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524 >
> >
> > Okay, thanks. I'm aware of this but it's only because certain
files
> don't
> > contain that field. The forecast runs out to 48 hours, but the
surface
> > ozone was only saved for the first 24 hours. The metcro files -
forecast
> > hours 0 through 23 - should be able to be processed. Do all files
have
> to
> > be present within the time range encompassing 48 hours to work?
> >
> > On Wed, Feb 3, 2021 at 2:58 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Edward,
> > >
> > > John pulled down the file from your log error message for me and
it
> looks
> > > like the fields you are requesting are not found in that file:
> > >
> > > /usr/local/netcdf/bin/ncdump aqm.t12z.metcro2d.36.nc | grep
float
> > > float LAT(lat, lon) ;
> > > float LON(lat, lon) ;
> > > float PBL(lat, lon) ;
> > > float WDIR10(lat, lon) ;
> > > float WSPD10(lat, lon) ;
> > > float TMP(lat, lon) ;
> > > float DPT(lat, lon) ;
> > >
> > > That would explain why it can't find that field. Also, it looks
like
> the
> > > fields in that file are 2D NetCDF, so I think you would want to
specify
> > the
> > > level with the NetCDF notation, so instead of A8, you would use
"(*,*)"
> > >
> > > On Wed, Feb 3, 2021 at 10:43 AM Edward Strobach - NOAA Affiliate
via
> RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Wed Feb 03 10:42:39 2021: Request 98524 was acted upon.
> > > > Transaction: Ticket created by edward.strobach at noaa.gov
> > > > Queue: met_help
> > > > Subject: field name and level strings are not separated
when
> > running
> > > > pointstat for a new model
> > > > Owner: Nobody
> > > > Requestors: edward.strobach at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524
> > >
> > > >
> > > >
> > > > Good afternoon,
> > > >
> > > > We are trying to generate statistics of the AQ model inline
system
> > which
> > > > has a different file structure than the offline system. As
before, I
> > > > generate netcdf files with the fields I want to process which
are
> taken
> > > > from files of the inline system. So far, nothing has really
changed
> in
> > > my
> > > > processing steps, just ingesting data from a different source
and
> > > grouping
> > > > verifiable fields. The netcdf file for a run that works looks
like
> > this:
> > > >
> > > > For prod (offiline system that works):
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *>>> Dataset('aqm.t12z.metcro2d.05.nc
> > > > <http://aqm.t12z.metcro2d.05.nc>')<class
> > 'netCDF4._netCDF4.Dataset'>root
> > > > group (NETCDF4 data model, file format HDF5): FileOrigins:
NA
> > > > MET_version: V9.1 Projection: Lambert Conformal
hemisphere: N
> > > > accum: 00 scale_lat_1: 33.000000 scale_lat_2: 45.000000
> > lat_pin:
> > > > 21.821000 lon_pin: -120.628000 x_pin: 0.000000 y_pin:
> 0.000000
> > > > lon_orient: -97.000000 d_km: 12.000000 r_km: 6371.200000
nx:
> > 442
> > > > ny: 265 grid_points dimensions(sizes): lon(442), lat(265)
> > > > variables(dimensions): float32 LAT(lat,lon), float32
LON(lat,lon),
> > > float32
> > > > PMTF(lat,lon), float32 OZCON(lat,lon) groups: *
> > > >
> > > > *For r101_v1 (inline system that doesn't work):*
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *Dataset('aqm.t12z.metcro2d.05.nc
<http://aqm.t12z.metcro2d.05.nc
> > > >')<class
> > > > 'netCDF4._netCDF4.Dataset'>root group (NETCDF4 data model,
file
> format
> > > > HDF5): FileOrigins: NA MET_version: V9.1 hemisphere:
N
> > > > Projection: Lambert Conformal nx: 191 ny: 97 grid_points
> > > > dimensions(sizes): lat(97), lon(191) variables(dimensions):
> float32
> > > > LAT(lat,lon), float32 LON(lat,lon), float32 PBL(lat,lon),
float32
> > > > WDIR10(lat,lon), float32 WSPD10(lat,lon), float32
TMP(lat,lon),
> float32
> > > > DPT(lat,lon), float32 OZCON(lat,lon), float32 PMTF(lat,lon)
> groups:
> > *
> > > >
> > > > the meta data is shortened up for the inline system because I
don't
> > have
> > > > all the information I need, but I don't think that's important
here.
> > > When
> > > > printing out the attributes of one of the inline fields, I see
the
> > > expected
> > > > information:
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *Dataset('aqm.t12z.metcro2d.05.nc
> > > > <http://aqm.t12z.metcro2d.05.nc>').variables['PMTF']<class
> > > > 'netCDF4._netCDF4.Variable'>float32 PMTF(lat, lon)
long_name: PMTF
> > > > standard_name: Surface PM2.5 units: ug/m**3 level: L1
> > init_time:
> > > > 20190810_120000 init_time_ut: 1565438400 valid_time:
> > > 20190810_170000
> > > > valid_time_ut: 1565456400*
> > > >
> > > > What's happening is that the long_name and level are being
joined in
> > the
> > > > pointstat processing, which is strange to me. I don't know
why that
> > > would
> > > > be. In the log file you can see that they are separate up to
a
> point.
> > > > Here's print out confirming one such example:
> > > >
> > > >
> > > > *02/03 17:26:11.109 metplus.PointStat (command_builder.py:168)
DEBUG:
> > > > COPYABLE ENVIRONMENT FOR NEXT COMMAND:02/03 17:26:11.109
> > > metplus.PointStat
> > > > (command_builder.py:256) DEBUG: export CLIMO_MEAN_FILE="";
export
> > > > CLIMO_STDEV_FILE=""; export FCST_FIELD="{ name=\"PMTF\";
> level=\"L1\";
> > > },{
> > > > name=\"OZCON\"; level=\"A8\"; }"; export
> > > >
> > > >
> > >
> >
>
MET_TMP_DIR="/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/tmp";
> > > > export MODEL="R101_V1"; export OBS_FIELD="{ name=\"COPOPM\";
> > > level=\"A1\";
> > > > convert(x) = x * 10^9; },{*
> > > >
> > > > Below is where the error occurs:
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *DEBUG 2: Reading data for PMTFL1.WARNING:WARNING:
> > > > process_fcst_climo_files() -> no fields matching PMTFL1 found
in
> file:
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190808/
> > > > aqm.t12z.metcro2d.36.nc
> > > > <http://aqm.t12z.metcro2d.36.nc>WARNING:DEBUG 2:DEBUG 2:
> > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > 2:DEBUG 2: Reading data for OZCONA8.WARNING:WARNING:
> > > > process_fcst_climo_files() -> no fields matching OZCONA8 found
in
> file:
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190808/
> > > > aqm.t12z.metcro2d.36.nc
> > > > <http://aqm.t12z.metcro2d.36.nc>WARNING:ERROR :ERROR :
> > > > process_fcst_climo_files() -> no requested forecast data
found!
> > > > Exiting...ERROR :*
> > > >
> > > > You can see that the longname and level are joined, and as
result,
> > fails
> > > > because that field doesn't exist. You can also clearly see
that my
> > > > configure file separates these fields:
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *FCST_VAR1_NAME = OZCONFCST_VAR1_LEVELS = A1OBS_VAR1_NAME=
> > > > COPOOBS_VAR1_LEVELS= A1OBS_VAR1_OPTIONS = convert(x) = x *
> > > > 10^9;FCST_VAR2_NAME = OZCONFCST_VAR2_LEVELS = A8OBS_VAR2_NAME=
> > > > COPOOBS_VAR2_LEVELS= A8*
> > > >
> > > > It's not clear to me why this issue is occurring if everything
else
> > > appears
> > > > correct.
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: field name and level strings are not separated when running pointstat for a new model
From: George McCabe
Time: Wed Feb 03 13:33:28 2021
Hi Edward,
It sounds like the log output combines the name and level when
reporting
that it can't find the data. Usually there is a forward slash in
between
them, like OZCON/A8, and I am not sure why it is not there in your log
output.
I asked John to pull down the 36 hour file because that was the path I
saw
in the log output right above the error. I didn't realize the file you
were
referring to was different. I will have John pull down a 20 hour file
from
the day you mentioned so I can take a look and look for clues as to
why it
would fail when the field name/level exists.
Thanks,
George
On Wed, Feb 3, 2021 at 1:13 PM Edward Strobach - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524 >
>
> I'm trying to understand why this failure happens even though many
of these
> files have the PMTF/OZCON fields in them. You won't find OZCON or
PMTF at
> forecast hour 36. For the earlier ones you will. My log file was
for
> 20190810, which has PMTF and OZCON for the metcro files. For
example,
> aqm.t12z.metcro2d.20.nc will have these fields while the
> aqm.t12z.metcro2d.36.nc. Of course I will check these files for any
> issues
> before emailing you. That's what I did originally when I submitted
the
> ticket.
>
> On Wed, Feb 3, 2021 at 3:07 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Edward,
> >
> > What do you mean when you ask if it will work? If running through
the
> > METplus wrappers, it should succeed for the files that have the
field
> > available and fail for the rest. You will get a lot of errors, but
it
> will
> > still continue on and try to run the rest.
> >
> > On Wed, Feb 3, 2021 at 1:02 PM Edward Strobach - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524 >
> > >
> > > Okay, thanks. I'm aware of this but it's only because certain
files
> > don't
> > > contain that field. The forecast runs out to 48 hours, but the
surface
> > > ozone was only saved for the first 24 hours. The metcro files -
> forecast
> > > hours 0 through 23 - should be able to be processed. Do all
files have
> > to
> > > be present within the time range encompassing 48 hours to work?
> > >
> > > On Wed, Feb 3, 2021 at 2:58 PM George McCabe via RT
<met_help at ucar.edu
> >
> > > wrote:
> > >
> > > > Hi Edward,
> > > >
> > > > John pulled down the file from your log error message for me
and it
> > looks
> > > > like the fields you are requesting are not found in that file:
> > > >
> > > > /usr/local/netcdf/bin/ncdump aqm.t12z.metcro2d.36.nc | grep
float
> > > > float LAT(lat, lon) ;
> > > > float LON(lat, lon) ;
> > > > float PBL(lat, lon) ;
> > > > float WDIR10(lat, lon) ;
> > > > float WSPD10(lat, lon) ;
> > > > float TMP(lat, lon) ;
> > > > float DPT(lat, lon) ;
> > > >
> > > > That would explain why it can't find that field. Also, it
looks like
> > the
> > > > fields in that file are 2D NetCDF, so I think you would want
to
> specify
> > > the
> > > > level with the NetCDF notation, so instead of A8, you would
use
> "(*,*)"
> > > >
> > > > On Wed, Feb 3, 2021 at 10:43 AM Edward Strobach - NOAA
Affiliate via
> > RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Wed Feb 03 10:42:39 2021: Request 98524 was acted upon.
> > > > > Transaction: Ticket created by edward.strobach at noaa.gov
> > > > > Queue: met_help
> > > > > Subject: field name and level strings are not separated
when
> > > running
> > > > > pointstat for a new model
> > > > > Owner: Nobody
> > > > > Requestors: edward.strobach at noaa.gov
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524
> > > >
> > > > >
> > > > >
> > > > > Good afternoon,
> > > > >
> > > > > We are trying to generate statistics of the AQ model inline
system
> > > which
> > > > > has a different file structure than the offline system. As
> before, I
> > > > > generate netcdf files with the fields I want to process
which are
> > taken
> > > > > from files of the inline system. So far, nothing has really
> changed
> > in
> > > > my
> > > > > processing steps, just ingesting data from a different
source and
> > > > grouping
> > > > > verifiable fields. The netcdf file for a run that works
looks like
> > > this:
> > > > >
> > > > > For prod (offiline system that works):
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *>>> Dataset('aqm.t12z.metcro2d.05.nc
> > > > > <http://aqm.t12z.metcro2d.05.nc>')<class
> > > 'netCDF4._netCDF4.Dataset'>root
> > > > > group (NETCDF4 data model, file format HDF5):
FileOrigins: NA
> > > > > MET_version: V9.1 Projection: Lambert Conformal
hemisphere: N
> > > > > accum: 00 scale_lat_1: 33.000000 scale_lat_2:
45.000000
> > > lat_pin:
> > > > > 21.821000 lon_pin: -120.628000 x_pin: 0.000000
y_pin:
> > 0.000000
> > > > > lon_orient: -97.000000 d_km: 12.000000 r_km:
6371.200000
> nx:
> > > 442
> > > > > ny: 265 grid_points dimensions(sizes): lon(442),
lat(265)
> > > > > variables(dimensions): float32 LAT(lat,lon), float32
LON(lat,lon),
> > > > float32
> > > > > PMTF(lat,lon), float32 OZCON(lat,lon) groups: *
> > > > >
> > > > > *For r101_v1 (inline system that doesn't work):*
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *Dataset('aqm.t12z.metcro2d.05.nc
<http://aqm.t12z.metcro2d.05.nc
> > > > >')<class
> > > > > 'netCDF4._netCDF4.Dataset'>root group (NETCDF4 data model,
file
> > format
> > > > > HDF5): FileOrigins: NA MET_version: V9.1
hemisphere: N
> > > > > Projection: Lambert Conformal nx: 191 ny: 97
grid_points
> > > > > dimensions(sizes): lat(97), lon(191)
variables(dimensions):
> > float32
> > > > > LAT(lat,lon), float32 LON(lat,lon), float32 PBL(lat,lon),
float32
> > > > > WDIR10(lat,lon), float32 WSPD10(lat,lon), float32
TMP(lat,lon),
> > float32
> > > > > DPT(lat,lon), float32 OZCON(lat,lon), float32 PMTF(lat,lon)
> > groups:
> > > *
> > > > >
> > > > > the meta data is shortened up for the inline system because
I don't
> > > have
> > > > > all the information I need, but I don't think that's
important
> here.
> > > > When
> > > > > printing out the attributes of one of the inline fields, I
see the
> > > > expected
> > > > > information:
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *Dataset('aqm.t12z.metcro2d.05.nc
> > > > > <http://aqm.t12z.metcro2d.05.nc>').variables['PMTF']<class
> > > > > 'netCDF4._netCDF4.Variable'>float32 PMTF(lat, lon)
long_name:
> PMTF
> > > > > standard_name: Surface PM2.5 units: ug/m**3 level: L1
> > > init_time:
> > > > > 20190810_120000 init_time_ut: 1565438400 valid_time:
> > > > 20190810_170000
> > > > > valid_time_ut: 1565456400*
> > > > >
> > > > > What's happening is that the long_name and level are being
joined
> in
> > > the
> > > > > pointstat processing, which is strange to me. I don't know
why
> that
> > > > would
> > > > > be. In the log file you can see that they are separate up
to a
> > point.
> > > > > Here's print out confirming one such example:
> > > > >
> > > > >
> > > > > *02/03 17:26:11.109 metplus.PointStat
(command_builder.py:168)
> DEBUG:
> > > > > COPYABLE ENVIRONMENT FOR NEXT COMMAND:02/03 17:26:11.109
> > > > metplus.PointStat
> > > > > (command_builder.py:256) DEBUG: export CLIMO_MEAN_FILE="";
export
> > > > > CLIMO_STDEV_FILE=""; export FCST_FIELD="{ name=\"PMTF\";
> > level=\"L1\";
> > > > },{
> > > > > name=\"OZCON\"; level=\"A8\"; }"; export
> > > > >
> > > > >
> > > >
> > >
> >
>
MET_TMP_DIR="/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/tmp";
> > > > > export MODEL="R101_V1"; export OBS_FIELD="{ name=\"COPOPM\";
> > > > level=\"A1\";
> > > > > convert(x) = x * 10^9; },{*
> > > > >
> > > > > Below is where the error occurs:
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *DEBUG 2: Reading data for PMTFL1.WARNING:WARNING:
> > > > > process_fcst_climo_files() -> no fields matching PMTFL1
found in
> > file:
> > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190808/
> > > > > aqm.t12z.metcro2d.36.nc
> > > > > <http://aqm.t12z.metcro2d.36.nc>WARNING:DEBUG 2:DEBUG 2:
> > > > >
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > > 2:DEBUG 2: Reading data for OZCONA8.WARNING:WARNING:
> > > > > process_fcst_climo_files() -> no fields matching OZCONA8
found in
> > file:
> > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190808/
> > > > > aqm.t12z.metcro2d.36.nc
> > > > > <http://aqm.t12z.metcro2d.36.nc>WARNING:ERROR :ERROR :
> > > > > process_fcst_climo_files() -> no requested forecast data
found!
> > > > > Exiting...ERROR :*
> > > > >
> > > > > You can see that the longname and level are joined, and as
result,
> > > fails
> > > > > because that field doesn't exist. You can also clearly see
that my
> > > > > configure file separates these fields:
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *FCST_VAR1_NAME = OZCONFCST_VAR1_LEVELS = A1OBS_VAR1_NAME=
> > > > > COPOOBS_VAR1_LEVELS= A1OBS_VAR1_OPTIONS = convert(x) = x *
> > > > > 10^9;FCST_VAR2_NAME = OZCONFCST_VAR2_LEVELS =
A8OBS_VAR2_NAME=
> > > > > COPOOBS_VAR2_LEVELS= A8*
> > > > >
> > > > > It's not clear to me why this issue is occurring if
everything else
> > > > appears
> > > > > correct.
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: field name and level strings are not separated when running pointstat for a new model
From: George McCabe
Time: Wed Feb 03 14:33:29 2021
Hi Edward,
Could you send me the command you are trying to run as well as the
full log
file and the MET config file you are using? I'm not understanding what
the
issue is from this information. It looks like it should run fine for
the
forecasts 0-23 and it should fail for the leads above that since the
fields
are not present.
On Wed, Feb 3, 2021 at 1:33 PM George McCabe <mccabe at ucar.edu> wrote:
> Hi Edward,
>
> It sounds like the log output combines the name and level when
reporting
> that it can't find the data. Usually there is a forward slash in
between
> them, like OZCON/A8, and I am not sure why it is not there in your
log
> output.
>
> I asked John to pull down the 36 hour file because that was the path
I saw
> in the log output right above the error. I didn't realize the file
you were
> referring to was different. I will have John pull down a 20 hour
file from
> the day you mentioned so I can take a look and look for clues as to
why it
> would fail when the field name/level exists.
>
> Thanks,
> George
>
> On Wed, Feb 3, 2021 at 1:13 PM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524 >
>>
>> I'm trying to understand why this failure happens even though many
of
>> these
>> files have the PMTF/OZCON fields in them. You won't find OZCON or
PMTF at
>> forecast hour 36. For the earlier ones you will. My log file was
for
>> 20190810, which has PMTF and OZCON for the metcro files. For
example,
>> aqm.t12z.metcro2d.20.nc will have these fields while the
>> aqm.t12z.metcro2d.36.nc. Of course I will check these files for
any
>> issues
>> before emailing you. That's what I did originally when I submitted
the
>> ticket.
>>
>> On Wed, Feb 3, 2021 at 3:07 PM George McCabe via RT
<met_help at ucar.edu>
>> wrote:
>>
>> > Hi Edward,
>> >
>> > What do you mean when you ask if it will work? If running through
the
>> > METplus wrappers, it should succeed for the files that have the
field
>> > available and fail for the rest. You will get a lot of errors,
but it
>> will
>> > still continue on and try to run the rest.
>> >
>> > On Wed, Feb 3, 2021 at 1:02 PM Edward Strobach - NOAA Affiliate
via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524 >
>> > >
>> > > Okay, thanks. I'm aware of this but it's only because certain
files
>> > don't
>> > > contain that field. The forecast runs out to 48 hours, but the
>> surface
>> > > ozone was only saved for the first 24 hours. The metcro files
-
>> forecast
>> > > hours 0 through 23 - should be able to be processed. Do all
files
>> have
>> > to
>> > > be present within the time range encompassing 48 hours to work?
>> > >
>> > > On Wed, Feb 3, 2021 at 2:58 PM George McCabe via RT <
>> met_help at ucar.edu>
>> > > wrote:
>> > >
>> > > > Hi Edward,
>> > > >
>> > > > John pulled down the file from your log error message for me
and it
>> > looks
>> > > > like the fields you are requesting are not found in that
file:
>> > > >
>> > > > /usr/local/netcdf/bin/ncdump aqm.t12z.metcro2d.36.nc | grep
float
>> > > > float LAT(lat, lon) ;
>> > > > float LON(lat, lon) ;
>> > > > float PBL(lat, lon) ;
>> > > > float WDIR10(lat, lon) ;
>> > > > float WSPD10(lat, lon) ;
>> > > > float TMP(lat, lon) ;
>> > > > float DPT(lat, lon) ;
>> > > >
>> > > > That would explain why it can't find that field. Also, it
looks like
>> > the
>> > > > fields in that file are 2D NetCDF, so I think you would want
to
>> specify
>> > > the
>> > > > level with the NetCDF notation, so instead of A8, you would
use
>> "(*,*)"
>> > > >
>> > > > On Wed, Feb 3, 2021 at 10:43 AM Edward Strobach - NOAA
Affiliate via
>> > RT <
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > > >
>> > > > > Wed Feb 03 10:42:39 2021: Request 98524 was acted upon.
>> > > > > Transaction: Ticket created by edward.strobach at noaa.gov
>> > > > > Queue: met_help
>> > > > > Subject: field name and level strings are not
separated when
>> > > running
>> > > > > pointstat for a new model
>> > > > > Owner: Nobody
>> > > > > Requestors: edward.strobach at noaa.gov
>> > > > > Status: new
>> > > > > Ticket <URL:
>> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524
>> > > >
>> > > > >
>> > > > >
>> > > > > Good afternoon,
>> > > > >
>> > > > > We are trying to generate statistics of the AQ model inline
system
>> > > which
>> > > > > has a different file structure than the offline system. As
>> before, I
>> > > > > generate netcdf files with the fields I want to process
which are
>> > taken
>> > > > > from files of the inline system. So far, nothing has
really
>> changed
>> > in
>> > > > my
>> > > > > processing steps, just ingesting data from a different
source and
>> > > > grouping
>> > > > > verifiable fields. The netcdf file for a run that works
looks
>> like
>> > > this:
>> > > > >
>> > > > > For prod (offiline system that works):
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > *>>> Dataset('aqm.t12z.metcro2d.05.nc
>> > > > > <http://aqm.t12z.metcro2d.05.nc>')<class
>> > > 'netCDF4._netCDF4.Dataset'>root
>> > > > > group (NETCDF4 data model, file format HDF5):
FileOrigins: NA
>> > > > > MET_version: V9.1 Projection: Lambert Conformal
hemisphere:
>> N
>> > > > > accum: 00 scale_lat_1: 33.000000 scale_lat_2:
45.000000
>> > > lat_pin:
>> > > > > 21.821000 lon_pin: -120.628000 x_pin: 0.000000
y_pin:
>> > 0.000000
>> > > > > lon_orient: -97.000000 d_km: 12.000000 r_km:
6371.200000
>> nx:
>> > > 442
>> > > > > ny: 265 grid_points dimensions(sizes): lon(442),
lat(265)
>> > > > > variables(dimensions): float32 LAT(lat,lon), float32
LON(lat,lon),
>> > > > float32
>> > > > > PMTF(lat,lon), float32 OZCON(lat,lon) groups: *
>> > > > >
>> > > > > *For r101_v1 (inline system that doesn't work):*
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > *Dataset('aqm.t12z.metcro2d.05.nc
<http://aqm.t12z.metcro2d.05.nc
>> > > > >')<class
>> > > > > 'netCDF4._netCDF4.Dataset'>root group (NETCDF4 data model,
file
>> > format
>> > > > > HDF5): FileOrigins: NA MET_version: V9.1
hemisphere: N
>> > > > > Projection: Lambert Conformal nx: 191 ny: 97
grid_points
>> > > > > dimensions(sizes): lat(97), lon(191)
variables(dimensions):
>> > float32
>> > > > > LAT(lat,lon), float32 LON(lat,lon), float32 PBL(lat,lon),
float32
>> > > > > WDIR10(lat,lon), float32 WSPD10(lat,lon), float32
TMP(lat,lon),
>> > float32
>> > > > > DPT(lat,lon), float32 OZCON(lat,lon), float32 PMTF(lat,lon)
>> > groups:
>> > > *
>> > > > >
>> > > > > the meta data is shortened up for the inline system because
I
>> don't
>> > > have
>> > > > > all the information I need, but I don't think that's
important
>> here.
>> > > > When
>> > > > > printing out the attributes of one of the inline fields, I
see the
>> > > > expected
>> > > > > information:
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > *Dataset('aqm.t12z.metcro2d.05.nc
>> > > > > <http://aqm.t12z.metcro2d.05.nc>').variables['PMTF']<class
>> > > > > 'netCDF4._netCDF4.Variable'>float32 PMTF(lat, lon)
long_name:
>> PMTF
>> > > > > standard_name: Surface PM2.5 units: ug/m**3 level: L1
>> > > init_time:
>> > > > > 20190810_120000 init_time_ut: 1565438400 valid_time:
>> > > > 20190810_170000
>> > > > > valid_time_ut: 1565456400*
>> > > > >
>> > > > > What's happening is that the long_name and level are being
joined
>> in
>> > > the
>> > > > > pointstat processing, which is strange to me. I don't know
why
>> that
>> > > > would
>> > > > > be. In the log file you can see that they are separate up
to a
>> > point.
>> > > > > Here's print out confirming one such example:
>> > > > >
>> > > > >
>> > > > > *02/03 17:26:11.109 metplus.PointStat
(command_builder.py:168)
>> DEBUG:
>> > > > > COPYABLE ENVIRONMENT FOR NEXT COMMAND:02/03 17:26:11.109
>> > > > metplus.PointStat
>> > > > > (command_builder.py:256) DEBUG: export CLIMO_MEAN_FILE="";
export
>> > > > > CLIMO_STDEV_FILE=""; export FCST_FIELD="{ name=\"PMTF\";
>> > level=\"L1\";
>> > > > },{
>> > > > > name=\"OZCON\"; level=\"A8\"; }"; export
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
MET_TMP_DIR="/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/tmp";
>> > > > > export MODEL="R101_V1"; export OBS_FIELD="{
name=\"COPOPM\";
>> > > > level=\"A1\";
>> > > > > convert(x) = x * 10^9; },{*
>> > > > >
>> > > > > Below is where the error occurs:
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > *DEBUG 2: Reading data for PMTFL1.WARNING:WARNING:
>> > > > > process_fcst_climo_files() -> no fields matching PMTFL1
found in
>> > file:
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190808/
>> > > > > aqm.t12z.metcro2d.36.nc
>> > > > > <http://aqm.t12z.metcro2d.36.nc>WARNING:DEBUG 2:DEBUG 2:
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
--------------------------------------------------------------------------------DEBUG
>> > > > > 2:DEBUG 2: Reading data for OZCONA8.WARNING:WARNING:
>> > > > > process_fcst_climo_files() -> no fields matching OZCONA8
found in
>> > file:
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190808/
>> > > > > aqm.t12z.metcro2d.36.nc
>> > > > > <http://aqm.t12z.metcro2d.36.nc>WARNING:ERROR :ERROR :
>> > > > > process_fcst_climo_files() -> no requested forecast data
found!
>> > > > > Exiting...ERROR :*
>> > > > >
>> > > > > You can see that the longname and level are joined, and as
result,
>> > > fails
>> > > > > because that field doesn't exist. You can also clearly see
that
>> my
>> > > > > configure file separates these fields:
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > *FCST_VAR1_NAME = OZCONFCST_VAR1_LEVELS = A1OBS_VAR1_NAME=
>> > > > > COPOOBS_VAR1_LEVELS= A1OBS_VAR1_OPTIONS = convert(x) = x *
>> > > > > 10^9;FCST_VAR2_NAME = OZCONFCST_VAR2_LEVELS =
A8OBS_VAR2_NAME=
>> > > > > COPOOBS_VAR2_LEVELS= A8*
>> > > > >
>> > > > > It's not clear to me why this issue is occurring if
everything
>> else
>> > > > appears
>> > > > > correct.
>> > > > >
>> > > > >
>> > > >
>> > > > --
>> > > > George McCabe - Software Engineer III
>> > > > National Center for Atmospheric Research
>> > > > Research Applications Laboratory
>> > > > 303-497-2768
>> > > > ---
>> > > > My working day may not be your working day. Please do not
feel
>> obliged
>> > to
>> > > > reply to this email outside of your normal working hours.
>> > > >
>> > > >
>> > >
>> > >
>> >
>> > --
>> > George McCabe - Software Engineer III
>> > National Center for Atmospheric Research
>> > Research Applications Laboratory
>> > 303-497-2768
>> > ---
>> > My working day may not be your working day. Please do not feel
obliged
>> to
>> > reply to this email outside of your normal working hours.
>> >
>> >
>>
>>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: field name and level strings are not separated when running pointstat for a new model
From: Edward Strobach - NOAA Affiliate
Time: Wed Feb 03 15:04:53 2021
Hi,
There is no specific command since I'm running this all under a bash
script
that has a series different processing steps. I can say that I
construct
several configure files that are then passed into master_metplus.py.
This
has worked for me for some time, so I don't think commands are the
issue
here. The log file is way to big too attach but can be viewed here:
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210203172449.
You can start on line 84297, for example; that is for the
aqm.t12z.metcro2d.06.nc file. Shortly below that is an instance where
the
level and name is joined:
*DEBUG 1: Forecast File:
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190810/aqm.t12z.metcro2d.06.nc
<http://aqm.t12z.metcro2d.06.nc>DEBUG 1: Observation File:
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r101_v1/aq/prepbufr.aqm.20190810.nc
<http://prepbufr.aqm.20190810.nc>DEBUG 7: check_nc_data_2d() count:
valid=18527, zero=0, missing=0, 18527 == 18527DEBUG 7:
MetNcFile::data()
const -> took 0 seconds to read NetCDF dataDEBUG 7: MetNcFile::data()
const
-> took 0 seconds to fill data planeDEBUG 2:DEBUG 2:
--------------------------------------------------------------------------------DEBUG
2:DEBUG 2: Reading data for OZCONA1.DEBUG 3: Grid Definition:
Projection:
Lambert Conformal Nx: 191 Ny: 97 Lat_LL: -89.986 Lon_LL: 142.318
Lon_orient: -81.000 Alpha: 1.950 Cone: 0.988 Bx: -9999.0000 By:
-9998.8416DEBUG 2: Processing masking regions.DEBUG 3: Processing grid
mask: FULL*
There are some things that are filled because they are not being
specified
in the meta data structure. This seems to be carried over as seen
above.
For example,
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_NcMet".
DEBUG 4:
DEBUG 4: Lambert Conformal Grid Data:
DEBUG 4: hemisphere: N
*DEBUG 4: scale_lat_1: -9999DEBUG 4: scale_lat_2: -9999DEBUG 4:
lat_pin: -9999DEBUG 4: lon_pin: 9999DEBUG 4: x_pin:
-9999DEBUG 4: y_pin: -9999DEBUG 4: lon_orient: 9999DEBUG 4:
d_km: -9999DEBUG 4: r_km: -9999*
DEBUG 4: nx: 191
DEBUG 4: ny: 97
DEBUG 4: so2_angle: 0
I'm not sure if this is responsible for the issue. Do I need to
define all
grid attributes in the netcdf file?
Here's the configure file. Everything looks okay there to me.
On Wed, Feb 3, 2021 at 4:33 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Edward,
>
> Could you send me the command you are trying to run as well as the
full log
> file and the MET config file you are using? I'm not understanding
what the
> issue is from this information. It looks like it should run fine for
the
> forecasts 0-23 and it should fail for the leads above that since the
fields
> are not present.
>
> On Wed, Feb 3, 2021 at 1:33 PM George McCabe <mccabe at ucar.edu>
wrote:
>
> > Hi Edward,
> >
> > It sounds like the log output combines the name and level when
reporting
> > that it can't find the data. Usually there is a forward slash in
between
> > them, like OZCON/A8, and I am not sure why it is not there in your
log
> > output.
> >
> > I asked John to pull down the 36 hour file because that was the
path I
> saw
> > in the log output right above the error. I didn't realize the file
you
> were
> > referring to was different. I will have John pull down a 20 hour
file
> from
> > the day you mentioned so I can take a look and look for clues as
to why
> it
> > would fail when the field name/level exists.
> >
> > Thanks,
> > George
> >
> > On Wed, Feb 3, 2021 at 1:13 PM Edward Strobach - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524 >
> >>
> >> I'm trying to understand why this failure happens even though
many of
> >> these
> >> files have the PMTF/OZCON fields in them. You won't find OZCON
or PMTF
> at
> >> forecast hour 36. For the earlier ones you will. My log file
was for
> >> 20190810, which has PMTF and OZCON for the metcro files. For
example,
> >> aqm.t12z.metcro2d.20.nc will have these fields while the
> >> aqm.t12z.metcro2d.36.nc. Of course I will check these files for
any
> >> issues
> >> before emailing you. That's what I did originally when I
submitted the
> >> ticket.
> >>
> >> On Wed, Feb 3, 2021 at 3:07 PM George McCabe via RT
<met_help at ucar.edu>
> >> wrote:
> >>
> >> > Hi Edward,
> >> >
> >> > What do you mean when you ask if it will work? If running
through the
> >> > METplus wrappers, it should succeed for the files that have the
field
> >> > available and fail for the rest. You will get a lot of errors,
but it
> >> will
> >> > still continue on and try to run the rest.
> >> >
> >> > On Wed, Feb 3, 2021 at 1:02 PM Edward Strobach - NOAA Affiliate
via
> RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524
>
> >> > >
> >> > > Okay, thanks. I'm aware of this but it's only because
certain files
> >> > don't
> >> > > contain that field. The forecast runs out to 48 hours, but
the
> >> surface
> >> > > ozone was only saved for the first 24 hours. The metcro
files -
> >> forecast
> >> > > hours 0 through 23 - should be able to be processed. Do all
files
> >> have
> >> > to
> >> > > be present within the time range encompassing 48 hours to
work?
> >> > >
> >> > > On Wed, Feb 3, 2021 at 2:58 PM George McCabe via RT <
> >> met_help at ucar.edu>
> >> > > wrote:
> >> > >
> >> > > > Hi Edward,
> >> > > >
> >> > > > John pulled down the file from your log error message for
me and
> it
> >> > looks
> >> > > > like the fields you are requesting are not found in that
file:
> >> > > >
> >> > > > /usr/local/netcdf/bin/ncdump aqm.t12z.metcro2d.36.nc | grep
float
> >> > > > float LAT(lat, lon) ;
> >> > > > float LON(lat, lon) ;
> >> > > > float PBL(lat, lon) ;
> >> > > > float WDIR10(lat, lon) ;
> >> > > > float WSPD10(lat, lon) ;
> >> > > > float TMP(lat, lon) ;
> >> > > > float DPT(lat, lon) ;
> >> > > >
> >> > > > That would explain why it can't find that field. Also, it
looks
> like
> >> > the
> >> > > > fields in that file are 2D NetCDF, so I think you would
want to
> >> specify
> >> > > the
> >> > > > level with the NetCDF notation, so instead of A8, you would
use
> >> "(*,*)"
> >> > > >
> >> > > > On Wed, Feb 3, 2021 at 10:43 AM Edward Strobach - NOAA
Affiliate
> via
> >> > RT <
> >> > > > met_help at ucar.edu> wrote:
> >> > > >
> >> > > > >
> >> > > > > Wed Feb 03 10:42:39 2021: Request 98524 was acted upon.
> >> > > > > Transaction: Ticket created by edward.strobach at noaa.gov
> >> > > > > Queue: met_help
> >> > > > > Subject: field name and level strings are not
separated
> when
> >> > > running
> >> > > > > pointstat for a new model
> >> > > > > Owner: Nobody
> >> > > > > Requestors: edward.strobach at noaa.gov
> >> > > > > Status: new
> >> > > > > Ticket <URL:
> >> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524
> >> > > >
> >> > > > >
> >> > > > >
> >> > > > > Good afternoon,
> >> > > > >
> >> > > > > We are trying to generate statistics of the AQ model
inline
> system
> >> > > which
> >> > > > > has a different file structure than the offline system.
As
> >> before, I
> >> > > > > generate netcdf files with the fields I want to process
which
> are
> >> > taken
> >> > > > > from files of the inline system. So far, nothing has
really
> >> changed
> >> > in
> >> > > > my
> >> > > > > processing steps, just ingesting data from a different
source
> and
> >> > > > grouping
> >> > > > > verifiable fields. The netcdf file for a run that works
looks
> >> like
> >> > > this:
> >> > > > >
> >> > > > > For prod (offiline system that works):
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > *>>> Dataset('aqm.t12z.metcro2d.05.nc
> >> > > > > <http://aqm.t12z.metcro2d.05.nc>')<class
> >> > > 'netCDF4._netCDF4.Dataset'>root
> >> > > > > group (NETCDF4 data model, file format HDF5):
FileOrigins: NA
> >> > > > > MET_version: V9.1 Projection: Lambert Conformal
> hemisphere:
> >> N
> >> > > > > accum: 00 scale_lat_1: 33.000000 scale_lat_2:
45.000000
> >> > > lat_pin:
> >> > > > > 21.821000 lon_pin: -120.628000 x_pin: 0.000000
y_pin:
> >> > 0.000000
> >> > > > > lon_orient: -97.000000 d_km: 12.000000 r_km:
6371.200000
> >> nx:
> >> > > 442
> >> > > > > ny: 265 grid_points dimensions(sizes): lon(442),
lat(265)
> >> > > > > variables(dimensions): float32 LAT(lat,lon), float32
> LON(lat,lon),
> >> > > > float32
> >> > > > > PMTF(lat,lon), float32 OZCON(lat,lon) groups: *
> >> > > > >
> >> > > > > *For r101_v1 (inline system that doesn't work):*
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > *Dataset('aqm.t12z.metcro2d.05.nc <
> http://aqm.t12z.metcro2d.05.nc
> >> > > > >')<class
> >> > > > > 'netCDF4._netCDF4.Dataset'>root group (NETCDF4 data
model, file
> >> > format
> >> > > > > HDF5): FileOrigins: NA MET_version: V9.1
hemisphere: N
> >> > > > > Projection: Lambert Conformal nx: 191 ny: 97
grid_points
> >> > > > > dimensions(sizes): lat(97), lon(191)
variables(dimensions):
> >> > float32
> >> > > > > LAT(lat,lon), float32 LON(lat,lon), float32 PBL(lat,lon),
> float32
> >> > > > > WDIR10(lat,lon), float32 WSPD10(lat,lon), float32
TMP(lat,lon),
> >> > float32
> >> > > > > DPT(lat,lon), float32 OZCON(lat,lon), float32
PMTF(lat,lon)
> >> > groups:
> >> > > *
> >> > > > >
> >> > > > > the meta data is shortened up for the inline system
because I
> >> don't
> >> > > have
> >> > > > > all the information I need, but I don't think that's
important
> >> here.
> >> > > > When
> >> > > > > printing out the attributes of one of the inline fields,
I see
> the
> >> > > > expected
> >> > > > > information:
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > *Dataset('aqm.t12z.metcro2d.05.nc
> >> > > > >
<http://aqm.t12z.metcro2d.05.nc>').variables['PMTF']<class
> >> > > > > 'netCDF4._netCDF4.Variable'>float32 PMTF(lat, lon)
long_name:
> >> PMTF
> >> > > > > standard_name: Surface PM2.5 units: ug/m**3 level:
L1
> >> > > init_time:
> >> > > > > 20190810_120000 init_time_ut: 1565438400
valid_time:
> >> > > > 20190810_170000
> >> > > > > valid_time_ut: 1565456400*
> >> > > > >
> >> > > > > What's happening is that the long_name and level are
being
> joined
> >> in
> >> > > the
> >> > > > > pointstat processing, which is strange to me. I don't
know why
> >> that
> >> > > > would
> >> > > > > be. In the log file you can see that they are separate
up to a
> >> > point.
> >> > > > > Here's print out confirming one such example:
> >> > > > >
> >> > > > >
> >> > > > > *02/03 17:26:11.109 metplus.PointStat
(command_builder.py:168)
> >> DEBUG:
> >> > > > > COPYABLE ENVIRONMENT FOR NEXT COMMAND:02/03 17:26:11.109
> >> > > > metplus.PointStat
> >> > > > > (command_builder.py:256) DEBUG: export
CLIMO_MEAN_FILE="";
> export
> >> > > > > CLIMO_STDEV_FILE=""; export FCST_FIELD="{ name=\"PMTF\";
> >> > level=\"L1\";
> >> > > > },{
> >> > > > > name=\"OZCON\"; level=\"A8\"; }"; export
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
MET_TMP_DIR="/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/tmp";
> >> > > > > export MODEL="R101_V1"; export OBS_FIELD="{
name=\"COPOPM\";
> >> > > > level=\"A1\";
> >> > > > > convert(x) = x * 10^9; },{*
> >> > > > >
> >> > > > > Below is where the error occurs:
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > *DEBUG 2: Reading data for PMTFL1.WARNING:WARNING:
> >> > > > > process_fcst_climo_files() -> no fields matching PMTFL1
found in
> >> > file:
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190808/
> >> > > > > aqm.t12z.metcro2d.36.nc
> >> > > > > <http://aqm.t12z.metcro2d.36.nc>WARNING:DEBUG 2:DEBUG 2:
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
--------------------------------------------------------------------------------DEBUG
> >> > > > > 2:DEBUG 2: Reading data for OZCONA8.WARNING:WARNING:
> >> > > > > process_fcst_climo_files() -> no fields matching OZCONA8
found
> in
> >> > file:
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190808/
> >> > > > > aqm.t12z.metcro2d.36.nc
> >> > > > > <http://aqm.t12z.metcro2d.36.nc>WARNING:ERROR :ERROR :
> >> > > > > process_fcst_climo_files() -> no requested forecast data
found!
> >> > > > > Exiting...ERROR :*
> >> > > > >
> >> > > > > You can see that the longname and level are joined, and
as
> result,
> >> > > fails
> >> > > > > because that field doesn't exist. You can also clearly
see that
> >> my
> >> > > > > configure file separates these fields:
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > *FCST_VAR1_NAME = OZCONFCST_VAR1_LEVELS =
A1OBS_VAR1_NAME=
> >> > > > > COPOOBS_VAR1_LEVELS= A1OBS_VAR1_OPTIONS = convert(x) = x
*
> >> > > > > 10^9;FCST_VAR2_NAME = OZCONFCST_VAR2_LEVELS =
A8OBS_VAR2_NAME=
> >> > > > > COPOOBS_VAR2_LEVELS= A8*
> >> > > > >
> >> > > > > It's not clear to me why this issue is occurring if
everything
> >> else
> >> > > > appears
> >> > > > > correct.
> >> > > > >
> >> > > > >
> >> > > >
> >> > > > --
> >> > > > George McCabe - Software Engineer III
> >> > > > National Center for Atmospheric Research
> >> > > > Research Applications Laboratory
> >> > > > 303-497-2768
> >> > > > ---
> >> > > > My working day may not be your working day. Please do not
feel
> >> obliged
> >> > to
> >> > > > reply to this email outside of your normal working hours.
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> >
> >> > --
> >> > George McCabe - Software Engineer III
> >> > National Center for Atmospheric Research
> >> > Research Applications Laboratory
> >> > 303-497-2768
> >> > ---
> >> > My working day may not be your working day. Please do not feel
obliged
> >> to
> >> > reply to this email outside of your normal working hours.
> >> >
> >> >
> >>
> >>
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: field name and level strings are not separated when running pointstat for a new model
From: George McCabe
Time: Fri Feb 05 13:45:40 2021
Hi Edward,
Sorry for the delayed response.
The log output that appears to combine the name and level only
combines them in the log output. There should be a slash in between
the name and level, i.e. OZCON/A8, but the actual data that is read is
correct.
I looked at your log file and it appears to be running as expected. It
fails as expected only for forecast leads about 23 where the fields
don't exist, and succeeds otherwise. Reducing the list of forecast
leads would prevent processing the files that don't have the required
data. I see you are using:
INIT_SEQ = 12
LEAD_SEQ_MAX = 48
This will process leads that match up with files that are initialized
at 12Z up to a 48 hour lead. Changing the max value to 24 (or 23?)
would prevent the additional runs that are failing.
Let me know if I am still not understanding what you are asking for
this issue.
Thanks,
George
On Wed Feb 03 15:04:53 2021, edward.strobach at noaa.gov wrote:
> Hi,
>
> There is no specific command since I'm running this all under a bash
> script
> that has a series different processing steps. I can say that I
> construct
> several configure files that are then passed into master_metplus.py.
> This
> has worked for me for some time, so I don't think commands are the
> issue
> here. The log file is way to big too attach but can be viewed here:
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210203172449.
>
>
> You can start on line 84297, for example; that is for the
> aqm.t12z.metcro2d.06.nc file. Shortly below that is an instance
where
> the
> level and name is joined:
>
>
>
>
>
>
>
>
>
>
>
> *DEBUG 1: Forecast File:
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190810/aqm.t12z.metcro2d.06.nc
> <http://aqm.t12z.metcro2d.06.nc>DEBUG 1: Observation File:
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r101_v1/aq/prepbufr.aqm.20190810.nc
> <http://prepbufr.aqm.20190810.nc>DEBUG 7: check_nc_data_2d() count:
> valid=18527, zero=0, missing=0, 18527 == 18527DEBUG 7:
> MetNcFile::data()
> const -> took 0 seconds to read NetCDF dataDEBUG 7:
MetNcFile::data()
> const
> -> took 0 seconds to fill data planeDEBUG 2:DEBUG 2:
>
--------------------------------------------------------------------------------DEBUG
> 2:DEBUG 2: Reading data for OZCONA1.DEBUG 3: Grid Definition:
> Projection:
> Lambert Conformal Nx: 191 Ny: 97 Lat_LL: -89.986 Lon_LL: 142.318
> Lon_orient: -81.000 Alpha: 1.950 Cone: 0.988 Bx: -9999.0000 By:
> -9998.8416DEBUG 2: Processing masking regions.DEBUG 3: Processing
grid
> mask: FULL*
>
> There are some things that are filled because they are not being
> specified
> in the meta data structure. This seems to be carried over as seen
> above.
> For example,
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcMet".
> DEBUG 4:
> DEBUG 4: Lambert Conformal Grid Data:
> DEBUG 4: hemisphere: N
>
>
>
>
>
>
>
>
> *DEBUG 4: scale_lat_1: -9999DEBUG 4: scale_lat_2: -9999DEBUG 4:
> lat_pin: -9999DEBUG 4: lon_pin: 9999DEBUG 4: x_pin:
> -9999DEBUG 4: y_pin: -9999DEBUG 4: lon_orient: 9999DEBUG
4:
> d_km: -9999DEBUG 4: r_km: -9999*
> DEBUG 4: nx: 191
> DEBUG 4: ny: 97
> DEBUG 4: so2_angle: 0
>
> I'm not sure if this is responsible for the issue. Do I need to
> define all
> grid attributes in the netcdf file?
>
> Here's the configure file. Everything looks okay there to me.
>
> On Wed, Feb 3, 2021 at 4:33 PM George McCabe via RT
> <met_help at ucar.edu>
> wrote:
>
> > Hi Edward,
> >
> > Could you send me the command you are trying to run as well as the
> > full log
> > file and the MET config file you are using? I'm not understanding
> > what the
> > issue is from this information. It looks like it should run fine
for
> > the
> > forecasts 0-23 and it should fail for the leads above that since
the
> > fields
> > are not present.
> >
> > On Wed, Feb 3, 2021 at 1:33 PM George McCabe <mccabe at ucar.edu>
wrote:
> >
> > > Hi Edward,
> > >
> > > It sounds like the log output combines the name and level when
> > > reporting
> > > that it can't find the data. Usually there is a forward slash in
> > > between
> > > them, like OZCON/A8, and I am not sure why it is not there in
your
> > > log
> > > output.
> > >
> > > I asked John to pull down the 36 hour file because that was the
> > > path I
> > saw
> > > in the log output right above the error. I didn't realize the
file
> > > you
> > were
> > > referring to was different. I will have John pull down a 20 hour
> > > file
> > from
> > > the day you mentioned so I can take a look and look for clues as
to
> > > why
> > it
> > > would fail when the field name/level exists.
> > >
> > > Thanks,
> > > George
> > >
> > > On Wed, Feb 3, 2021 at 1:13 PM Edward Strobach - NOAA Affiliate
via
> > > RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524 >
> > >>
> > >> I'm trying to understand why this failure happens even though
many
> > >> of
> > >> these
> > >> files have the PMTF/OZCON fields in them. You won't find OZCON
or
> > >> PMTF
> > at
> > >> forecast hour 36. For the earlier ones you will. My log file
was
> > >> for
> > >> 20190810, which has PMTF and OZCON for the metcro files. For
> > >> example,
> > >> aqm.t12z.metcro2d.20.nc will have these fields while the
> > >> aqm.t12z.metcro2d.36.nc. Of course I will check these files
for
> > >> any
> > >> issues
> > >> before emailing you. That's what I did originally when I
> > >> submitted the
> > >> ticket.
> > >>
> > >> On Wed, Feb 3, 2021 at 3:07 PM George McCabe via RT
> > >> <met_help at ucar.edu>
> > >> wrote:
> > >>
> > >> > Hi Edward,
> > >> >
> > >> > What do you mean when you ask if it will work? If running
> > >> > through the
> > >> > METplus wrappers, it should succeed for the files that have
the
> > >> > field
> > >> > available and fail for the rest. You will get a lot of
errors,
> > >> > but it
> > >> will
> > >> > still continue on and try to run the rest.
> > >> >
> > >> > On Wed, Feb 3, 2021 at 1:02 PM Edward Strobach - NOAA
Affiliate
> > >> > via
> > RT <
> > >> > met_help at ucar.edu> wrote:
> > >> >
> > >> > >
> > >> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524
> > >> > > >
> > >> > >
> > >> > > Okay, thanks. I'm aware of this but it's only because
certain
> > >> > > files
> > >> > don't
> > >> > > contain that field. The forecast runs out to 48 hours, but
> > >> > > the
> > >> surface
> > >> > > ozone was only saved for the first 24 hours. The metcro
files
> > >> > > -
> > >> forecast
> > >> > > hours 0 through 23 - should be able to be processed. Do
all
> > >> > > files
> > >> have
> > >> > to
> > >> > > be present within the time range encompassing 48 hours to
> > >> > > work?
> > >> > >
> > >> > > On Wed, Feb 3, 2021 at 2:58 PM George McCabe via RT <
> > >> met_help at ucar.edu>
> > >> > > wrote:
> > >> > >
> > >> > > > Hi Edward,
> > >> > > >
> > >> > > > John pulled down the file from your log error message for
me
> > >> > > > and
> > it
> > >> > looks
> > >> > > > like the fields you are requesting are not found in that
> > >> > > > file:
> > >> > > >
> > >> > > > /usr/local/netcdf/bin/ncdump aqm.t12z.metcro2d.36.nc |
grep
> > >> > > > float
> > >> > > > float LAT(lat, lon) ;
> > >> > > > float LON(lat, lon) ;
> > >> > > > float PBL(lat, lon) ;
> > >> > > > float WDIR10(lat, lon) ;
> > >> > > > float WSPD10(lat, lon) ;
> > >> > > > float TMP(lat, lon) ;
> > >> > > > float DPT(lat, lon) ;
> > >> > > >
> > >> > > > That would explain why it can't find that field. Also, it
> > >> > > > looks
> > like
> > >> > the
> > >> > > > fields in that file are 2D NetCDF, so I think you would
want
> > >> > > > to
> > >> specify
> > >> > > the
> > >> > > > level with the NetCDF notation, so instead of A8, you
would
> > >> > > > use
> > >> "(*,*)"
> > >> > > >
> > >> > > > On Wed, Feb 3, 2021 at 10:43 AM Edward Strobach - NOAA
> > >> > > > Affiliate
> > via
> > >> > RT <
> > >> > > > met_help at ucar.edu> wrote:
> > >> > > >
> > >> > > > >
> > >> > > > > Wed Feb 03 10:42:39 2021: Request 98524 was acted upon.
> > >> > > > > Transaction: Ticket created by edward.strobach at noaa.gov
> > >> > > > > Queue: met_help
> > >> > > > > Subject: field name and level strings are not
> > >> > > > > separated
> > when
> > >> > > running
> > >> > > > > pointstat for a new model
> > >> > > > > Owner: Nobody
> > >> > > > > Requestors: edward.strobach at noaa.gov
> > >> > > > > Status: new
> > >> > > > > Ticket <URL:
> > >> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524
> > >> > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > Good afternoon,
> > >> > > > >
> > >> > > > > We are trying to generate statistics of the AQ model
> > >> > > > > inline
> > system
> > >> > > which
> > >> > > > > has a different file structure than the offline system.
> > >> > > > > As
> > >> before, I
> > >> > > > > generate netcdf files with the fields I want to process
> > >> > > > > which
> > are
> > >> > taken
> > >> > > > > from files of the inline system. So far, nothing has
> > >> > > > > really
> > >> changed
> > >> > in
> > >> > > > my
> > >> > > > > processing steps, just ingesting data from a different
> > >> > > > > source
> > and
> > >> > > > grouping
> > >> > > > > verifiable fields. The netcdf file for a run that
works
> > >> > > > > looks
> > >> like
> > >> > > this:
> > >> > > > >
> > >> > > > > For prod (offiline system that works):
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > *>>> Dataset('aqm.t12z.metcro2d.05.nc
> > >> > > > > <http://aqm.t12z.metcro2d.05.nc>')<class
> > >> > > 'netCDF4._netCDF4.Dataset'>root
> > >> > > > > group (NETCDF4 data model, file format HDF5):
> > >> > > > > FileOrigins: NA
> > >> > > > > MET_version: V9.1 Projection: Lambert Conformal
> > hemisphere:
> > >> N
> > >> > > > > accum: 00 scale_lat_1: 33.000000 scale_lat_2:
> > >> > > > > 45.000000
> > >> > > lat_pin:
> > >> > > > > 21.821000 lon_pin: -120.628000 x_pin: 0.000000
> > >> > > > > y_pin:
> > >> > 0.000000
> > >> > > > > lon_orient: -97.000000 d_km: 12.000000 r_km:
> > >> > > > > 6371.200000
> > >> nx:
> > >> > > 442
> > >> > > > > ny: 265 grid_points dimensions(sizes): lon(442),
> > >> > > > > lat(265)
> > >> > > > > variables(dimensions): float32 LAT(lat,lon), float32
> > LON(lat,lon),
> > >> > > > float32
> > >> > > > > PMTF(lat,lon), float32 OZCON(lat,lon) groups: *
> > >> > > > >
> > >> > > > > *For r101_v1 (inline system that doesn't work):*
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > *Dataset('aqm.t12z.metcro2d.05.nc <
> > http://aqm.t12z.metcro2d.05.nc
> > >> > > > > ')<class
> > >> > > > > 'netCDF4._netCDF4.Dataset'>root group (NETCDF4 data
> > >> > > > > model, file
> > >> > format
> > >> > > > > HDF5): FileOrigins: NA MET_version: V9.1
> > >> > > > > hemisphere: N
> > >> > > > > Projection: Lambert Conformal nx: 191 ny: 97
> > >> > > > > grid_points
> > >> > > > > dimensions(sizes): lat(97), lon(191)
> > >> > > > > variables(dimensions):
> > >> > float32
> > >> > > > > LAT(lat,lon), float32 LON(lat,lon), float32
PBL(lat,lon),
> > float32
> > >> > > > > WDIR10(lat,lon), float32 WSPD10(lat,lon), float32
> > >> > > > > TMP(lat,lon),
> > >> > float32
> > >> > > > > DPT(lat,lon), float32 OZCON(lat,lon), float32
> > >> > > > > PMTF(lat,lon)
> > >> > groups:
> > >> > > *
> > >> > > > >
> > >> > > > > the meta data is shortened up for the inline system
> > >> > > > > because I
> > >> don't
> > >> > > have
> > >> > > > > all the information I need, but I don't think that's
> > >> > > > > important
> > >> here.
> > >> > > > When
> > >> > > > > printing out the attributes of one of the inline
fields, I
> > >> > > > > see
> > the
> > >> > > > expected
> > >> > > > > information:
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > *Dataset('aqm.t12z.metcro2d.05.nc
> > >> > > > >
<http://aqm.t12z.metcro2d.05.nc>').variables['PMTF']<class
> > >> > > > > 'netCDF4._netCDF4.Variable'>float32 PMTF(lat, lon)
> > >> > > > > long_name:
> > >> PMTF
> > >> > > > > standard_name: Surface PM2.5 units: ug/m**3
level:
> > >> > > > > L1
> > >> > > init_time:
> > >> > > > > 20190810_120000 init_time_ut: 1565438400
valid_time:
> > >> > > > 20190810_170000
> > >> > > > > valid_time_ut: 1565456400*
> > >> > > > >
> > >> > > > > What's happening is that the long_name and level are
being
> > joined
> > >> in
> > >> > > the
> > >> > > > > pointstat processing, which is strange to me. I don't
> > >> > > > > know why
> > >> that
> > >> > > > would
> > >> > > > > be. In the log file you can see that they are separate
up
> > >> > > > > to a
> > >> > point.
> > >> > > > > Here's print out confirming one such example:
> > >> > > > >
> > >> > > > >
> > >> > > > > *02/03 17:26:11.109 metplus.PointStat
> > >> > > > > (command_builder.py:168)
> > >> DEBUG:
> > >> > > > > COPYABLE ENVIRONMENT FOR NEXT COMMAND:02/03
17:26:11.109
> > >> > > > metplus.PointStat
> > >> > > > > (command_builder.py:256) DEBUG: export
CLIMO_MEAN_FILE="";
> > export
> > >> > > > > CLIMO_STDEV_FILE=""; export FCST_FIELD="{
name=\"PMTF\";
> > >> > level=\"L1\";
> > >> > > > },{
> > >> > > > > name=\"OZCON\"; level=\"A8\"; }"; export
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
MET_TMP_DIR="/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/tmp";
> > >> > > > > export MODEL="R101_V1"; export OBS_FIELD="{
> > >> > > > > name=\"COPOPM\";
> > >> > > > level=\"A1\";
> > >> > > > > convert(x) = x * 10^9; },{*
> > >> > > > >
> > >> > > > > Below is where the error occurs:
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > *DEBUG 2: Reading data for PMTFL1.WARNING:WARNING:
> > >> > > > > process_fcst_climo_files() -> no fields matching PMTFL1
> > >> > > > > found in
> > >> > file:
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190808/
> > >> > > > > aqm.t12z.metcro2d.36.nc
> > >> > > > > <http://aqm.t12z.metcro2d.36.nc>WARNING:DEBUG 2:DEBUG
2:
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
--------------------------------------------------------------------------------DEBUG
> > >> > > > > 2:DEBUG 2: Reading data for OZCONA8.WARNING:WARNING:
> > >> > > > > process_fcst_climo_files() -> no fields matching
OZCONA8
> > >> > > > > found
> > in
> > >> > file:
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190808/
> > >> > > > > aqm.t12z.metcro2d.36.nc
> > >> > > > > <http://aqm.t12z.metcro2d.36.nc>WARNING:ERROR :ERROR
:
> > >> > > > > process_fcst_climo_files() -> no requested forecast
data
> > >> > > > > found!
> > >> > > > > Exiting...ERROR :*
> > >> > > > >
> > >> > > > > You can see that the longname and level are joined, and
as
> > result,
> > >> > > fails
> > >> > > > > because that field doesn't exist. You can also clearly
> > >> > > > > see that
> > >> my
> > >> > > > > configure file separates these fields:
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > *FCST_VAR1_NAME = OZCONFCST_VAR1_LEVELS =
A1OBS_VAR1_NAME=
> > >> > > > > COPOOBS_VAR1_LEVELS= A1OBS_VAR1_OPTIONS = convert(x) =
x
> > >> > > > > *
> > >> > > > > 10^9;FCST_VAR2_NAME = OZCONFCST_VAR2_LEVELS =
> > >> > > > > A8OBS_VAR2_NAME=
> > >> > > > > COPOOBS_VAR2_LEVELS= A8*
> > >> > > > >
> > >> > > > > It's not clear to me why this issue is occurring if
> > >> > > > > everything
> > >> else
> > >> > > > appears
> > >> > > > > correct.
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > > > --
> > >> > > > George McCabe - Software Engineer III
> > >> > > > National Center for Atmospheric Research
> > >> > > > Research Applications Laboratory
> > >> > > > 303-497-2768
> > >> > > > ---
> > >> > > > My working day may not be your working day. Please do not
> > >> > > > feel
> > >> obliged
> > >> > to
> > >> > > > reply to this email outside of your normal working hours.
> > >> > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> >
> > >> > --
> > >> > George McCabe - Software Engineer III
> > >> > National Center for Atmospheric Research
> > >> > Research Applications Laboratory
> > >> > 303-497-2768
> > >> > ---
> > >> > My working day may not be your working day. Please do not
feel
> > >> > obliged
> > >> to
> > >> > reply to this email outside of your normal working hours.
> > >> >
> > >> >
> > >>
> > >>
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
> > > obliged to
> > > reply to this email outside of your normal working hours.
> > >
> >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
> > obliged to
> > reply to this email outside of your normal working hours.
> >
> >
------------------------------------------------
Subject: field name and level strings are not separated when running pointstat for a new model
From: Edward Strobach - NOAA Affiliate
Time: Sat Feb 06 07:05:30 2021
I think I'm good. Thanks.
On Fri, Feb 5, 2021 at 3:45 PM George McCabe via RT
<met_help at ucar.edu>
wrote:
> Hi Edward,
>
> Sorry for the delayed response.
>
> The log output that appears to combine the name and level only
combines
> them in the log output. There should be a slash in between the name
and
> level, i.e. OZCON/A8, but the actual data that is read is correct.
>
> I looked at your log file and it appears to be running as expected.
It
> fails as expected only for forecast leads about 23 where the fields
don't
> exist, and succeeds otherwise. Reducing the list of forecast leads
would
> prevent processing the files that don't have the required data. I
see you
> are using:
>
> INIT_SEQ = 12
> LEAD_SEQ_MAX = 48
>
> This will process leads that match up with files that are
initialized at
> 12Z up to a 48 hour lead. Changing the max value to 24 (or 23?)
would
> prevent the additional runs that are failing.
>
> Let me know if I am still not understanding what you are asking for
this
> issue.
>
> Thanks,
> George
>
>
> On Wed Feb 03 15:04:53 2021, edward.strobach at noaa.gov wrote:
> > Hi,
> >
> > There is no specific command since I'm running this all under a
bash
> > script
> > that has a series different processing steps. I can say that I
> > construct
> > several configure files that are then passed into
master_metplus.py.
> > This
> > has worked for me for some time, so I don't think commands are the
> > issue
> > here. The log file is way to big too attach but can be viewed
here:
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210203172449.
> >
> >
> > You can start on line 84297, for example; that is for the
> > aqm.t12z.metcro2d.06.nc file. Shortly below that is an instance
where
> > the
> > level and name is joined:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *DEBUG 1: Forecast File:
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190810/
> aqm.t12z.metcro2d.06.nc
> > <http://aqm.t12z.metcro2d.06.nc>DEBUG 1: Observation File:
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r101_v1/aq/
> prepbufr.aqm.20190810.nc
> > <http://prepbufr.aqm.20190810.nc>DEBUG 7: check_nc_data_2d()
count:
> > valid=18527, zero=0, missing=0, 18527 == 18527DEBUG 7:
> > MetNcFile::data()
> > const -> took 0 seconds to read NetCDF dataDEBUG 7:
MetNcFile::data()
> > const
> > -> took 0 seconds to fill data planeDEBUG 2:DEBUG 2:
> >
>
--------------------------------------------------------------------------------DEBUG
> > 2:DEBUG 2: Reading data for OZCONA1.DEBUG 3: Grid Definition:
> > Projection:
> > Lambert Conformal Nx: 191 Ny: 97 Lat_LL: -89.986 Lon_LL: 142.318
> > Lon_orient: -81.000 Alpha: 1.950 Cone: 0.988 Bx: -9999.0000 By:
> > -9998.8416DEBUG 2: Processing masking regions.DEBUG 3: Processing
grid
> > mask: FULL*
> >
> > There are some things that are filled because they are not being
> > specified
> > in the meta data structure. This seems to be carried over as seen
> > above.
> > For example,
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_NcMet".
> > DEBUG 4:
> > DEBUG 4: Lambert Conformal Grid Data:
> > DEBUG 4: hemisphere: N
> >
> >
> >
> >
> >
> >
> >
> >
> > *DEBUG 4: scale_lat_1: -9999DEBUG 4: scale_lat_2: -9999DEBUG
4:
> > lat_pin: -9999DEBUG 4: lon_pin: 9999DEBUG 4: x_pin:
> > -9999DEBUG 4: y_pin: -9999DEBUG 4: lon_orient:
9999DEBUG 4:
> > d_km: -9999DEBUG 4: r_km: -9999*
> > DEBUG 4: nx: 191
> > DEBUG 4: ny: 97
> > DEBUG 4: so2_angle: 0
> >
> > I'm not sure if this is responsible for the issue. Do I need to
> > define all
> > grid attributes in the netcdf file?
> >
> > Here's the configure file. Everything looks okay there to me.
> >
> > On Wed, Feb 3, 2021 at 4:33 PM George McCabe via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > Hi Edward,
> > >
> > > Could you send me the command you are trying to run as well as
the
> > > full log
> > > file and the MET config file you are using? I'm not
understanding
> > > what the
> > > issue is from this information. It looks like it should run fine
for
> > > the
> > > forecasts 0-23 and it should fail for the leads above that since
the
> > > fields
> > > are not present.
> > >
> > > On Wed, Feb 3, 2021 at 1:33 PM George McCabe <mccabe at ucar.edu>
wrote:
> > >
> > > > Hi Edward,
> > > >
> > > > It sounds like the log output combines the name and level when
> > > > reporting
> > > > that it can't find the data. Usually there is a forward slash
in
> > > > between
> > > > them, like OZCON/A8, and I am not sure why it is not there in
your
> > > > log
> > > > output.
> > > >
> > > > I asked John to pull down the 36 hour file because that was
the
> > > > path I
> > > saw
> > > > in the log output right above the error. I didn't realize the
file
> > > > you
> > > were
> > > > referring to was different. I will have John pull down a 20
hour
> > > > file
> > > from
> > > > the day you mentioned so I can take a look and look for clues
as to
> > > > why
> > > it
> > > > would fail when the field name/level exists.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Wed, Feb 3, 2021 at 1:13 PM Edward Strobach - NOAA
Affiliate via
> > > > RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >>
> > > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524
>
> > > >>
> > > >> I'm trying to understand why this failure happens even though
many
> > > >> of
> > > >> these
> > > >> files have the PMTF/OZCON fields in them. You won't find
OZCON or
> > > >> PMTF
> > > at
> > > >> forecast hour 36. For the earlier ones you will. My log
file was
> > > >> for
> > > >> 20190810, which has PMTF and OZCON for the metcro files. For
> > > >> example,
> > > >> aqm.t12z.metcro2d.20.nc will have these fields while the
> > > >> aqm.t12z.metcro2d.36.nc. Of course I will check these files
for
> > > >> any
> > > >> issues
> > > >> before emailing you. That's what I did originally when I
> > > >> submitted the
> > > >> ticket.
> > > >>
> > > >> On Wed, Feb 3, 2021 at 3:07 PM George McCabe via RT
> > > >> <met_help at ucar.edu>
> > > >> wrote:
> > > >>
> > > >> > Hi Edward,
> > > >> >
> > > >> > What do you mean when you ask if it will work? If running
> > > >> > through the
> > > >> > METplus wrappers, it should succeed for the files that have
the
> > > >> > field
> > > >> > available and fail for the rest. You will get a lot of
errors,
> > > >> > but it
> > > >> will
> > > >> > still continue on and try to run the rest.
> > > >> >
> > > >> > On Wed, Feb 3, 2021 at 1:02 PM Edward Strobach - NOAA
Affiliate
> > > >> > via
> > > RT <
> > > >> > met_help at ucar.edu> wrote:
> > > >> >
> > > >> > >
> > > >> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524
> > > >> > > >
> > > >> > >
> > > >> > > Okay, thanks. I'm aware of this but it's only because
certain
> > > >> > > files
> > > >> > don't
> > > >> > > contain that field. The forecast runs out to 48 hours,
but
> > > >> > > the
> > > >> surface
> > > >> > > ozone was only saved for the first 24 hours. The metcro
files
> > > >> > > -
> > > >> forecast
> > > >> > > hours 0 through 23 - should be able to be processed. Do
all
> > > >> > > files
> > > >> have
> > > >> > to
> > > >> > > be present within the time range encompassing 48 hours to
> > > >> > > work?
> > > >> > >
> > > >> > > On Wed, Feb 3, 2021 at 2:58 PM George McCabe via RT <
> > > >> met_help at ucar.edu>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Hi Edward,
> > > >> > > >
> > > >> > > > John pulled down the file from your log error message
for me
> > > >> > > > and
> > > it
> > > >> > looks
> > > >> > > > like the fields you are requesting are not found in
that
> > > >> > > > file:
> > > >> > > >
> > > >> > > > /usr/local/netcdf/bin/ncdump aqm.t12z.metcro2d.36.nc |
grep
> > > >> > > > float
> > > >> > > > float LAT(lat, lon) ;
> > > >> > > > float LON(lat, lon) ;
> > > >> > > > float PBL(lat, lon) ;
> > > >> > > > float WDIR10(lat, lon) ;
> > > >> > > > float WSPD10(lat, lon) ;
> > > >> > > > float TMP(lat, lon) ;
> > > >> > > > float DPT(lat, lon) ;
> > > >> > > >
> > > >> > > > That would explain why it can't find that field. Also,
it
> > > >> > > > looks
> > > like
> > > >> > the
> > > >> > > > fields in that file are 2D NetCDF, so I think you would
want
> > > >> > > > to
> > > >> specify
> > > >> > > the
> > > >> > > > level with the NetCDF notation, so instead of A8, you
would
> > > >> > > > use
> > > >> "(*,*)"
> > > >> > > >
> > > >> > > > On Wed, Feb 3, 2021 at 10:43 AM Edward Strobach - NOAA
> > > >> > > > Affiliate
> > > via
> > > >> > RT <
> > > >> > > > met_help at ucar.edu> wrote:
> > > >> > > >
> > > >> > > > >
> > > >> > > > > Wed Feb 03 10:42:39 2021: Request 98524 was acted
upon.
> > > >> > > > > Transaction: Ticket created by
edward.strobach at noaa.gov
> > > >> > > > > Queue: met_help
> > > >> > > > > Subject: field name and level strings are not
> > > >> > > > > separated
> > > when
> > > >> > > running
> > > >> > > > > pointstat for a new model
> > > >> > > > > Owner: Nobody
> > > >> > > > > Requestors: edward.strobach at noaa.gov
> > > >> > > > > Status: new
> > > >> > > > > Ticket <URL:
> > > >> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524
> > > >> > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > Good afternoon,
> > > >> > > > >
> > > >> > > > > We are trying to generate statistics of the AQ model
> > > >> > > > > inline
> > > system
> > > >> > > which
> > > >> > > > > has a different file structure than the offline
system.
> > > >> > > > > As
> > > >> before, I
> > > >> > > > > generate netcdf files with the fields I want to
process
> > > >> > > > > which
> > > are
> > > >> > taken
> > > >> > > > > from files of the inline system. So far, nothing has
> > > >> > > > > really
> > > >> changed
> > > >> > in
> > > >> > > > my
> > > >> > > > > processing steps, just ingesting data from a
different
> > > >> > > > > source
> > > and
> > > >> > > > grouping
> > > >> > > > > verifiable fields. The netcdf file for a run that
works
> > > >> > > > > looks
> > > >> like
> > > >> > > this:
> > > >> > > > >
> > > >> > > > > For prod (offiline system that works):
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > *>>> Dataset('aqm.t12z.metcro2d.05.nc
> > > >> > > > > <http://aqm.t12z.metcro2d.05.nc>')<class
> > > >> > > 'netCDF4._netCDF4.Dataset'>root
> > > >> > > > > group (NETCDF4 data model, file format HDF5):
> > > >> > > > > FileOrigins: NA
> > > >> > > > > MET_version: V9.1 Projection: Lambert Conformal
> > > hemisphere:
> > > >> N
> > > >> > > > > accum: 00 scale_lat_1: 33.000000 scale_lat_2:
> > > >> > > > > 45.000000
> > > >> > > lat_pin:
> > > >> > > > > 21.821000 lon_pin: -120.628000 x_pin: 0.000000
> > > >> > > > > y_pin:
> > > >> > 0.000000
> > > >> > > > > lon_orient: -97.000000 d_km: 12.000000 r_km:
> > > >> > > > > 6371.200000
> > > >> nx:
> > > >> > > 442
> > > >> > > > > ny: 265 grid_points dimensions(sizes): lon(442),
> > > >> > > > > lat(265)
> > > >> > > > > variables(dimensions): float32 LAT(lat,lon), float32
> > > LON(lat,lon),
> > > >> > > > float32
> > > >> > > > > PMTF(lat,lon), float32 OZCON(lat,lon) groups: *
> > > >> > > > >
> > > >> > > > > *For r101_v1 (inline system that doesn't work):*
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > *Dataset('aqm.t12z.metcro2d.05.nc <
> > > http://aqm.t12z.metcro2d.05.nc
> > > >> > > > > ')<class
> > > >> > > > > 'netCDF4._netCDF4.Dataset'>root group (NETCDF4 data
> > > >> > > > > model, file
> > > >> > format
> > > >> > > > > HDF5): FileOrigins: NA MET_version: V9.1
> > > >> > > > > hemisphere: N
> > > >> > > > > Projection: Lambert Conformal nx: 191 ny: 97
> > > >> > > > > grid_points
> > > >> > > > > dimensions(sizes): lat(97), lon(191)
> > > >> > > > > variables(dimensions):
> > > >> > float32
> > > >> > > > > LAT(lat,lon), float32 LON(lat,lon), float32
PBL(lat,lon),
> > > float32
> > > >> > > > > WDIR10(lat,lon), float32 WSPD10(lat,lon), float32
> > > >> > > > > TMP(lat,lon),
> > > >> > float32
> > > >> > > > > DPT(lat,lon), float32 OZCON(lat,lon), float32
> > > >> > > > > PMTF(lat,lon)
> > > >> > groups:
> > > >> > > *
> > > >> > > > >
> > > >> > > > > the meta data is shortened up for the inline system
> > > >> > > > > because I
> > > >> don't
> > > >> > > have
> > > >> > > > > all the information I need, but I don't think that's
> > > >> > > > > important
> > > >> here.
> > > >> > > > When
> > > >> > > > > printing out the attributes of one of the inline
fields, I
> > > >> > > > > see
> > > the
> > > >> > > > expected
> > > >> > > > > information:
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > *Dataset('aqm.t12z.metcro2d.05.nc
> > > >> > > > >
<http://aqm.t12z.metcro2d.05.nc>').variables['PMTF']<class
> > > >> > > > > 'netCDF4._netCDF4.Variable'>float32 PMTF(lat, lon)
> > > >> > > > > long_name:
> > > >> PMTF
> > > >> > > > > standard_name: Surface PM2.5 units: ug/m**3
level:
> > > >> > > > > L1
> > > >> > > init_time:
> > > >> > > > > 20190810_120000 init_time_ut: 1565438400
valid_time:
> > > >> > > > 20190810_170000
> > > >> > > > > valid_time_ut: 1565456400*
> > > >> > > > >
> > > >> > > > > What's happening is that the long_name and level are
being
> > > joined
> > > >> in
> > > >> > > the
> > > >> > > > > pointstat processing, which is strange to me. I
don't
> > > >> > > > > know why
> > > >> that
> > > >> > > > would
> > > >> > > > > be. In the log file you can see that they are
separate up
> > > >> > > > > to a
> > > >> > point.
> > > >> > > > > Here's print out confirming one such example:
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > *02/03 17:26:11.109 metplus.PointStat
> > > >> > > > > (command_builder.py:168)
> > > >> DEBUG:
> > > >> > > > > COPYABLE ENVIRONMENT FOR NEXT COMMAND:02/03
17:26:11.109
> > > >> > > > metplus.PointStat
> > > >> > > > > (command_builder.py:256) DEBUG: export
CLIMO_MEAN_FILE="";
> > > export
> > > >> > > > > CLIMO_STDEV_FILE=""; export FCST_FIELD="{
name=\"PMTF\";
> > > >> > level=\"L1\";
> > > >> > > > },{
> > > >> > > > > name=\"OZCON\"; level=\"A8\"; }"; export
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
>
MET_TMP_DIR="/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/tmp";
> > > >> > > > > export MODEL="R101_V1"; export OBS_FIELD="{
> > > >> > > > > name=\"COPOPM\";
> > > >> > > > level=\"A1\";
> > > >> > > > > convert(x) = x * 10^9; },{*
> > > >> > > > >
> > > >> > > > > Below is where the error occurs:
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > *DEBUG 2: Reading data for PMTFL1.WARNING:WARNING:
> > > >> > > > > process_fcst_climo_files() -> no fields matching
PMTFL1
> > > >> > > > > found in
> > > >> > file:
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190808/
> > > >> > > > > aqm.t12z.metcro2d.36.nc
> > > >> > > > > <http://aqm.t12z.metcro2d.36.nc>WARNING:DEBUG 2:DEBUG
2:
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
>
--------------------------------------------------------------------------------DEBUG
> > > >> > > > > 2:DEBUG 2: Reading data for OZCONA8.WARNING:WARNING:
> > > >> > > > > process_fcst_climo_files() -> no fields matching
OZCONA8
> > > >> > > > > found
> > > in
> > > >> > file:
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190808/
> > > >> > > > > aqm.t12z.metcro2d.36.nc
> > > >> > > > > <http://aqm.t12z.metcro2d.36.nc>WARNING:ERROR :ERROR
:
> > > >> > > > > process_fcst_climo_files() -> no requested forecast
data
> > > >> > > > > found!
> > > >> > > > > Exiting...ERROR :*
> > > >> > > > >
> > > >> > > > > You can see that the longname and level are joined,
and as
> > > result,
> > > >> > > fails
> > > >> > > > > because that field doesn't exist. You can also
clearly
> > > >> > > > > see that
> > > >> my
> > > >> > > > > configure file separates these fields:
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > *FCST_VAR1_NAME = OZCONFCST_VAR1_LEVELS =
A1OBS_VAR1_NAME=
> > > >> > > > > COPOOBS_VAR1_LEVELS= A1OBS_VAR1_OPTIONS = convert(x)
= x
> > > >> > > > > *
> > > >> > > > > 10^9;FCST_VAR2_NAME = OZCONFCST_VAR2_LEVELS =
> > > >> > > > > A8OBS_VAR2_NAME=
> > > >> > > > > COPOOBS_VAR2_LEVELS= A8*
> > > >> > > > >
> > > >> > > > > It's not clear to me why this issue is occurring if
> > > >> > > > > everything
> > > >> else
> > > >> > > > appears
> > > >> > > > > correct.
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > > > --
> > > >> > > > George McCabe - Software Engineer III
> > > >> > > > National Center for Atmospheric Research
> > > >> > > > Research Applications Laboratory
> > > >> > > > 303-497-2768
> > > >> > > > ---
> > > >> > > > My working day may not be your working day. Please do
not
> > > >> > > > feel
> > > >> obliged
> > > >> > to
> > > >> > > > reply to this email outside of your normal working
hours.
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> > --
> > > >> > George McCabe - Software Engineer III
> > > >> > National Center for Atmospheric Research
> > > >> > Research Applications Laboratory
> > > >> > 303-497-2768
> > > >> > ---
> > > >> > My working day may not be your working day. Please do not
feel
> > > >> > obliged
> > > >> to
> > > >> > reply to this email outside of your normal working hours.
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> > > > obliged to
> > > > reply to this email outside of your normal working hours.
> > > >
> > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
> > > obliged to
> > > reply to this email outside of your normal working hours.
> > >
> > >
>
>
>
>
------------------------------------------------
Subject: field name and level strings are not separated when running pointstat for a new model
From: George McCabe
Time: Mon Feb 08 08:40:26 2021
Hi Edward,
Good to hear. FYI, we made a fix to the log output so that there will
be a
slash in between name and level. It will be available in the next MET
release. Here is the pull request with the changes:
https://github.com/dtcenter/MET/pull/1656
On Sat, Feb 6, 2021 at 7:06 AM Edward Strobach - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524 >
>
> I think I'm good. Thanks.
>
> On Fri, Feb 5, 2021 at 3:45 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Edward,
> >
> > Sorry for the delayed response.
> >
> > The log output that appears to combine the name and level only
combines
> > them in the log output. There should be a slash in between the
name and
> > level, i.e. OZCON/A8, but the actual data that is read is correct.
> >
> > I looked at your log file and it appears to be running as
expected. It
> > fails as expected only for forecast leads about 23 where the
fields don't
> > exist, and succeeds otherwise. Reducing the list of forecast leads
would
> > prevent processing the files that don't have the required data. I
see you
> > are using:
> >
> > INIT_SEQ = 12
> > LEAD_SEQ_MAX = 48
> >
> > This will process leads that match up with files that are
initialized at
> > 12Z up to a 48 hour lead. Changing the max value to 24 (or 23?)
would
> > prevent the additional runs that are failing.
> >
> > Let me know if I am still not understanding what you are asking
for this
> > issue.
> >
> > Thanks,
> > George
> >
> >
> > On Wed Feb 03 15:04:53 2021, edward.strobach at noaa.gov wrote:
> > > Hi,
> > >
> > > There is no specific command since I'm running this all under a
bash
> > > script
> > > that has a series different processing steps. I can say that I
> > > construct
> > > several configure files that are then passed into
master_metplus.py.
> > > This
> > > has worked for me for some time, so I don't think commands are
the
> > > issue
> > > here. The log file is way to big too attach but can be viewed
here:
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/logs/master_metplus.log.20210203172449.
> > >
> > >
> > > You can start on line 84297, for example; that is for the
> > > aqm.t12z.metcro2d.06.nc file. Shortly below that is an instance
where
> > > the
> > > level and name is joined:
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *DEBUG 1: Forecast File:
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190810/
> > aqm.t12z.metcro2d.06.nc
> > > <http://aqm.t12z.metcro2d.06.nc>DEBUG 1: Observation File:
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/aqm/conus_sfc/r101_v1/aq/
> > prepbufr.aqm.20190810.nc
> > > <http://prepbufr.aqm.20190810.nc>DEBUG 7: check_nc_data_2d()
count:
> > > valid=18527, zero=0, missing=0, 18527 == 18527DEBUG 7:
> > > MetNcFile::data()
> > > const -> took 0 seconds to read NetCDF dataDEBUG 7:
MetNcFile::data()
> > > const
> > > -> took 0 seconds to fill data planeDEBUG 2:DEBUG 2:
> > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > 2:DEBUG 2: Reading data for OZCONA1.DEBUG 3: Grid Definition:
> > > Projection:
> > > Lambert Conformal Nx: 191 Ny: 97 Lat_LL: -89.986 Lon_LL: 142.318
> > > Lon_orient: -81.000 Alpha: 1.950 Cone: 0.988 Bx: -9999.0000 By:
> > > -9998.8416DEBUG 2: Processing masking regions.DEBUG 3:
Processing grid
> > > mask: FULL*
> > >
> > > There are some things that are filled because they are not being
> > > specified
> > > in the meta data structure. This seems to be carried over as
seen
> > > above.
> > > For example,
> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > > Met2dDataFile object of type "FileType_NcMet".
> > > DEBUG 4:
> > > DEBUG 4: Lambert Conformal Grid Data:
> > > DEBUG 4: hemisphere: N
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > *DEBUG 4: scale_lat_1: -9999DEBUG 4: scale_lat_2: -9999DEBUG
4:
> > > lat_pin: -9999DEBUG 4: lon_pin: 9999DEBUG 4:
x_pin:
> > > -9999DEBUG 4: y_pin: -9999DEBUG 4: lon_orient:
9999DEBUG 4:
> > > d_km: -9999DEBUG 4: r_km: -9999*
> > > DEBUG 4: nx: 191
> > > DEBUG 4: ny: 97
> > > DEBUG 4: so2_angle: 0
> > >
> > > I'm not sure if this is responsible for the issue. Do I need to
> > > define all
> > > grid attributes in the netcdf file?
> > >
> > > Here's the configure file. Everything looks okay there to me.
> > >
> > > On Wed, Feb 3, 2021 at 4:33 PM George McCabe via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Edward,
> > > >
> > > > Could you send me the command you are trying to run as well as
the
> > > > full log
> > > > file and the MET config file you are using? I'm not
understanding
> > > > what the
> > > > issue is from this information. It looks like it should run
fine for
> > > > the
> > > > forecasts 0-23 and it should fail for the leads above that
since the
> > > > fields
> > > > are not present.
> > > >
> > > > On Wed, Feb 3, 2021 at 1:33 PM George McCabe <mccabe at ucar.edu>
> wrote:
> > > >
> > > > > Hi Edward,
> > > > >
> > > > > It sounds like the log output combines the name and level
when
> > > > > reporting
> > > > > that it can't find the data. Usually there is a forward
slash in
> > > > > between
> > > > > them, like OZCON/A8, and I am not sure why it is not there
in your
> > > > > log
> > > > > output.
> > > > >
> > > > > I asked John to pull down the 36 hour file because that was
the
> > > > > path I
> > > > saw
> > > > > in the log output right above the error. I didn't realize
the file
> > > > > you
> > > > were
> > > > > referring to was different. I will have John pull down a 20
hour
> > > > > file
> > > > from
> > > > > the day you mentioned so I can take a look and look for
clues as to
> > > > > why
> > > > it
> > > > > would fail when the field name/level exists.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > > On Wed, Feb 3, 2021 at 1:13 PM Edward Strobach - NOAA
Affiliate via
> > > > > RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > >>
> > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524 >
> > > > >>
> > > > >> I'm trying to understand why this failure happens even
though many
> > > > >> of
> > > > >> these
> > > > >> files have the PMTF/OZCON fields in them. You won't find
OZCON or
> > > > >> PMTF
> > > > at
> > > > >> forecast hour 36. For the earlier ones you will. My log
file was
> > > > >> for
> > > > >> 20190810, which has PMTF and OZCON for the metcro files.
For
> > > > >> example,
> > > > >> aqm.t12z.metcro2d.20.nc will have these fields while the
> > > > >> aqm.t12z.metcro2d.36.nc. Of course I will check these
files for
> > > > >> any
> > > > >> issues
> > > > >> before emailing you. That's what I did originally when I
> > > > >> submitted the
> > > > >> ticket.
> > > > >>
> > > > >> On Wed, Feb 3, 2021 at 3:07 PM George McCabe via RT
> > > > >> <met_help at ucar.edu>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Edward,
> > > > >> >
> > > > >> > What do you mean when you ask if it will work? If running
> > > > >> > through the
> > > > >> > METplus wrappers, it should succeed for the files that
have the
> > > > >> > field
> > > > >> > available and fail for the rest. You will get a lot of
errors,
> > > > >> > but it
> > > > >> will
> > > > >> > still continue on and try to run the rest.
> > > > >> >
> > > > >> > On Wed, Feb 3, 2021 at 1:02 PM Edward Strobach - NOAA
Affiliate
> > > > >> > via
> > > > RT <
> > > > >> > met_help at ucar.edu> wrote:
> > > > >> >
> > > > >> > >
> > > > >> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524
> > > > >> > > >
> > > > >> > >
> > > > >> > > Okay, thanks. I'm aware of this but it's only because
certain
> > > > >> > > files
> > > > >> > don't
> > > > >> > > contain that field. The forecast runs out to 48 hours,
but
> > > > >> > > the
> > > > >> surface
> > > > >> > > ozone was only saved for the first 24 hours. The
metcro files
> > > > >> > > -
> > > > >> forecast
> > > > >> > > hours 0 through 23 - should be able to be processed.
Do all
> > > > >> > > files
> > > > >> have
> > > > >> > to
> > > > >> > > be present within the time range encompassing 48 hours
to
> > > > >> > > work?
> > > > >> > >
> > > > >> > > On Wed, Feb 3, 2021 at 2:58 PM George McCabe via RT <
> > > > >> met_help at ucar.edu>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > Hi Edward,
> > > > >> > > >
> > > > >> > > > John pulled down the file from your log error message
for me
> > > > >> > > > and
> > > > it
> > > > >> > looks
> > > > >> > > > like the fields you are requesting are not found in
that
> > > > >> > > > file:
> > > > >> > > >
> > > > >> > > > /usr/local/netcdf/bin/ncdump aqm.t12z.metcro2d.36.nc
| grep
> > > > >> > > > float
> > > > >> > > > float LAT(lat, lon) ;
> > > > >> > > > float LON(lat, lon) ;
> > > > >> > > > float PBL(lat, lon) ;
> > > > >> > > > float WDIR10(lat, lon) ;
> > > > >> > > > float WSPD10(lat, lon) ;
> > > > >> > > > float TMP(lat, lon) ;
> > > > >> > > > float DPT(lat, lon) ;
> > > > >> > > >
> > > > >> > > > That would explain why it can't find that field.
Also, it
> > > > >> > > > looks
> > > > like
> > > > >> > the
> > > > >> > > > fields in that file are 2D NetCDF, so I think you
would want
> > > > >> > > > to
> > > > >> specify
> > > > >> > > the
> > > > >> > > > level with the NetCDF notation, so instead of A8, you
would
> > > > >> > > > use
> > > > >> "(*,*)"
> > > > >> > > >
> > > > >> > > > On Wed, Feb 3, 2021 at 10:43 AM Edward Strobach -
NOAA
> > > > >> > > > Affiliate
> > > > via
> > > > >> > RT <
> > > > >> > > > met_help at ucar.edu> wrote:
> > > > >> > > >
> > > > >> > > > >
> > > > >> > > > > Wed Feb 03 10:42:39 2021: Request 98524 was acted
upon.
> > > > >> > > > > Transaction: Ticket created by
edward.strobach at noaa.gov
> > > > >> > > > > Queue: met_help
> > > > >> > > > > Subject: field name and level strings are not
> > > > >> > > > > separated
> > > > when
> > > > >> > > running
> > > > >> > > > > pointstat for a new model
> > > > >> > > > > Owner: Nobody
> > > > >> > > > > Requestors: edward.strobach at noaa.gov
> > > > >> > > > > Status: new
> > > > >> > > > > Ticket <URL:
> > > > >> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98524
> > > > >> > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > Good afternoon,
> > > > >> > > > >
> > > > >> > > > > We are trying to generate statistics of the AQ
model
> > > > >> > > > > inline
> > > > system
> > > > >> > > which
> > > > >> > > > > has a different file structure than the offline
system.
> > > > >> > > > > As
> > > > >> before, I
> > > > >> > > > > generate netcdf files with the fields I want to
process
> > > > >> > > > > which
> > > > are
> > > > >> > taken
> > > > >> > > > > from files of the inline system. So far, nothing
has
> > > > >> > > > > really
> > > > >> changed
> > > > >> > in
> > > > >> > > > my
> > > > >> > > > > processing steps, just ingesting data from a
different
> > > > >> > > > > source
> > > > and
> > > > >> > > > grouping
> > > > >> > > > > verifiable fields. The netcdf file for a run that
works
> > > > >> > > > > looks
> > > > >> like
> > > > >> > > this:
> > > > >> > > > >
> > > > >> > > > > For prod (offiline system that works):
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > *>>> Dataset('aqm.t12z.metcro2d.05.nc
> > > > >> > > > > <http://aqm.t12z.metcro2d.05.nc>')<class
> > > > >> > > 'netCDF4._netCDF4.Dataset'>root
> > > > >> > > > > group (NETCDF4 data model, file format HDF5):
> > > > >> > > > > FileOrigins: NA
> > > > >> > > > > MET_version: V9.1 Projection: Lambert Conformal
> > > > hemisphere:
> > > > >> N
> > > > >> > > > > accum: 00 scale_lat_1: 33.000000 scale_lat_2:
> > > > >> > > > > 45.000000
> > > > >> > > lat_pin:
> > > > >> > > > > 21.821000 lon_pin: -120.628000 x_pin:
0.000000
> > > > >> > > > > y_pin:
> > > > >> > 0.000000
> > > > >> > > > > lon_orient: -97.000000 d_km: 12.000000 r_km:
> > > > >> > > > > 6371.200000
> > > > >> nx:
> > > > >> > > 442
> > > > >> > > > > ny: 265 grid_points dimensions(sizes): lon(442),
> > > > >> > > > > lat(265)
> > > > >> > > > > variables(dimensions): float32 LAT(lat,lon),
float32
> > > > LON(lat,lon),
> > > > >> > > > float32
> > > > >> > > > > PMTF(lat,lon), float32 OZCON(lat,lon) groups: *
> > > > >> > > > >
> > > > >> > > > > *For r101_v1 (inline system that doesn't work):*
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > *Dataset('aqm.t12z.metcro2d.05.nc <
> > > > http://aqm.t12z.metcro2d.05.nc
> > > > >> > > > > ')<class
> > > > >> > > > > 'netCDF4._netCDF4.Dataset'>root group (NETCDF4
data
> > > > >> > > > > model, file
> > > > >> > format
> > > > >> > > > > HDF5): FileOrigins: NA MET_version: V9.1
> > > > >> > > > > hemisphere: N
> > > > >> > > > > Projection: Lambert Conformal nx: 191 ny: 97
> > > > >> > > > > grid_points
> > > > >> > > > > dimensions(sizes): lat(97), lon(191)
> > > > >> > > > > variables(dimensions):
> > > > >> > float32
> > > > >> > > > > LAT(lat,lon), float32 LON(lat,lon), float32
PBL(lat,lon),
> > > > float32
> > > > >> > > > > WDIR10(lat,lon), float32 WSPD10(lat,lon), float32
> > > > >> > > > > TMP(lat,lon),
> > > > >> > float32
> > > > >> > > > > DPT(lat,lon), float32 OZCON(lat,lon), float32
> > > > >> > > > > PMTF(lat,lon)
> > > > >> > groups:
> > > > >> > > *
> > > > >> > > > >
> > > > >> > > > > the meta data is shortened up for the inline system
> > > > >> > > > > because I
> > > > >> don't
> > > > >> > > have
> > > > >> > > > > all the information I need, but I don't think
that's
> > > > >> > > > > important
> > > > >> here.
> > > > >> > > > When
> > > > >> > > > > printing out the attributes of one of the inline
fields, I
> > > > >> > > > > see
> > > > the
> > > > >> > > > expected
> > > > >> > > > > information:
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > *Dataset('aqm.t12z.metcro2d.05.nc
> > > > >> > > > > <http://aqm.t12z.metcro2d.05.nc
> >').variables['PMTF']<class
> > > > >> > > > > 'netCDF4._netCDF4.Variable'>float32 PMTF(lat, lon)
> > > > >> > > > > long_name:
> > > > >> PMTF
> > > > >> > > > > standard_name: Surface PM2.5 units: ug/m**3
level:
> > > > >> > > > > L1
> > > > >> > > init_time:
> > > > >> > > > > 20190810_120000 init_time_ut: 1565438400
valid_time:
> > > > >> > > > 20190810_170000
> > > > >> > > > > valid_time_ut: 1565456400*
> > > > >> > > > >
> > > > >> > > > > What's happening is that the long_name and level
are being
> > > > joined
> > > > >> in
> > > > >> > > the
> > > > >> > > > > pointstat processing, which is strange to me. I
don't
> > > > >> > > > > know why
> > > > >> that
> > > > >> > > > would
> > > > >> > > > > be. In the log file you can see that they are
separate up
> > > > >> > > > > to a
> > > > >> > point.
> > > > >> > > > > Here's print out confirming one such example:
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > *02/03 17:26:11.109 metplus.PointStat
> > > > >> > > > > (command_builder.py:168)
> > > > >> DEBUG:
> > > > >> > > > > COPYABLE ENVIRONMENT FOR NEXT COMMAND:02/03
17:26:11.109
> > > > >> > > > metplus.PointStat
> > > > >> > > > > (command_builder.py:256) DEBUG: export
CLIMO_MEAN_FILE="";
> > > > export
> > > > >> > > > > CLIMO_STDEV_FILE=""; export FCST_FIELD="{
name=\"PMTF\";
> > > > >> > level=\"L1\";
> > > > >> > > > },{
> > > > >> > > > > name=\"OZCON\"; level=\"A8\"; }"; export
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> >
>
MET_TMP_DIR="/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/CMAQ/tmp";
> > > > >> > > > > export MODEL="R101_V1"; export OBS_FIELD="{
> > > > >> > > > > name=\"COPOPM\";
> > > > >> > > > level=\"A1\";
> > > > >> > > > > convert(x) = x * 10^9; },{*
> > > > >> > > > >
> > > > >> > > > > Below is where the error occurs:
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > *DEBUG 2: Reading data for PMTFL1.WARNING:WARNING:
> > > > >> > > > > process_fcst_climo_files() -> no fields matching
PMTFL1
> > > > >> > > > > found in
> > > > >> > file:
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190808/
> > > > >> > > > > aqm.t12z.metcro2d.36.nc
> > > > >> > > > > <http://aqm.t12z.metcro2d.36.nc>WARNING:DEBUG
2:DEBUG 2:
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> >
>
--------------------------------------------------------------------------------DEBUG
> > > > >> > > > > 2:DEBUG 2: Reading data for
OZCONA8.WARNING:WARNING:
> > > > >> > > > > process_fcst_climo_files() -> no fields matching
OZCONA8
> > > > >> > > > > found
> > > > in
> > > > >> > file:
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/CMAQ_Converted_Files/r101_v1/aqm.20190808/
> > > > >> > > > > aqm.t12z.metcro2d.36.nc
> > > > >> > > > > <http://aqm.t12z.metcro2d.36.nc>WARNING:ERROR
:ERROR :
> > > > >> > > > > process_fcst_climo_files() -> no requested forecast
data
> > > > >> > > > > found!
> > > > >> > > > > Exiting...ERROR :*
> > > > >> > > > >
> > > > >> > > > > You can see that the longname and level are joined,
and as
> > > > result,
> > > > >> > > fails
> > > > >> > > > > because that field doesn't exist. You can also
clearly
> > > > >> > > > > see that
> > > > >> my
> > > > >> > > > > configure file separates these fields:
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > *FCST_VAR1_NAME = OZCONFCST_VAR1_LEVELS =
A1OBS_VAR1_NAME=
> > > > >> > > > > COPOOBS_VAR1_LEVELS= A1OBS_VAR1_OPTIONS =
convert(x) = x
> > > > >> > > > > *
> > > > >> > > > > 10^9;FCST_VAR2_NAME = OZCONFCST_VAR2_LEVELS =
> > > > >> > > > > A8OBS_VAR2_NAME=
> > > > >> > > > > COPOOBS_VAR2_LEVELS= A8*
> > > > >> > > > >
> > > > >> > > > > It's not clear to me why this issue is occurring if
> > > > >> > > > > everything
> > > > >> else
> > > > >> > > > appears
> > > > >> > > > > correct.
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > > > --
> > > > >> > > > George McCabe - Software Engineer III
> > > > >> > > > National Center for Atmospheric Research
> > > > >> > > > Research Applications Laboratory
> > > > >> > > > 303-497-2768
> > > > >> > > > ---
> > > > >> > > > My working day may not be your working day. Please do
not
> > > > >> > > > feel
> > > > >> obliged
> > > > >> > to
> > > > >> > > > reply to this email outside of your normal working
hours.
> > > > >> > > >
> > > > >> > > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >> > --
> > > > >> > George McCabe - Software Engineer III
> > > > >> > National Center for Atmospheric Research
> > > > >> > Research Applications Laboratory
> > > > >> > 303-497-2768
> > > > >> > ---
> > > > >> > My working day may not be your working day. Please do not
feel
> > > > >> > obliged
> > > > >> to
> > > > >> > reply to this email outside of your normal working hours.
> > > > >> >
> > > > >> >
> > > > >>
> > > > >>
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > > > > obliged to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> > > > obliged to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> >
> >
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
More information about the Met_help
mailing list