[Met_help] [rt.rap.ucar.edu #93338] History for point_stat "unrecognized GRIB1 field"; file_type="NETCDF_MET" in config file
Julie Prestopnik via RT
met_help at ucar.edu
Tue Dec 17 10:20:29 MST 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Good afternoon,
I am having trouble running MET on netCDF data. I used pcp_combine
-subtract to successfully create a netCDF file
(APCP_wrfout_d05_2017120912_f007_f005.nc) containing accumulation data for
domain five of the AMPS evaluation area. I then used ascii2nc to create a
netCDF file (BATG_ob1.nc) containing an accumulation observation. "psc" is
my configuration file, in which I specify 'field_type="NETCDF_MET"' in the
fcst dictionary. With 'name="APCP"', running point_stat yields the
following:
>point_stat APCP_wrfout_d05_2017120912_f007_f005.nc BATG_ob1.nc psc
DEBUG 1: Default Config File:
/usr/local/met-8.1.1/share/met/config/PointStatConfig_default
DEBUG 1: User Config File: psc
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=18446744072084449373
DEBUG 1: Forecast File: APCP_wrfout_d05_2017120912_f007_f005.nc
DEBUG 1: Observation File: BATG_ob1.nc
DEBUG 2:
DEBUG 2:
--------------------------------------------------------------------------------
DEBUG 2:
DEBUG 2: Reading data for APCPA2.
WARNING:
WARNING: process_fcst_climo_files() -> no fields matching APCPA2 found in
file: APCP_wrfout_d05_2017120912_f007_f005.nc
WARNING:
ERROR :
ERROR : process_fcst_climo_files() -> no requested forecast data found!
Exiting...
ERROR :
yar:/d1/lave/antarctica> vi psc
yar:/d1/lave/antarctica> point_stat APCP_wrfout_d05_2017120912_f007_f005.nc
BATG_ob1.nc psc
DEBUG 1: Default Config File:
/usr/local/met-8.1.1/share/met/config/PointStatConfig_default
DEBUG 1: User Config File: psc
ERROR :
ERROR : VarInfoGrib::add_grib_code() -> unrecognized GRIB1 field
abbreviation 'APCP_02' for table version 2
ERROR :
So, following the answer given in [Met_help] [rt.rap.ucar.edu #85872], I
adjusted my config file to include 'name="APCP_02"'. Running point_stat
then yielded:
>point_stat APCP_wrfout_d05_2017120912_f007_f005.nc BATG_ob1.nc psc
DEBUG 1: Default Config File:
/usr/local/met-8.1.1/share/met/config/PointStatConfig_default
DEBUG 1: User Config File: psc
ERROR :
ERROR : VarInfoGrib::add_grib_code() -> unrecognized GRIB1 field
abbreviation 'APCP_02' for table version 2
ERROR :
What is going on here? It seems like point_stat is expecting a GRIB1 file,
though I specified the input to be a MET-generated netCDF file. I uploaded
the three files to your ftp site.
Sincerely,
Joshua Lave
--
Joshua Lave
*Associate Scientist INational Center for Atmospheric Research*
*(303) 497-8258*
*lave at ucar.edu <lave at ucar.edu>*
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: point_stat "unrecognized GRIB1 field"; file_type="NETCDF_MET" in config file
From: Julie Prestopnik
Time: Thu Dec 05 14:53:50 2019
Hi Joshua.
I see that you are having some difficult running point_stat with
NetCDF
data.
In your config file, I noticed that you had set the "output_flag" for
"pstd"
to "BOTH", but had left all other values as "NONE". "pstd" is for
probabilistic data. Is your intention to use this for verification of
probabilistic data or for traditional unit verification?
I made some changes to your config file and have attached a new config
file. I'll outline the differences below:
- I removed your entries for init_time, valid_time, and lead_time.
Those fields are typically used for file that have many lead times
specified. In this case, the file only has one time, so it is not
necessary to specify those values. You only want to add those
values if
they are needed. MET will be able to parse the file for those
values in
this case.
- I removed "file_type = "NETCDF_MET";" as it is also not necessary
in
this case.
- I changed the values of the "output_flag" section, assuming that
you
wanted to do traditional unit verification. So, I changed "pstd"
to
"NONE". I changed "ctc" (contingency table counts) and "cts"
(statistics
on those contingency table counts) to "BOTH". The "cat_thresh", or
categorical threshold, is used to define what constitutes an
"event" when
computing a 2x2 contingency table. I also changed "cnt",
continuous
statistics to "BOTH", because a lot of folks like to use that
information
as well.
- I also changed the "obs = fcst;" field to give "obs" its own
value.
In this case, obs=fcst won't work because "APCP_02" does not exist
in the
obs file, so MET would not find any matches. In order to have MET
be
backward compatible, we are stuck with the GRIB naming convention.
The
obs file contains a GRIB value for "obs_gc". This triggers logic
in MET to
do a table lookup, but MET would find "APCP" and not "APCP_02". I
can see
that the obs file BATG_ob1 was derived from using ascii2nc. If
"APCP_02" had been listed in the ascii file as the value for the
"Var_Name(or GRIB_Code)" field, and the NetCDF file then contained
"APCP_02", you may have likely been able to use "obs = fcst;".
However,
that brings us to another issue in that the message type, aka
"hdr_typ_table" (hdr_typ_table:long_name = "message type" ;), value
in the
obs file is "default" so MET will not find any matches there
either, as
you have set "message_type = [ "ADPSFC" ];" in the config file and
"default" wouldn't match "ADPSFC". You could edit the ascii data
and send
it through ascii2nc again. It is recommended to change "default"
to
something that is more meaningful for your data. So, MET will not
find any
matched pairs until that issue is resolved.
If I now run your data using point_stat with the changes and a higher
verbosity level:
> /usr/local/met-8.1.1/bin/point_stat
> APCP_wrfout_d05_2017120912_f007_f005.nc BATG_ob1.nc psc_new -v 3
>
I do get a successful run, however, there are still no matched pairs
for a
different reason. The unmatched message_type problem would still show
up
at some point, but the first problem I encountered when running with
the
new config file is:
> DEBUG 2: Processing APCP_02A2 versus APCP/Z0, for observation type
ADPSFC,
> over region FULL, for interpolation method NEAREST(1), using 0
matched
> pairs.
> DEBUG 3: Number of matched pairs = 0
> DEBUG 3: Observations processed = 1
> DEBUG 3: Rejected: SID exclusion = 0
> DEBUG 3: Rejected: obs type = 0
> DEBUG 3: Rejected: valid time = 0
> DEBUG 3: Rejected: bad obs value = 0
> DEBUG 3: Rejected: off the grid = 1
> DEBUG 3: Rejected: topography = 0
> DEBUG 3: Rejected: level mismatch = 0
> DEBUG 3: Rejected: quality marker = 0
> DEBUG 3: Rejected: message type = 0
> DEBUG 3: Rejected: masking region = 0
> DEBUG 3: Rejected: bad fcst value = 0
> DEBUG 3: Rejected: duplicates = 0
>
We can see that there are no matched pairs and that they were rejected
because they are "off the grid".
We can use plot_point_obs to visualize where the observation lies:
> /usr/local/met-8.1.1/bin/plot_point_obs BATG_ob1.nc BATG_ob1.ps
-dotsize 10
>
I have attached BATG_ob1.ps for you. We can see the observation is
near
Antarctica, and hopefully that is what you expect.
We can use plot_data_plane to visualize the gridded model output:
> /usr/local/met-8.1.1/bin/plot_data_plane
> APCP_wrfout_d05_2017120912_f007_f005.nc
> APCP_wrfout_d05_2017120912_f007_f005.ps 'name="APCP_02";
level="A2";'
>
However, there seems to be a problem with the projection information.
I
have also attached APCP_wrfout_d05_2017120912_f007_f005.ps for you.
Looking in APCP_wrfout_d05_2017120912_f007_f005.nc, I can see the
following delta values:
> :delta_lat = "0.005932 degrees" ;
> :delta_lon = "0.418030 degrees" ;
>
which don't look like typical values. Usually they are the same or
close
to the same.
We can use the regrid_data_plane tool to regrid to NCEP grid 4 to see
where
MET thinks the data is:
> /usr/local/met-8.1.1/bin/regrid_data_plane
> APCP_wrfout_d05_2017120912_f007_f005.nc G004 global.nc -field
> 'name="APCP_02"; level="A2";'
>
We can view this data using ncview:
> /usr/local/ncview/bin/ncview global.nc &
>
I took a screenshot of that image and attached it as global.nc.png.
You
can see the data is smooshed with a chuck missing as well. This seems
problematic. If you'd like to pass on the GRIB data that you used in
running pcp_combine, we'd be happy to take a look at that.
Also, since you're at NCAR, we could also get together to discuss how
you'd
like to use MET and how we can help you with it. My colleague, John
Halley
Gotway, helped me through this work. He is our MET expert. Please
let us
know if you'd like to arrange a time to meet with us.
I hope this is helpful. Please let us know what other questions you
have.
Thanks,
Julie
On Wed, Dec 4, 2019 at 2:48 PM lave at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> Wed Dec 04 14:48:26 2019: Request 93338 was acted upon.
> Transaction: Ticket created by lave at ucar.edu
> Queue: met_help
> Subject: point_stat "unrecognized GRIB1 field";
> file_type="NETCDF_MET" in config file
> Owner: Nobody
> Requestors: lave at ucar.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338 >
>
>
> Good afternoon,
>
> I am having trouble running MET on netCDF data. I used pcp_combine
> -subtract to successfully create a netCDF file
> (APCP_wrfout_d05_2017120912_f007_f005.nc) containing accumulation
data for
> domain five of the AMPS evaluation area. I then used ascii2nc to
create a
> netCDF file (BATG_ob1.nc) containing an accumulation observation.
"psc" is
> my configuration file, in which I specify 'field_type="NETCDF_MET"'
in the
> fcst dictionary. With 'name="APCP"', running point_stat yields the
> following:
>
> >point_stat APCP_wrfout_d05_2017120912_f007_f005.nc BATG_ob1.nc psc
>
> DEBUG 1: Default Config File:
> /usr/local/met-8.1.1/share/met/config/PointStatConfig_default
>
> DEBUG 1: User Config File: psc
>
> GSL_RNG_TYPE=mt19937
>
> GSL_RNG_SEED=18446744072084449373
>
> DEBUG 1: Forecast File: APCP_wrfout_d05_2017120912_f007_f005.nc
>
> DEBUG 1: Observation File: BATG_ob1.nc
>
> DEBUG 2:
>
> DEBUG 2:
>
>
--------------------------------------------------------------------------------
>
> DEBUG 2:
>
> DEBUG 2: Reading data for APCPA2.
>
> WARNING:
>
> WARNING: process_fcst_climo_files() -> no fields matching APCPA2
found in
> file: APCP_wrfout_d05_2017120912_f007_f005.nc
>
> WARNING:
>
> ERROR :
>
> ERROR : process_fcst_climo_files() -> no requested forecast data
found!
> Exiting...
>
> ERROR :
>
> yar:/d1/lave/antarctica> vi psc
>
> yar:/d1/lave/antarctica> point_stat APCP_wrfout_d05_2017120912_
> f007_f005.nc
> BATG_ob1.nc psc
>
> DEBUG 1: Default Config File:
> /usr/local/met-8.1.1/share/met/config/PointStatConfig_default
>
> DEBUG 1: User Config File: psc
>
> ERROR :
>
> ERROR : VarInfoGrib::add_grib_code() -> unrecognized GRIB1 field
> abbreviation 'APCP_02' for table version 2
>
> ERROR :
>
> So, following the answer given in [Met_help] [rt.rap.ucar.edu
#85872], I
> adjusted my config file to include 'name="APCP_02"'. Running
point_stat
> then yielded:
>
> >point_stat APCP_wrfout_d05_2017120912_f007_f005.nc BATG_ob1.nc psc
>
> DEBUG 1: Default Config File:
> /usr/local/met-8.1.1/share/met/config/PointStatConfig_default
>
> DEBUG 1: User Config File: psc
>
> ERROR :
>
> ERROR : VarInfoGrib::add_grib_code() -> unrecognized GRIB1 field
> abbreviation 'APCP_02' for table version 2
> ERROR :
>
> What is going on here? It seems like point_stat is expecting a GRIB1
file,
> though I specified the input to be a MET-generated netCDF file. I
uploaded
> the three files to your ftp site.
>
> Sincerely,
> Joshua Lave
> --
> Joshua Lave
>
> *Associate Scientist INational Center for Atmospheric Research*
> *(303) 497-8258*
> *lave at ucar.edu <lave at ucar.edu>*
>
>
--
Julie Prestopnik
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: point_stat "unrecognized GRIB1 field"; file_type="NETCDF_MET" in config file
From: lave at ucar.edu
Time: Fri Dec 13 15:04:00 2019
Julie,
Sorry it has taken me so long to reply, but thank you for your
detailed
response. This is my first time working with MET and I am doing so at
the
suggestion of another scientist. Most of my struggles so far have
related
to properly formatting my datasets. The GRIB data that I used in
pcp_combine is a set that I converted from netCDF via wgrib and cdo as
I
could not get pcp_combine to accept the nc files.
I would like to accept your offer to meet with me to discuss MET and
how I
plan to use it. I am available most of next week (12/16-12:20),
including
all day Monday, Wednesday, and Friday.
Joshua
On Thu, Dec 5, 2019 at 2:54 PM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
> Hi Joshua.
>
> I see that you are having some difficult running point_stat with
NetCDF
> data.
>
> In your config file, I noticed that you had set the "output_flag"
for
> "pstd"
> to "BOTH", but had left all other values as "NONE". "pstd" is for
> probabilistic data. Is your intention to use this for verification
of
> probabilistic data or for traditional unit verification?
>
> I made some changes to your config file and have attached a new
config
> file. I'll outline the differences below:
>
> - I removed your entries for init_time, valid_time, and
lead_time.
> Those fields are typically used for file that have many lead
times
> specified. In this case, the file only has one time, so it is
not
> necessary to specify those values. You only want to add those
values if
> they are needed. MET will be able to parse the file for those
values in
> this case.
> - I removed "file_type = "NETCDF_MET";" as it is also not
necessary in
> this case.
> - I changed the values of the "output_flag" section, assuming
that you
> wanted to do traditional unit verification. So, I changed "pstd"
to
> "NONE". I changed "ctc" (contingency table counts) and "cts"
> (statistics
> on those contingency table counts) to "BOTH". The "cat_thresh",
or
> categorical threshold, is used to define what constitutes an
"event"
> when
> computing a 2x2 contingency table. I also changed "cnt",
continuous
> statistics to "BOTH", because a lot of folks like to use that
> information
> as well.
> - I also changed the "obs = fcst;" field to give "obs" its own
value.
> In this case, obs=fcst won't work because "APCP_02" does not
exist in
> the
> obs file, so MET would not find any matches. In order to have
MET be
> backward compatible, we are stuck with the GRIB naming
convention. The
> obs file contains a GRIB value for "obs_gc". This triggers logic
in
> MET to
> do a table lookup, but MET would find "APCP" and not "APCP_02".
I can
> see
> that the obs file BATG_ob1 was derived from using ascii2nc. If
> "APCP_02" had been listed in the ascii file as the value for the
> "Var_Name(or GRIB_Code)" field, and the NetCDF file then
contained
> "APCP_02", you may have likely been able to use "obs = fcst;".
However,
> that brings us to another issue in that the message type, aka
> "hdr_typ_table" (hdr_typ_table:long_name = "message type" ;),
value in
> the
> obs file is "default" so MET will not find any matches there
either, as
> you have set "message_type = [ "ADPSFC" ];" in the config file
and
> "default" wouldn't match "ADPSFC". You could edit the ascii data
and
> send
> it through ascii2nc again. It is recommended to change "default"
to
> something that is more meaningful for your data. So, MET will
not find
> any
> matched pairs until that issue is resolved.
>
> If I now run your data using point_stat with the changes and a
higher
> verbosity level:
>
> > /usr/local/met-8.1.1/bin/point_stat
> > APCP_wrfout_d05_2017120912_f007_f005.nc BATG_ob1.nc psc_new -v 3
> >
>
> I do get a successful run, however, there are still no matched pairs
for a
> different reason. The unmatched message_type problem would still
show up
> at some point, but the first problem I encountered when running with
the
> new config file is:
>
> > DEBUG 2: Processing APCP_02A2 versus APCP/Z0, for observation type
> ADPSFC,
> > over region FULL, for interpolation method NEAREST(1), using 0
matched
> > pairs.
> > DEBUG 3: Number of matched pairs = 0
> > DEBUG 3: Observations processed = 1
> > DEBUG 3: Rejected: SID exclusion = 0
> > DEBUG 3: Rejected: obs type = 0
> > DEBUG 3: Rejected: valid time = 0
> > DEBUG 3: Rejected: bad obs value = 0
> > DEBUG 3: Rejected: off the grid = 1
> > DEBUG 3: Rejected: topography = 0
> > DEBUG 3: Rejected: level mismatch = 0
> > DEBUG 3: Rejected: quality marker = 0
> > DEBUG 3: Rejected: message type = 0
> > DEBUG 3: Rejected: masking region = 0
> > DEBUG 3: Rejected: bad fcst value = 0
> > DEBUG 3: Rejected: duplicates = 0
> >
>
> We can see that there are no matched pairs and that they were
rejected
> because they are "off the grid".
>
> We can use plot_point_obs to visualize where the observation lies:
>
> > /usr/local/met-8.1.1/bin/plot_point_obs BATG_ob1.nc BATG_ob1.ps
-dotsize
> 10
> >
>
> I have attached BATG_ob1.ps for you. We can see the observation is
near
> Antarctica, and hopefully that is what you expect.
>
> We can use plot_data_plane to visualize the gridded model output:
>
> > /usr/local/met-8.1.1/bin/plot_data_plane
> > APCP_wrfout_d05_2017120912_f007_f005.nc
> > APCP_wrfout_d05_2017120912_f007_f005.ps 'name="APCP_02";
level="A2";'
> >
>
> However, there seems to be a problem with the projection
information. I
> have also attached APCP_wrfout_d05_2017120912_f007_f005.ps for you.
> Looking in APCP_wrfout_d05_2017120912_f007_f005.nc, I can see the
> following delta values:
>
> > :delta_lat = "0.005932 degrees" ;
> > :delta_lon = "0.418030 degrees" ;
> >
> which don't look like typical values. Usually they are the same or
close
> to the same.
>
> We can use the regrid_data_plane tool to regrid to NCEP grid 4 to
see where
> MET thinks the data is:
>
> > /usr/local/met-8.1.1/bin/regrid_data_plane
> > APCP_wrfout_d05_2017120912_f007_f005.nc G004 global.nc -field
> > 'name="APCP_02"; level="A2";'
> >
> We can view this data using ncview:
>
> > /usr/local/ncview/bin/ncview global.nc &
> >
>
> I took a screenshot of that image and attached it as global.nc.png.
You
> can see the data is smooshed with a chuck missing as well. This
seems
> problematic. If you'd like to pass on the GRIB data that you used
in
> running pcp_combine, we'd be happy to take a look at that.
>
> Also, since you're at NCAR, we could also get together to discuss
how you'd
> like to use MET and how we can help you with it. My colleague, John
Halley
> Gotway, helped me through this work. He is our MET expert. Please
let us
> know if you'd like to arrange a time to meet with us.
>
> I hope this is helpful. Please let us know what other questions you
have.
>
> Thanks,
> Julie
>
> On Wed, Dec 4, 2019 at 2:48 PM lave at ucar.edu via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Wed Dec 04 14:48:26 2019: Request 93338 was acted upon.
> > Transaction: Ticket created by lave at ucar.edu
> > Queue: met_help
> > Subject: point_stat "unrecognized GRIB1 field";
> > file_type="NETCDF_MET" in config file
> > Owner: Nobody
> > Requestors: lave at ucar.edu
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338 >
> >
> >
> > Good afternoon,
> >
> > I am having trouble running MET on netCDF data. I used pcp_combine
> > -subtract to successfully create a netCDF file
> > (APCP_wrfout_d05_2017120912_f007_f005.nc) containing accumulation
data
> for
> > domain five of the AMPS evaluation area. I then used ascii2nc to
create a
> > netCDF file (BATG_ob1.nc) containing an accumulation observation.
"psc"
> is
> > my configuration file, in which I specify
'field_type="NETCDF_MET"' in
> the
> > fcst dictionary. With 'name="APCP"', running point_stat yields the
> > following:
> >
> > >point_stat APCP_wrfout_d05_2017120912_f007_f005.nc BATG_ob1.nc
psc
> >
> > DEBUG 1: Default Config File:
> > /usr/local/met-8.1.1/share/met/config/PointStatConfig_default
> >
> > DEBUG 1: User Config File: psc
> >
> > GSL_RNG_TYPE=mt19937
> >
> > GSL_RNG_SEED=18446744072084449373
> >
> > DEBUG 1: Forecast File: APCP_wrfout_d05_2017120912_f007_f005.nc
> >
> > DEBUG 1: Observation File: BATG_ob1.nc
> >
> > DEBUG 2:
> >
> > DEBUG 2:
> >
> >
>
--------------------------------------------------------------------------------
> >
> > DEBUG 2:
> >
> > DEBUG 2: Reading data for APCPA2.
> >
> > WARNING:
> >
> > WARNING: process_fcst_climo_files() -> no fields matching APCPA2
found in
> > file: APCP_wrfout_d05_2017120912_f007_f005.nc
> >
> > WARNING:
> >
> > ERROR :
> >
> > ERROR : process_fcst_climo_files() -> no requested forecast data
found!
> > Exiting...
> >
> > ERROR :
> >
> > yar:/d1/lave/antarctica> vi psc
> >
> > yar:/d1/lave/antarctica> point_stat APCP_wrfout_d05_2017120912_
> > f007_f005.nc
> > BATG_ob1.nc psc
> >
> > DEBUG 1: Default Config File:
> > /usr/local/met-8.1.1/share/met/config/PointStatConfig_default
> >
> > DEBUG 1: User Config File: psc
> >
> > ERROR :
> >
> > ERROR : VarInfoGrib::add_grib_code() -> unrecognized GRIB1 field
> > abbreviation 'APCP_02' for table version 2
> >
> > ERROR :
> >
> > So, following the answer given in [Met_help] [rt.rap.ucar.edu
#85872], I
> > adjusted my config file to include 'name="APCP_02"'. Running
point_stat
> > then yielded:
> >
> > >point_stat APCP_wrfout_d05_2017120912_f007_f005.nc BATG_ob1.nc
psc
> >
> > DEBUG 1: Default Config File:
> > /usr/local/met-8.1.1/share/met/config/PointStatConfig_default
> >
> > DEBUG 1: User Config File: psc
> >
> > ERROR :
> >
> > ERROR : VarInfoGrib::add_grib_code() -> unrecognized GRIB1 field
> > abbreviation 'APCP_02' for table version 2
> > ERROR :
> >
> > What is going on here? It seems like point_stat is expecting a
GRIB1
> file,
> > though I specified the input to be a MET-generated netCDF file. I
> uploaded
> > the three files to your ftp site.
> >
> > Sincerely,
> > Joshua Lave
> > --
> > Joshua Lave
> >
> > *Associate Scientist INational Center for Atmospheric Research*
> > *(303) 497-8258*
> > *lave at ucar.edu <lave at ucar.edu>*
> >
> >
>
> --
> Julie Prestopnik
> Software Engineer
> National Center for Atmospheric Research
> Research Applications Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
--
Joshua Lave
*Associate Scientist INational Center for Atmospheric Research*
*(303) 497-8258*
*lave at ucar.edu <lave at ucar.edu>*
------------------------------------------------
Subject: point_stat "unrecognized GRIB1 field"; file_type="NETCDF_MET" in config file
From: John Halley Gotway
Time: Fri Dec 13 16:45:01 2019
Just put it on the calendar for Monday at 3pm in FL3-1029, the same
room as
the METplus Engineering meeting.
Have a good weekend. Actually we're going to see the Best Xmas
Pageant on
Saturday night, so maybe we'll see you there.
John
On Fri, Dec 13, 2019 at 3:04 PM lave at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338 >
>
> Julie,
>
> Sorry it has taken me so long to reply, but thank you for your
detailed
> response. This is my first time working with MET and I am doing so
at the
> suggestion of another scientist. Most of my struggles so far have
related
> to properly formatting my datasets. The GRIB data that I used in
> pcp_combine is a set that I converted from netCDF via wgrib and cdo
as I
> could not get pcp_combine to accept the nc files.
>
> I would like to accept your offer to meet with me to discuss MET and
how I
> plan to use it. I am available most of next week (12/16-12:20),
including
> all day Monday, Wednesday, and Friday.
>
> Joshua
>
> On Thu, Dec 5, 2019 at 2:54 PM Julie Prestopnik via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Joshua.
> >
> > I see that you are having some difficult running point_stat with
NetCDF
> > data.
> >
> > In your config file, I noticed that you had set the "output_flag"
for
> > "pstd"
> > to "BOTH", but had left all other values as "NONE". "pstd" is for
> > probabilistic data. Is your intention to use this for
verification of
> > probabilistic data or for traditional unit verification?
> >
> > I made some changes to your config file and have attached a new
config
> > file. I'll outline the differences below:
> >
> > - I removed your entries for init_time, valid_time, and
lead_time.
> > Those fields are typically used for file that have many lead
times
> > specified. In this case, the file only has one time, so it is
not
> > necessary to specify those values. You only want to add those
values
> if
> > they are needed. MET will be able to parse the file for those
values
> in
> > this case.
> > - I removed "file_type = "NETCDF_MET";" as it is also not
necessary in
> > this case.
> > - I changed the values of the "output_flag" section, assuming
that you
> > wanted to do traditional unit verification. So, I changed
"pstd" to
> > "NONE". I changed "ctc" (contingency table counts) and "cts"
> > (statistics
> > on those contingency table counts) to "BOTH". The
"cat_thresh", or
> > categorical threshold, is used to define what constitutes an
"event"
> > when
> > computing a 2x2 contingency table. I also changed "cnt",
continuous
> > statistics to "BOTH", because a lot of folks like to use that
> > information
> > as well.
> > - I also changed the "obs = fcst;" field to give "obs" its own
value.
> > In this case, obs=fcst won't work because "APCP_02" does not
exist in
> > the
> > obs file, so MET would not find any matches. In order to have
MET be
> > backward compatible, we are stuck with the GRIB naming
convention.
> The
> > obs file contains a GRIB value for "obs_gc". This triggers
logic in
> > MET to
> > do a table lookup, but MET would find "APCP" and not "APCP_02".
I can
> > see
> > that the obs file BATG_ob1 was derived from using ascii2nc. If
> > "APCP_02" had been listed in the ascii file as the value for
the
> > "Var_Name(or GRIB_Code)" field, and the NetCDF file then
contained
> > "APCP_02", you may have likely been able to use "obs = fcst;".
> However,
> > that brings us to another issue in that the message type, aka
> > "hdr_typ_table" (hdr_typ_table:long_name = "message type" ;),
value in
> > the
> > obs file is "default" so MET will not find any matches there
either,
> as
> > you have set "message_type = [ "ADPSFC" ];" in the config file
and
> > "default" wouldn't match "ADPSFC". You could edit the ascii
data and
> > send
> > it through ascii2nc again. It is recommended to change
"default" to
> > something that is more meaningful for your data. So, MET will
not
> find
> > any
> > matched pairs until that issue is resolved.
> >
> > If I now run your data using point_stat with the changes and a
higher
> > verbosity level:
> >
> > > /usr/local/met-8.1.1/bin/point_stat
> > > APCP_wrfout_d05_2017120912_f007_f005.nc BATG_ob1.nc psc_new -v 3
> > >
> >
> > I do get a successful run, however, there are still no matched
pairs for
> a
> > different reason. The unmatched message_type problem would still
show up
> > at some point, but the first problem I encountered when running
with the
> > new config file is:
> >
> > > DEBUG 2: Processing APCP_02A2 versus APCP/Z0, for observation
type
> > ADPSFC,
> > > over region FULL, for interpolation method NEAREST(1), using 0
matched
> > > pairs.
> > > DEBUG 3: Number of matched pairs = 0
> > > DEBUG 3: Observations processed = 1
> > > DEBUG 3: Rejected: SID exclusion = 0
> > > DEBUG 3: Rejected: obs type = 0
> > > DEBUG 3: Rejected: valid time = 0
> > > DEBUG 3: Rejected: bad obs value = 0
> > > DEBUG 3: Rejected: off the grid = 1
> > > DEBUG 3: Rejected: topography = 0
> > > DEBUG 3: Rejected: level mismatch = 0
> > > DEBUG 3: Rejected: quality marker = 0
> > > DEBUG 3: Rejected: message type = 0
> > > DEBUG 3: Rejected: masking region = 0
> > > DEBUG 3: Rejected: bad fcst value = 0
> > > DEBUG 3: Rejected: duplicates = 0
> > >
> >
> > We can see that there are no matched pairs and that they were
rejected
> > because they are "off the grid".
> >
> > We can use plot_point_obs to visualize where the observation lies:
> >
> > > /usr/local/met-8.1.1/bin/plot_point_obs BATG_ob1.nc BATG_ob1.ps
> -dotsize
> > 10
> > >
> >
> > I have attached BATG_ob1.ps for you. We can see the observation
is near
> > Antarctica, and hopefully that is what you expect.
> >
> > We can use plot_data_plane to visualize the gridded model output:
> >
> > > /usr/local/met-8.1.1/bin/plot_data_plane
> > > APCP_wrfout_d05_2017120912_f007_f005.nc
> > > APCP_wrfout_d05_2017120912_f007_f005.ps 'name="APCP_02";
level="A2";'
> > >
> >
> > However, there seems to be a problem with the projection
information. I
> > have also attached APCP_wrfout_d05_2017120912_f007_f005.ps for
you.
> > Looking in APCP_wrfout_d05_2017120912_f007_f005.nc, I can see the
> > following delta values:
> >
> > > :delta_lat = "0.005932 degrees" ;
> > > :delta_lon = "0.418030 degrees" ;
> > >
> > which don't look like typical values. Usually they are the same
or close
> > to the same.
> >
> > We can use the regrid_data_plane tool to regrid to NCEP grid 4 to
see
> where
> > MET thinks the data is:
> >
> > > /usr/local/met-8.1.1/bin/regrid_data_plane
> > > APCP_wrfout_d05_2017120912_f007_f005.nc G004 global.nc -field
> > > 'name="APCP_02"; level="A2";'
> > >
> > We can view this data using ncview:
> >
> > > /usr/local/ncview/bin/ncview global.nc &
> > >
> >
> > I took a screenshot of that image and attached it as
global.nc.png. You
> > can see the data is smooshed with a chuck missing as well. This
seems
> > problematic. If you'd like to pass on the GRIB data that you used
in
> > running pcp_combine, we'd be happy to take a look at that.
> >
> > Also, since you're at NCAR, we could also get together to discuss
how
> you'd
> > like to use MET and how we can help you with it. My colleague,
John
> Halley
> > Gotway, helped me through this work. He is our MET expert.
Please let
> us
> > know if you'd like to arrange a time to meet with us.
> >
> > I hope this is helpful. Please let us know what other questions
you
> have.
> >
> > Thanks,
> > Julie
> >
> > On Wed, Dec 4, 2019 at 2:48 PM lave at ucar.edu via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > Wed Dec 04 14:48:26 2019: Request 93338 was acted upon.
> > > Transaction: Ticket created by lave at ucar.edu
> > > Queue: met_help
> > > Subject: point_stat "unrecognized GRIB1 field";
> > > file_type="NETCDF_MET" in config file
> > > Owner: Nobody
> > > Requestors: lave at ucar.edu
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338
> >
> > >
> > >
> > > Good afternoon,
> > >
> > > I am having trouble running MET on netCDF data. I used
pcp_combine
> > > -subtract to successfully create a netCDF file
> > > (APCP_wrfout_d05_2017120912_f007_f005.nc) containing
accumulation data
> > for
> > > domain five of the AMPS evaluation area. I then used ascii2nc to
> create a
> > > netCDF file (BATG_ob1.nc) containing an accumulation
observation. "psc"
> > is
> > > my configuration file, in which I specify
'field_type="NETCDF_MET"' in
> > the
> > > fcst dictionary. With 'name="APCP"', running point_stat yields
the
> > > following:
> > >
> > > >point_stat APCP_wrfout_d05_2017120912_f007_f005.nc BATG_ob1.nc
psc
> > >
> > > DEBUG 1: Default Config File:
> > > /usr/local/met-8.1.1/share/met/config/PointStatConfig_default
> > >
> > > DEBUG 1: User Config File: psc
> > >
> > > GSL_RNG_TYPE=mt19937
> > >
> > > GSL_RNG_SEED=18446744072084449373
> > >
> > > DEBUG 1: Forecast File: APCP_wrfout_d05_2017120912_f007_f005.nc
> > >
> > > DEBUG 1: Observation File: BATG_ob1.nc
> > >
> > > DEBUG 2:
> > >
> > > DEBUG 2:
> > >
> > >
> >
>
--------------------------------------------------------------------------------
> > >
> > > DEBUG 2:
> > >
> > > DEBUG 2: Reading data for APCPA2.
> > >
> > > WARNING:
> > >
> > > WARNING: process_fcst_climo_files() -> no fields matching APCPA2
found
> in
> > > file: APCP_wrfout_d05_2017120912_f007_f005.nc
> > >
> > > WARNING:
> > >
> > > ERROR :
> > >
> > > ERROR : process_fcst_climo_files() -> no requested forecast
data
> found!
> > > Exiting...
> > >
> > > ERROR :
> > >
> > > yar:/d1/lave/antarctica> vi psc
> > >
> > > yar:/d1/lave/antarctica> point_stat APCP_wrfout_d05_2017120912_
> > > f007_f005.nc
> > > BATG_ob1.nc psc
> > >
> > > DEBUG 1: Default Config File:
> > > /usr/local/met-8.1.1/share/met/config/PointStatConfig_default
> > >
> > > DEBUG 1: User Config File: psc
> > >
> > > ERROR :
> > >
> > > ERROR : VarInfoGrib::add_grib_code() -> unrecognized GRIB1
field
> > > abbreviation 'APCP_02' for table version 2
> > >
> > > ERROR :
> > >
> > > So, following the answer given in [Met_help] [rt.rap.ucar.edu
> #85872], I
> > > adjusted my config file to include 'name="APCP_02"'. Running
point_stat
> > > then yielded:
> > >
> > > >point_stat APCP_wrfout_d05_2017120912_f007_f005.nc BATG_ob1.nc
psc
> > >
> > > DEBUG 1: Default Config File:
> > > /usr/local/met-8.1.1/share/met/config/PointStatConfig_default
> > >
> > > DEBUG 1: User Config File: psc
> > >
> > > ERROR :
> > >
> > > ERROR : VarInfoGrib::add_grib_code() -> unrecognized GRIB1
field
> > > abbreviation 'APCP_02' for table version 2
> > > ERROR :
> > >
> > > What is going on here? It seems like point_stat is expecting a
GRIB1
> > file,
> > > though I specified the input to be a MET-generated netCDF file.
I
> > uploaded
> > > the three files to your ftp site.
> > >
> > > Sincerely,
> > > Joshua Lave
> > > --
> > > Joshua Lave
> > >
> > > *Associate Scientist INational Center for Atmospheric Research*
> > > *(303) 497-8258*
> > > *lave at ucar.edu <lave at ucar.edu>*
> > >
> > >
> >
> > --
> > Julie Prestopnik
> > Software Engineer
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > Phone: 303.497.8399
> > Email: jpresto at ucar.edu
> >
> > My working day may not be your working day. Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
> Joshua Lave
>
> *Associate Scientist INational Center for Atmospheric Research*
> *(303) 497-8258*
> *lave at ucar.edu <lave at ucar.edu>*
>
>
------------------------------------------------
Subject: point_stat "unrecognized GRIB1 field"; file_type="NETCDF_MET" in config file
From: Julie Prestopnik
Time: Mon Dec 16 09:40:44 2019
Hi Joshua.
The delay in response is no problem. It's a busy time of year. We're
happy to help you get started using MET.
My colleague, John, set up a meeting for us today at 3pm in FL3-1029.
I
just wanted to make sure you received the invitation and check in to
see if
you'll be able to meet at that time? I'll be able to stay for about
the
first 20 minutes of the meeting.
Thanks,
Julie
On Fri, Dec 13, 2019 at 3:04 PM lave at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338 >
>
> Julie,
>
> Sorry it has taken me so long to reply, but thank you for your
detailed
> response. This is my first time working with MET and I am doing so
at the
> suggestion of another scientist. Most of my struggles so far have
related
> to properly formatting my datasets. The GRIB data that I used in
> pcp_combine is a set that I converted from netCDF via wgrib and cdo
as I
> could not get pcp_combine to accept the nc files.
>
> I would like to accept your offer to meet with me to discuss MET and
how I
> plan to use it. I am available most of next week (12/16-12:20),
including
> all day Monday, Wednesday, and Friday.
>
> Joshua
>
> On Thu, Dec 5, 2019 at 2:54 PM Julie Prestopnik via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Joshua.
> >
> > I see that you are having some difficult running point_stat with
NetCDF
> > data.
> >
> > In your config file, I noticed that you had set the "output_flag"
for
> > "pstd"
> > to "BOTH", but had left all other values as "NONE". "pstd" is for
> > probabilistic data. Is your intention to use this for
verification of
> > probabilistic data or for traditional unit verification?
> >
> > I made some changes to your config file and have attached a new
config
> > file. I'll outline the differences below:
> >
> > - I removed your entries for init_time, valid_time, and
lead_time.
> > Those fields are typically used for file that have many lead
times
> > specified. In this case, the file only has one time, so it is
not
> > necessary to specify those values. You only want to add those
values
> if
> > they are needed. MET will be able to parse the file for those
values
> in
> > this case.
> > - I removed "file_type = "NETCDF_MET";" as it is also not
necessary in
> > this case.
> > - I changed the values of the "output_flag" section, assuming
that you
> > wanted to do traditional unit verification. So, I changed
"pstd" to
> > "NONE". I changed "ctc" (contingency table counts) and "cts"
> > (statistics
> > on those contingency table counts) to "BOTH". The
"cat_thresh", or
> > categorical threshold, is used to define what constitutes an
"event"
> > when
> > computing a 2x2 contingency table. I also changed "cnt",
continuous
> > statistics to "BOTH", because a lot of folks like to use that
> > information
> > as well.
> > - I also changed the "obs = fcst;" field to give "obs" its own
value.
> > In this case, obs=fcst won't work because "APCP_02" does not
exist in
> > the
> > obs file, so MET would not find any matches. In order to have
MET be
> > backward compatible, we are stuck with the GRIB naming
convention.
> The
> > obs file contains a GRIB value for "obs_gc". This triggers
logic in
> > MET to
> > do a table lookup, but MET would find "APCP" and not "APCP_02".
I can
> > see
> > that the obs file BATG_ob1 was derived from using ascii2nc. If
> > "APCP_02" had been listed in the ascii file as the value for
the
> > "Var_Name(or GRIB_Code)" field, and the NetCDF file then
contained
> > "APCP_02", you may have likely been able to use "obs = fcst;".
> However,
> > that brings us to another issue in that the message type, aka
> > "hdr_typ_table" (hdr_typ_table:long_name = "message type" ;),
value in
> > the
> > obs file is "default" so MET will not find any matches there
either,
> as
> > you have set "message_type = [ "ADPSFC" ];" in the config file
and
> > "default" wouldn't match "ADPSFC". You could edit the ascii
data and
> > send
> > it through ascii2nc again. It is recommended to change
"default" to
> > something that is more meaningful for your data. So, MET will
not
> find
> > any
> > matched pairs until that issue is resolved.
> >
> > If I now run your data using point_stat with the changes and a
higher
> > verbosity level:
> >
> > > /usr/local/met-8.1.1/bin/point_stat
> > > APCP_wrfout_d05_2017120912_f007_f005.nc BATG_ob1.nc psc_new -v 3
> > >
> >
> > I do get a successful run, however, there are still no matched
pairs for
> a
> > different reason. The unmatched message_type problem would still
show up
> > at some point, but the first problem I encountered when running
with the
> > new config file is:
> >
> > > DEBUG 2: Processing APCP_02A2 versus APCP/Z0, for observation
type
> > ADPSFC,
> > > over region FULL, for interpolation method NEAREST(1), using 0
matched
> > > pairs.
> > > DEBUG 3: Number of matched pairs = 0
> > > DEBUG 3: Observations processed = 1
> > > DEBUG 3: Rejected: SID exclusion = 0
> > > DEBUG 3: Rejected: obs type = 0
> > > DEBUG 3: Rejected: valid time = 0
> > > DEBUG 3: Rejected: bad obs value = 0
> > > DEBUG 3: Rejected: off the grid = 1
> > > DEBUG 3: Rejected: topography = 0
> > > DEBUG 3: Rejected: level mismatch = 0
> > > DEBUG 3: Rejected: quality marker = 0
> > > DEBUG 3: Rejected: message type = 0
> > > DEBUG 3: Rejected: masking region = 0
> > > DEBUG 3: Rejected: bad fcst value = 0
> > > DEBUG 3: Rejected: duplicates = 0
> > >
> >
> > We can see that there are no matched pairs and that they were
rejected
> > because they are "off the grid".
> >
> > We can use plot_point_obs to visualize where the observation lies:
> >
> > > /usr/local/met-8.1.1/bin/plot_point_obs BATG_ob1.nc BATG_ob1.ps
> -dotsize
> > 10
> > >
> >
> > I have attached BATG_ob1.ps for you. We can see the observation
is near
> > Antarctica, and hopefully that is what you expect.
> >
> > We can use plot_data_plane to visualize the gridded model output:
> >
> > > /usr/local/met-8.1.1/bin/plot_data_plane
> > > APCP_wrfout_d05_2017120912_f007_f005.nc
> > > APCP_wrfout_d05_2017120912_f007_f005.ps 'name="APCP_02";
level="A2";'
> > >
> >
> > However, there seems to be a problem with the projection
information. I
> > have also attached APCP_wrfout_d05_2017120912_f007_f005.ps for
you.
> > Looking in APCP_wrfout_d05_2017120912_f007_f005.nc, I can see the
> > following delta values:
> >
> > > :delta_lat = "0.005932 degrees" ;
> > > :delta_lon = "0.418030 degrees" ;
> > >
> > which don't look like typical values. Usually they are the same
or close
> > to the same.
> >
> > We can use the regrid_data_plane tool to regrid to NCEP grid 4 to
see
> where
> > MET thinks the data is:
> >
> > > /usr/local/met-8.1.1/bin/regrid_data_plane
> > > APCP_wrfout_d05_2017120912_f007_f005.nc G004 global.nc -field
> > > 'name="APCP_02"; level="A2";'
> > >
> > We can view this data using ncview:
> >
> > > /usr/local/ncview/bin/ncview global.nc &
> > >
> >
> > I took a screenshot of that image and attached it as
global.nc.png. You
> > can see the data is smooshed with a chuck missing as well. This
seems
> > problematic. If you'd like to pass on the GRIB data that you used
in
> > running pcp_combine, we'd be happy to take a look at that.
> >
> > Also, since you're at NCAR, we could also get together to discuss
how
> you'd
> > like to use MET and how we can help you with it. My colleague,
John
> Halley
> > Gotway, helped me through this work. He is our MET expert.
Please let
> us
> > know if you'd like to arrange a time to meet with us.
> >
> > I hope this is helpful. Please let us know what other questions
you
> have.
> >
> > Thanks,
> > Julie
> >
> > On Wed, Dec 4, 2019 at 2:48 PM lave at ucar.edu via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > Wed Dec 04 14:48:26 2019: Request 93338 was acted upon.
> > > Transaction: Ticket created by lave at ucar.edu
> > > Queue: met_help
> > > Subject: point_stat "unrecognized GRIB1 field";
> > > file_type="NETCDF_MET" in config file
> > > Owner: Nobody
> > > Requestors: lave at ucar.edu
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338
> >
> > >
> > >
> > > Good afternoon,
> > >
> > > I am having trouble running MET on netCDF data. I used
pcp_combine
> > > -subtract to successfully create a netCDF file
> > > (APCP_wrfout_d05_2017120912_f007_f005.nc) containing
accumulation data
> > for
> > > domain five of the AMPS evaluation area. I then used ascii2nc to
> create a
> > > netCDF file (BATG_ob1.nc) containing an accumulation
observation. "psc"
> > is
> > > my configuration file, in which I specify
'field_type="NETCDF_MET"' in
> > the
> > > fcst dictionary. With 'name="APCP"', running point_stat yields
the
> > > following:
> > >
> > > >point_stat APCP_wrfout_d05_2017120912_f007_f005.nc BATG_ob1.nc
psc
> > >
> > > DEBUG 1: Default Config File:
> > > /usr/local/met-8.1.1/share/met/config/PointStatConfig_default
> > >
> > > DEBUG 1: User Config File: psc
> > >
> > > GSL_RNG_TYPE=mt19937
> > >
> > > GSL_RNG_SEED=18446744072084449373
> > >
> > > DEBUG 1: Forecast File: APCP_wrfout_d05_2017120912_f007_f005.nc
> > >
> > > DEBUG 1: Observation File: BATG_ob1.nc
> > >
> > > DEBUG 2:
> > >
> > > DEBUG 2:
> > >
> > >
> >
>
--------------------------------------------------------------------------------
> > >
> > > DEBUG 2:
> > >
> > > DEBUG 2: Reading data for APCPA2.
> > >
> > > WARNING:
> > >
> > > WARNING: process_fcst_climo_files() -> no fields matching APCPA2
found
> in
> > > file: APCP_wrfout_d05_2017120912_f007_f005.nc
> > >
> > > WARNING:
> > >
> > > ERROR :
> > >
> > > ERROR : process_fcst_climo_files() -> no requested forecast
data
> found!
> > > Exiting...
> > >
> > > ERROR :
> > >
> > > yar:/d1/lave/antarctica> vi psc
> > >
> > > yar:/d1/lave/antarctica> point_stat APCP_wrfout_d05_2017120912_
> > > f007_f005.nc
> > > BATG_ob1.nc psc
> > >
> > > DEBUG 1: Default Config File:
> > > /usr/local/met-8.1.1/share/met/config/PointStatConfig_default
> > >
> > > DEBUG 1: User Config File: psc
> > >
> > > ERROR :
> > >
> > > ERROR : VarInfoGrib::add_grib_code() -> unrecognized GRIB1
field
> > > abbreviation 'APCP_02' for table version 2
> > >
> > > ERROR :
> > >
> > > So, following the answer given in [Met_help] [rt.rap.ucar.edu
> #85872], I
> > > adjusted my config file to include 'name="APCP_02"'. Running
point_stat
> > > then yielded:
> > >
> > > >point_stat APCP_wrfout_d05_2017120912_f007_f005.nc BATG_ob1.nc
psc
> > >
> > > DEBUG 1: Default Config File:
> > > /usr/local/met-8.1.1/share/met/config/PointStatConfig_default
> > >
> > > DEBUG 1: User Config File: psc
> > >
> > > ERROR :
> > >
> > > ERROR : VarInfoGrib::add_grib_code() -> unrecognized GRIB1
field
> > > abbreviation 'APCP_02' for table version 2
> > > ERROR :
> > >
> > > What is going on here? It seems like point_stat is expecting a
GRIB1
> > file,
> > > though I specified the input to be a MET-generated netCDF file.
I
> > uploaded
> > > the three files to your ftp site.
> > >
> > > Sincerely,
> > > Joshua Lave
> > > --
> > > Joshua Lave
> > >
> > > *Associate Scientist INational Center for Atmospheric Research*
> > > *(303) 497-8258*
> > > *lave at ucar.edu <lave at ucar.edu>*
> > >
> > >
> >
> > --
> > Julie Prestopnik
> > Software Engineer
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > Phone: 303.497.8399
> > Email: jpresto at ucar.edu
> >
> > My working day may not be your working day. Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
> Joshua Lave
>
> *Associate Scientist INational Center for Atmospheric Research*
> *(303) 497-8258*
> *lave at ucar.edu <lave at ucar.edu>*
>
>
--
Julie Prestopnik
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: point_stat "unrecognized GRIB1 field"; file_type="NETCDF_MET" in config file
From: lave at ucar.edu
Time: Mon Dec 16 13:14:27 2019
Yes, thank you for checking in. I did receive the invitation and
accepted
it. I'll see you at 3.
Joshua
On Mon, Dec 16, 2019 at 9:41 AM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
> Hi Joshua.
>
> The delay in response is no problem. It's a busy time of year.
We're
> happy to help you get started using MET.
>
> My colleague, John, set up a meeting for us today at 3pm in FL3-
1029. I
> just wanted to make sure you received the invitation and check in to
see if
> you'll be able to meet at that time? I'll be able to stay for about
the
> first 20 minutes of the meeting.
>
> Thanks,
> Julie
>
> On Fri, Dec 13, 2019 at 3:04 PM lave at ucar.edu via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338 >
> >
> > Julie,
> >
> > Sorry it has taken me so long to reply, but thank you for your
detailed
> > response. This is my first time working with MET and I am doing so
at the
> > suggestion of another scientist. Most of my struggles so far have
related
> > to properly formatting my datasets. The GRIB data that I used in
> > pcp_combine is a set that I converted from netCDF via wgrib and
cdo as I
> > could not get pcp_combine to accept the nc files.
> >
> > I would like to accept your offer to meet with me to discuss MET
and how
> I
> > plan to use it. I am available most of next week (12/16-12:20),
including
> > all day Monday, Wednesday, and Friday.
> >
> > Joshua
> >
> > On Thu, Dec 5, 2019 at 2:54 PM Julie Prestopnik via RT <
> met_help at ucar.edu>
> > wrote:
> >
> > > Hi Joshua.
> > >
> > > I see that you are having some difficult running point_stat with
NetCDF
> > > data.
> > >
> > > In your config file, I noticed that you had set the
"output_flag" for
> > > "pstd"
> > > to "BOTH", but had left all other values as "NONE". "pstd" is
for
> > > probabilistic data. Is your intention to use this for
verification of
> > > probabilistic data or for traditional unit verification?
> > >
> > > I made some changes to your config file and have attached a new
config
> > > file. I'll outline the differences below:
> > >
> > > - I removed your entries for init_time, valid_time, and
lead_time.
> > > Those fields are typically used for file that have many lead
times
> > > specified. In this case, the file only has one time, so it
is not
> > > necessary to specify those values. You only want to add
those
> values
> > if
> > > they are needed. MET will be able to parse the file for
those
> values
> > in
> > > this case.
> > > - I removed "file_type = "NETCDF_MET";" as it is also not
necessary
> in
> > > this case.
> > > - I changed the values of the "output_flag" section, assuming
that
> you
> > > wanted to do traditional unit verification. So, I changed
"pstd" to
> > > "NONE". I changed "ctc" (contingency table counts) and "cts"
> > > (statistics
> > > on those contingency table counts) to "BOTH". The
"cat_thresh", or
> > > categorical threshold, is used to define what constitutes an
"event"
> > > when
> > > computing a 2x2 contingency table. I also changed "cnt",
continuous
> > > statistics to "BOTH", because a lot of folks like to use that
> > > information
> > > as well.
> > > - I also changed the "obs = fcst;" field to give "obs" its
own
> value.
> > > In this case, obs=fcst won't work because "APCP_02" does not
exist
> in
> > > the
> > > obs file, so MET would not find any matches. In order to
have MET
> be
> > > backward compatible, we are stuck with the GRIB naming
convention.
> > The
> > > obs file contains a GRIB value for "obs_gc". This triggers
logic in
> > > MET to
> > > do a table lookup, but MET would find "APCP" and not
"APCP_02". I
> can
> > > see
> > > that the obs file BATG_ob1 was derived from using ascii2nc.
If
> > > "APCP_02" had been listed in the ascii file as the value for
the
> > > "Var_Name(or GRIB_Code)" field, and the NetCDF file then
contained
> > > "APCP_02", you may have likely been able to use "obs =
fcst;".
> > However,
> > > that brings us to another issue in that the message type, aka
> > > "hdr_typ_table" (hdr_typ_table:long_name = "message type" ;),
value
> in
> > > the
> > > obs file is "default" so MET will not find any matches there
either,
> > as
> > > you have set "message_type = [ "ADPSFC" ];" in the config
file and
> > > "default" wouldn't match "ADPSFC". You could edit the ascii
data
> and
> > > send
> > > it through ascii2nc again. It is recommended to change
"default" to
> > > something that is more meaningful for your data. So, MET
will not
> > find
> > > any
> > > matched pairs until that issue is resolved.
> > >
> > > If I now run your data using point_stat with the changes and a
higher
> > > verbosity level:
> > >
> > > > /usr/local/met-8.1.1/bin/point_stat
> > > > APCP_wrfout_d05_2017120912_f007_f005.nc BATG_ob1.nc psc_new -v
3
> > > >
> > >
> > > I do get a successful run, however, there are still no matched
pairs
> for
> > a
> > > different reason. The unmatched message_type problem would
still show
> up
> > > at some point, but the first problem I encountered when running
with
> the
> > > new config file is:
> > >
> > > > DEBUG 2: Processing APCP_02A2 versus APCP/Z0, for observation
type
> > > ADPSFC,
> > > > over region FULL, for interpolation method NEAREST(1), using 0
> matched
> > > > pairs.
> > > > DEBUG 3: Number of matched pairs = 0
> > > > DEBUG 3: Observations processed = 1
> > > > DEBUG 3: Rejected: SID exclusion = 0
> > > > DEBUG 3: Rejected: obs type = 0
> > > > DEBUG 3: Rejected: valid time = 0
> > > > DEBUG 3: Rejected: bad obs value = 0
> > > > DEBUG 3: Rejected: off the grid = 1
> > > > DEBUG 3: Rejected: topography = 0
> > > > DEBUG 3: Rejected: level mismatch = 0
> > > > DEBUG 3: Rejected: quality marker = 0
> > > > DEBUG 3: Rejected: message type = 0
> > > > DEBUG 3: Rejected: masking region = 0
> > > > DEBUG 3: Rejected: bad fcst value = 0
> > > > DEBUG 3: Rejected: duplicates = 0
> > > >
> > >
> > > We can see that there are no matched pairs and that they were
rejected
> > > because they are "off the grid".
> > >
> > > We can use plot_point_obs to visualize where the observation
lies:
> > >
> > > > /usr/local/met-8.1.1/bin/plot_point_obs BATG_ob1.nc
BATG_ob1.ps
> > -dotsize
> > > 10
> > > >
> > >
> > > I have attached BATG_ob1.ps for you. We can see the observation
is
> near
> > > Antarctica, and hopefully that is what you expect.
> > >
> > > We can use plot_data_plane to visualize the gridded model
output:
> > >
> > > > /usr/local/met-8.1.1/bin/plot_data_plane
> > > > APCP_wrfout_d05_2017120912_f007_f005.nc
> > > > APCP_wrfout_d05_2017120912_f007_f005.ps 'name="APCP_02";
> level="A2";'
> > > >
> > >
> > > However, there seems to be a problem with the projection
information. I
> > > have also attached APCP_wrfout_d05_2017120912_f007_f005.ps for
you.
> > > Looking in APCP_wrfout_d05_2017120912_f007_f005.nc, I can see
the
> > > following delta values:
> > >
> > > > :delta_lat = "0.005932 degrees" ;
> > > > :delta_lon = "0.418030 degrees" ;
> > > >
> > > which don't look like typical values. Usually they are the same
or
> close
> > > to the same.
> > >
> > > We can use the regrid_data_plane tool to regrid to NCEP grid 4
to see
> > where
> > > MET thinks the data is:
> > >
> > > > /usr/local/met-8.1.1/bin/regrid_data_plane
> > > > APCP_wrfout_d05_2017120912_f007_f005.nc G004 global.nc -field
> > > > 'name="APCP_02"; level="A2";'
> > > >
> > > We can view this data using ncview:
> > >
> > > > /usr/local/ncview/bin/ncview global.nc &
> > > >
> > >
> > > I took a screenshot of that image and attached it as
global.nc.png.
> You
> > > can see the data is smooshed with a chuck missing as well. This
seems
> > > problematic. If you'd like to pass on the GRIB data that you
used in
> > > running pcp_combine, we'd be happy to take a look at that.
> > >
> > > Also, since you're at NCAR, we could also get together to
discuss how
> > you'd
> > > like to use MET and how we can help you with it. My colleague,
John
> > Halley
> > > Gotway, helped me through this work. He is our MET expert.
Please let
> > us
> > > know if you'd like to arrange a time to meet with us.
> > >
> > > I hope this is helpful. Please let us know what other questions
you
> > have.
> > >
> > > Thanks,
> > > Julie
> > >
> > > On Wed, Dec 4, 2019 at 2:48 PM lave at ucar.edu via RT
<met_help at ucar.edu
> >
> > > wrote:
> > >
> > > >
> > > > Wed Dec 04 14:48:26 2019: Request 93338 was acted upon.
> > > > Transaction: Ticket created by lave at ucar.edu
> > > > Queue: met_help
> > > > Subject: point_stat "unrecognized GRIB1 field";
> > > > file_type="NETCDF_MET" in config file
> > > > Owner: Nobody
> > > > Requestors: lave at ucar.edu
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338
> > >
> > > >
> > > >
> > > > Good afternoon,
> > > >
> > > > I am having trouble running MET on netCDF data. I used
pcp_combine
> > > > -subtract to successfully create a netCDF file
> > > > (APCP_wrfout_d05_2017120912_f007_f005.nc) containing
accumulation
> data
> > > for
> > > > domain five of the AMPS evaluation area. I then used ascii2nc
to
> > create a
> > > > netCDF file (BATG_ob1.nc) containing an accumulation
observation.
> "psc"
> > > is
> > > > my configuration file, in which I specify
'field_type="NETCDF_MET"'
> in
> > > the
> > > > fcst dictionary. With 'name="APCP"', running point_stat yields
the
> > > > following:
> > > >
> > > > >point_stat APCP_wrfout_d05_2017120912_f007_f005.nc
BATG_ob1.nc psc
> > > >
> > > > DEBUG 1: Default Config File:
> > > > /usr/local/met-8.1.1/share/met/config/PointStatConfig_default
> > > >
> > > > DEBUG 1: User Config File: psc
> > > >
> > > > GSL_RNG_TYPE=mt19937
> > > >
> > > > GSL_RNG_SEED=18446744072084449373
> > > >
> > > > DEBUG 1: Forecast File:
APCP_wrfout_d05_2017120912_f007_f005.nc
> > > >
> > > > DEBUG 1: Observation File: BATG_ob1.nc
> > > >
> > > > DEBUG 2:
> > > >
> > > > DEBUG 2:
> > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > >
> > > > DEBUG 2:
> > > >
> > > > DEBUG 2: Reading data for APCPA2.
> > > >
> > > > WARNING:
> > > >
> > > > WARNING: process_fcst_climo_files() -> no fields matching
APCPA2
> found
> > in
> > > > file: APCP_wrfout_d05_2017120912_f007_f005.nc
> > > >
> > > > WARNING:
> > > >
> > > > ERROR :
> > > >
> > > > ERROR : process_fcst_climo_files() -> no requested forecast
data
> > found!
> > > > Exiting...
> > > >
> > > > ERROR :
> > > >
> > > > yar:/d1/lave/antarctica> vi psc
> > > >
> > > > yar:/d1/lave/antarctica> point_stat
APCP_wrfout_d05_2017120912_
> > > > f007_f005.nc
> > > > BATG_ob1.nc psc
> > > >
> > > > DEBUG 1: Default Config File:
> > > > /usr/local/met-8.1.1/share/met/config/PointStatConfig_default
> > > >
> > > > DEBUG 1: User Config File: psc
> > > >
> > > > ERROR :
> > > >
> > > > ERROR : VarInfoGrib::add_grib_code() -> unrecognized GRIB1
field
> > > > abbreviation 'APCP_02' for table version 2
> > > >
> > > > ERROR :
> > > >
> > > > So, following the answer given in [Met_help] [rt.rap.ucar.edu
> > #85872], I
> > > > adjusted my config file to include 'name="APCP_02"'. Running
> point_stat
> > > > then yielded:
> > > >
> > > > >point_stat APCP_wrfout_d05_2017120912_f007_f005.nc
BATG_ob1.nc psc
> > > >
> > > > DEBUG 1: Default Config File:
> > > > /usr/local/met-8.1.1/share/met/config/PointStatConfig_default
> > > >
> > > > DEBUG 1: User Config File: psc
> > > >
> > > > ERROR :
> > > >
> > > > ERROR : VarInfoGrib::add_grib_code() -> unrecognized GRIB1
field
> > > > abbreviation 'APCP_02' for table version 2
> > > > ERROR :
> > > >
> > > > What is going on here? It seems like point_stat is expecting a
GRIB1
> > > file,
> > > > though I specified the input to be a MET-generated netCDF
file. I
> > > uploaded
> > > > the three files to your ftp site.
> > > >
> > > > Sincerely,
> > > > Joshua Lave
> > > > --
> > > > Joshua Lave
> > > >
> > > > *Associate Scientist INational Center for Atmospheric
Research*
> > > > *(303) 497-8258*
> > > > *lave at ucar.edu <lave at ucar.edu>*
> > > >
> > > >
> > >
> > > --
> > > Julie Prestopnik
> > > Software Engineer
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > Phone: 303.497.8399
> > > Email: jpresto at ucar.edu
> > >
> > > My working day may not be your working day. Please do not feel
obliged
> > to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> > --
> > Joshua Lave
> >
> > *Associate Scientist INational Center for Atmospheric Research*
> > *(303) 497-8258*
> > *lave at ucar.edu <lave at ucar.edu>*
> >
> >
>
> --
> Julie Prestopnik
> Software Engineer
> National Center for Atmospheric Research
> Research Applications Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
--
Joshua Lave
*Associate Scientist INational Center for Atmospheric Research*
*(303) 497-8258*
*lave at ucar.edu <lave at ucar.edu>*
------------------------------------------------
Subject: point_stat "unrecognized GRIB1 field"; file_type="NETCDF_MET" in config file
From: John Halley Gotway
Time: Mon Dec 16 17:10:21 2019
Joshua,
OK, I'm looking at the issues we identified while meeting today.
MET is not interpreting the time info correctly...
Running pcp_combine on your subsetted APCP file yields weird timing
info:
*/usr/local/met-8.1.1/bin/pcp_combine -add APCP_d05_2018020200_f002.nc
'name="RAINNC"; level="(0,*,*)"; file_type=NETCDF_PINT;' met.nc
<http://met.nc>*
But running the same command on the wrfout files does not:
*/usr/local/met-8.1.1/bin/pcp_combine -add
wrfout_d05_2018020200_f002.nc
<http://wrfout_d05_2018020200_f002.nc> 'name="RAINNC";
level="(0,*,*)";
file_type=NETCDF_PINT;' met.nc <http://met.nc>*
So there is something in your subsetting process causing this
behavior.
Diffing the headers, I see that the "wrfout" file includes a variable
named
"Times" that the subset does not. Please try re-subsetting but
include:
*variables: char Times(Time, DateStrLen) ;*
Hopefully that'll yield better results. These weird timestamps also
led to
the runtime error we saw when trying to read the output of
pcp_combine.
Regarding not matching "APCP" observations, you were right. It was
looking
for obs named "APCP_02". I've attached an updated Point-Stat config
file
and my resulting log file, and listed below are the commands I used to
get
a sensible run.
Note that I used awk to append the accumulation hour to the APCP
observation variable name. And then I referenced that in the Point-
Stat
config file. I couldn't immediately figure out how to make that 2
digits
by adding a leading zero. So that's why the fcst references RAINNC_02
(which pcp_combine wrote out) and the obs references APCP_2.
Thanks,
John
# Run pcp_combine
/usr/local/met-8.1.1/bin/pcp_combine -subtract \
wrfout_d05_2018020200_f002.nc 'name="RAINNC"; level="(0,*,*)";
file_type=NETCDF_PINT;' \
wrfout_d05_2018020200_f000.nc 'name="RAINNC"; level="(0,*,*)";
file_type=NETCDF_PINT;' \
wrfout_d05_2018020200_f000_to_f002.nc
# Plot the result
/usr/local/met-8.1.1/bin/plot_data_plane \
wrfout_d05_2018020200_f000_to_f002.nc \
wrfout_d05_2018020200_f000_to_f002.ps \
'name="RAINNC_02"; level="(*,*)";'
# Prepare point observations
cat WLLY_ascii.csv | awk '{print
$1,$2,$3,$4,$5,$6,"APCP_"$8,$8,$9,$10,$11}' > WLLY_ascii_APCP.csv
/usr/local/met-8.1.1/bin/ascii2nc WLLY_ascii_APCP.csv
WLLY_ascii_APCP.nc
# Run point_stat
mkdir -p out
/usr/local/met-8.1.1/bin/point_stat \
wrfout_d05_2018020200_f000_to_f002.nc \
WLLY_ascii_APCP.nc \
PointStatConfig_APCP \
-outdir out -v 3 -log run_ps.log
On Mon, Dec 16, 2019 at 1:14 PM lave at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338 >
>
> Yes, thank you for checking in. I did receive the invitation and
accepted
> it. I'll see you at 3.
>
> Joshua
>
> On Mon, Dec 16, 2019 at 9:41 AM Julie Prestopnik via RT
<met_help at ucar.edu
> >
> wrote:
>
> > Hi Joshua.
> >
> > The delay in response is no problem. It's a busy time of year.
We're
> > happy to help you get started using MET.
> >
> > My colleague, John, set up a meeting for us today at 3pm in FL3-
1029. I
> > just wanted to make sure you received the invitation and check in
to see
> if
> > you'll be able to meet at that time? I'll be able to stay for
about the
> > first 20 minutes of the meeting.
> >
> > Thanks,
> > Julie
> >
> > On Fri, Dec 13, 2019 at 3:04 PM lave at ucar.edu via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338 >
> > >
> > > Julie,
> > >
> > > Sorry it has taken me so long to reply, but thank you for your
detailed
> > > response. This is my first time working with MET and I am doing
so at
> the
> > > suggestion of another scientist. Most of my struggles so far
have
> related
> > > to properly formatting my datasets. The GRIB data that I used in
> > > pcp_combine is a set that I converted from netCDF via wgrib and
cdo as
> I
> > > could not get pcp_combine to accept the nc files.
> > >
> > > I would like to accept your offer to meet with me to discuss MET
and
> how
> > I
> > > plan to use it. I am available most of next week (12/16-12:20),
> including
> > > all day Monday, Wednesday, and Friday.
> > >
> > > Joshua
> > >
> > > On Thu, Dec 5, 2019 at 2:54 PM Julie Prestopnik via RT <
> > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Joshua.
> > > >
> > > > I see that you are having some difficult running point_stat
with
> NetCDF
> > > > data.
> > > >
> > > > In your config file, I noticed that you had set the
"output_flag" for
> > > > "pstd"
> > > > to "BOTH", but had left all other values as "NONE". "pstd" is
for
> > > > probabilistic data. Is your intention to use this for
verification
> of
> > > > probabilistic data or for traditional unit verification?
> > > >
> > > > I made some changes to your config file and have attached a
new
> config
> > > > file. I'll outline the differences below:
> > > >
> > > > - I removed your entries for init_time, valid_time, and
lead_time.
> > > > Those fields are typically used for file that have many
lead times
> > > > specified. In this case, the file only has one time, so it
is not
> > > > necessary to specify those values. You only want to add
those
> > values
> > > if
> > > > they are needed. MET will be able to parse the file for
those
> > values
> > > in
> > > > this case.
> > > > - I removed "file_type = "NETCDF_MET";" as it is also not
> necessary
> > in
> > > > this case.
> > > > - I changed the values of the "output_flag" section,
assuming that
> > you
> > > > wanted to do traditional unit verification. So, I changed
"pstd"
> to
> > > > "NONE". I changed "ctc" (contingency table counts) and
"cts"
> > > > (statistics
> > > > on those contingency table counts) to "BOTH". The
"cat_thresh",
> or
> > > > categorical threshold, is used to define what constitutes
an
> "event"
> > > > when
> > > > computing a 2x2 contingency table. I also changed "cnt",
> continuous
> > > > statistics to "BOTH", because a lot of folks like to use
that
> > > > information
> > > > as well.
> > > > - I also changed the "obs = fcst;" field to give "obs" its
own
> > value.
> > > > In this case, obs=fcst won't work because "APCP_02" does
not exist
> > in
> > > > the
> > > > obs file, so MET would not find any matches. In order to
have MET
> > be
> > > > backward compatible, we are stuck with the GRIB naming
convention.
> > > The
> > > > obs file contains a GRIB value for "obs_gc". This triggers
logic
> in
> > > > MET to
> > > > do a table lookup, but MET would find "APCP" and not
"APCP_02". I
> > can
> > > > see
> > > > that the obs file BATG_ob1 was derived from using ascii2nc.
If
> > > > "APCP_02" had been listed in the ascii file as the value
for the
> > > > "Var_Name(or GRIB_Code)" field, and the NetCDF file then
contained
> > > > "APCP_02", you may have likely been able to use "obs =
fcst;".
> > > However,
> > > > that brings us to another issue in that the message type,
aka
> > > > "hdr_typ_table" (hdr_typ_table:long_name = "message type"
;),
> value
> > in
> > > > the
> > > > obs file is "default" so MET will not find any matches
there
> either,
> > > as
> > > > you have set "message_type = [ "ADPSFC" ];" in the config
file
> and
> > > > "default" wouldn't match "ADPSFC". You could edit the
ascii data
> > and
> > > > send
> > > > it through ascii2nc again. It is recommended to change
"default"
> to
> > > > something that is more meaningful for your data. So, MET
will not
> > > find
> > > > any
> > > > matched pairs until that issue is resolved.
> > > >
> > > > If I now run your data using point_stat with the changes and a
higher
> > > > verbosity level:
> > > >
> > > > > /usr/local/met-8.1.1/bin/point_stat
> > > > > APCP_wrfout_d05_2017120912_f007_f005.nc BATG_ob1.nc psc_new
-v 3
> > > > >
> > > >
> > > > I do get a successful run, however, there are still no matched
pairs
> > for
> > > a
> > > > different reason. The unmatched message_type problem would
still
> show
> > up
> > > > at some point, but the first problem I encountered when
running with
> > the
> > > > new config file is:
> > > >
> > > > > DEBUG 2: Processing APCP_02A2 versus APCP/Z0, for
observation type
> > > > ADPSFC,
> > > > > over region FULL, for interpolation method NEAREST(1), using
0
> > matched
> > > > > pairs.
> > > > > DEBUG 3: Number of matched pairs = 0
> > > > > DEBUG 3: Observations processed = 1
> > > > > DEBUG 3: Rejected: SID exclusion = 0
> > > > > DEBUG 3: Rejected: obs type = 0
> > > > > DEBUG 3: Rejected: valid time = 0
> > > > > DEBUG 3: Rejected: bad obs value = 0
> > > > > DEBUG 3: Rejected: off the grid = 1
> > > > > DEBUG 3: Rejected: topography = 0
> > > > > DEBUG 3: Rejected: level mismatch = 0
> > > > > DEBUG 3: Rejected: quality marker = 0
> > > > > DEBUG 3: Rejected: message type = 0
> > > > > DEBUG 3: Rejected: masking region = 0
> > > > > DEBUG 3: Rejected: bad fcst value = 0
> > > > > DEBUG 3: Rejected: duplicates = 0
> > > > >
> > > >
> > > > We can see that there are no matched pairs and that they were
> rejected
> > > > because they are "off the grid".
> > > >
> > > > We can use plot_point_obs to visualize where the observation
lies:
> > > >
> > > > > /usr/local/met-8.1.1/bin/plot_point_obs BATG_ob1.nc
BATG_ob1.ps
> > > -dotsize
> > > > 10
> > > > >
> > > >
> > > > I have attached BATG_ob1.ps for you. We can see the
observation is
> > near
> > > > Antarctica, and hopefully that is what you expect.
> > > >
> > > > We can use plot_data_plane to visualize the gridded model
output:
> > > >
> > > > > /usr/local/met-8.1.1/bin/plot_data_plane
> > > > > APCP_wrfout_d05_2017120912_f007_f005.nc
> > > > > APCP_wrfout_d05_2017120912_f007_f005.ps 'name="APCP_02";
> > level="A2";'
> > > > >
> > > >
> > > > However, there seems to be a problem with the projection
> information. I
> > > > have also attached APCP_wrfout_d05_2017120912_f007_f005.ps for
you.
> > > > Looking in APCP_wrfout_d05_2017120912_f007_f005.nc, I can see
the
> > > > following delta values:
> > > >
> > > > > :delta_lat = "0.005932 degrees" ;
> > > > > :delta_lon = "0.418030 degrees" ;
> > > > >
> > > > which don't look like typical values. Usually they are the
same or
> > close
> > > > to the same.
> > > >
> > > > We can use the regrid_data_plane tool to regrid to NCEP grid 4
to see
> > > where
> > > > MET thinks the data is:
> > > >
> > > > > /usr/local/met-8.1.1/bin/regrid_data_plane
> > > > > APCP_wrfout_d05_2017120912_f007_f005.nc G004 global.nc
-field
> > > > > 'name="APCP_02"; level="A2";'
> > > > >
> > > > We can view this data using ncview:
> > > >
> > > > > /usr/local/ncview/bin/ncview global.nc &
> > > > >
> > > >
> > > > I took a screenshot of that image and attached it as
global.nc.png.
> > You
> > > > can see the data is smooshed with a chuck missing as well.
This
> seems
> > > > problematic. If you'd like to pass on the GRIB data that you
used in
> > > > running pcp_combine, we'd be happy to take a look at that.
> > > >
> > > > Also, since you're at NCAR, we could also get together to
discuss how
> > > you'd
> > > > like to use MET and how we can help you with it. My
colleague, John
> > > Halley
> > > > Gotway, helped me through this work. He is our MET expert.
Please
> let
> > > us
> > > > know if you'd like to arrange a time to meet with us.
> > > >
> > > > I hope this is helpful. Please let us know what other
questions you
> > > have.
> > > >
> > > > Thanks,
> > > > Julie
> > > >
> > > > On Wed, Dec 4, 2019 at 2:48 PM lave at ucar.edu via RT <
> met_help at ucar.edu
> > >
> > > > wrote:
> > > >
> > > > >
> > > > > Wed Dec 04 14:48:26 2019: Request 93338 was acted upon.
> > > > > Transaction: Ticket created by lave at ucar.edu
> > > > > Queue: met_help
> > > > > Subject: point_stat "unrecognized GRIB1 field";
> > > > > file_type="NETCDF_MET" in config file
> > > > > Owner: Nobody
> > > > > Requestors: lave at ucar.edu
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338
> > > >
> > > > >
> > > > >
> > > > > Good afternoon,
> > > > >
> > > > > I am having trouble running MET on netCDF data. I used
pcp_combine
> > > > > -subtract to successfully create a netCDF file
> > > > > (APCP_wrfout_d05_2017120912_f007_f005.nc) containing
accumulation
> > data
> > > > for
> > > > > domain five of the AMPS evaluation area. I then used
ascii2nc to
> > > create a
> > > > > netCDF file (BATG_ob1.nc) containing an accumulation
observation.
> > "psc"
> > > > is
> > > > > my configuration file, in which I specify
'field_type="NETCDF_MET"'
> > in
> > > > the
> > > > > fcst dictionary. With 'name="APCP"', running point_stat
yields the
> > > > > following:
> > > > >
> > > > > >point_stat APCP_wrfout_d05_2017120912_f007_f005.nc
BATG_ob1.nc
> psc
> > > > >
> > > > > DEBUG 1: Default Config File:
> > > > > /usr/local/met-
8.1.1/share/met/config/PointStatConfig_default
> > > > >
> > > > > DEBUG 1: User Config File: psc
> > > > >
> > > > > GSL_RNG_TYPE=mt19937
> > > > >
> > > > > GSL_RNG_SEED=18446744072084449373
> > > > >
> > > > > DEBUG 1: Forecast File:
APCP_wrfout_d05_2017120912_f007_f005.nc
> > > > >
> > > > > DEBUG 1: Observation File: BATG_ob1.nc
> > > > >
> > > > > DEBUG 2:
> > > > >
> > > > > DEBUG 2:
> > > > >
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > >
> > > > > DEBUG 2:
> > > > >
> > > > > DEBUG 2: Reading data for APCPA2.
> > > > >
> > > > > WARNING:
> > > > >
> > > > > WARNING: process_fcst_climo_files() -> no fields matching
APCPA2
> > found
> > > in
> > > > > file: APCP_wrfout_d05_2017120912_f007_f005.nc
> > > > >
> > > > > WARNING:
> > > > >
> > > > > ERROR :
> > > > >
> > > > > ERROR : process_fcst_climo_files() -> no requested forecast
data
> > > found!
> > > > > Exiting...
> > > > >
> > > > > ERROR :
> > > > >
> > > > > yar:/d1/lave/antarctica> vi psc
> > > > >
> > > > > yar:/d1/lave/antarctica> point_stat
APCP_wrfout_d05_2017120912_
> > > > > f007_f005.nc
> > > > > BATG_ob1.nc psc
> > > > >
> > > > > DEBUG 1: Default Config File:
> > > > > /usr/local/met-
8.1.1/share/met/config/PointStatConfig_default
> > > > >
> > > > > DEBUG 1: User Config File: psc
> > > > >
> > > > > ERROR :
> > > > >
> > > > > ERROR : VarInfoGrib::add_grib_code() -> unrecognized GRIB1
field
> > > > > abbreviation 'APCP_02' for table version 2
> > > > >
> > > > > ERROR :
> > > > >
> > > > > So, following the answer given in [Met_help]
[rt.rap.ucar.edu
> > > #85872], I
> > > > > adjusted my config file to include 'name="APCP_02"'. Running
> > point_stat
> > > > > then yielded:
> > > > >
> > > > > >point_stat APCP_wrfout_d05_2017120912_f007_f005.nc
BATG_ob1.nc
> psc
> > > > >
> > > > > DEBUG 1: Default Config File:
> > > > > /usr/local/met-
8.1.1/share/met/config/PointStatConfig_default
> > > > >
> > > > > DEBUG 1: User Config File: psc
> > > > >
> > > > > ERROR :
> > > > >
> > > > > ERROR : VarInfoGrib::add_grib_code() -> unrecognized GRIB1
field
> > > > > abbreviation 'APCP_02' for table version 2
> > > > > ERROR :
> > > > >
> > > > > What is going on here? It seems like point_stat is expecting
a
> GRIB1
> > > > file,
> > > > > though I specified the input to be a MET-generated netCDF
file. I
> > > > uploaded
> > > > > the three files to your ftp site.
> > > > >
> > > > > Sincerely,
> > > > > Joshua Lave
> > > > > --
> > > > > Joshua Lave
> > > > >
> > > > > *Associate Scientist INational Center for Atmospheric
Research*
> > > > > *(303) 497-8258*
> > > > > *lave at ucar.edu <lave at ucar.edu>*
> > > > >
> > > > >
> > > >
> > > > --
> > > > Julie Prestopnik
> > > > Software Engineer
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > Phone: 303.497.8399
> > > > Email: jpresto at ucar.edu
> > > >
> > > > My working day may not be your working day. Please do not
feel
> obliged
> > > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > > --
> > > Joshua Lave
> > >
> > > *Associate Scientist INational Center for Atmospheric Research*
> > > *(303) 497-8258*
> > > *lave at ucar.edu <lave at ucar.edu>*
> > >
> > >
> >
> > --
> > Julie Prestopnik
> > Software Engineer
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > Phone: 303.497.8399
> > Email: jpresto at ucar.edu
> >
> > My working day may not be your working day. Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
> Joshua Lave
>
> *Associate Scientist INational Center for Atmospheric Research*
> *(303) 497-8258*
> *lave at ucar.edu <lave at ucar.edu>*
>
>
------------------------------------------------
Subject: point_stat "unrecognized GRIB1 field"; file_type="NETCDF_MET" in config file
From: lave at ucar.edu
Time: Mon Dec 16 18:09:31 2019
John,
Including 'Times' in the subsetted file takes care of the valid time
issue.
However, I still run into the following error:
point_stat diff.nc WLLY_ascii_RAIN.nc PointStatConfig_APCP -v 3
-obs_valid_beg 20180202_02 -obs_valid_end 20180202_02
DEBUG 1: Default Config File:
/usr/local/met-8.1.1/share/met/config/PointStatConfig_default
DEBUG 1: User Config File: PointStatConfig_APCP
ERROR :
ERROR : MetNcPinterpDataFile::open(const char *) -> unable to open
NetCDF
file "diff.nc"
ERROR :
ERROR :
ERROR : Met2dDataFileFactory::new_met_2d_data_file() -> error opening
file
"diff.nc"
ERROR :
What could be causing this?
Joshua
On Mon, Dec 16, 2019 at 5:10 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Joshua,
>
> OK, I'm looking at the issues we identified while meeting today.
>
> MET is not interpreting the time info correctly...
>
> Running pcp_combine on your subsetted APCP file yields weird timing
info:
> */usr/local/met-8.1.1/bin/pcp_combine -add
APCP_d05_2018020200_f002.nc
> 'name="RAINNC"; level="(0,*,*)"; file_type=NETCDF_PINT;' met.nc
> <http://met.nc>*
>
> But running the same command on the wrfout files does not:
> */usr/local/met-8.1.1/bin/pcp_combine -add
wrfout_d05_2018020200_f002.nc
> <http://wrfout_d05_2018020200_f002.nc> 'name="RAINNC";
level="(0,*,*)";
> file_type=NETCDF_PINT;' met.nc <http://met.nc>*
>
> So there is something in your subsetting process causing this
behavior.
> Diffing the headers, I see that the "wrfout" file includes a
variable named
> "Times" that the subset does not. Please try re-subsetting but
include:
>
>
> *variables: char Times(Time, DateStrLen) ;*
>
> Hopefully that'll yield better results. These weird timestamps also
led to
> the runtime error we saw when trying to read the output of
pcp_combine.
>
> Regarding not matching "APCP" observations, you were right. It was
looking
> for obs named "APCP_02". I've attached an updated Point-Stat config
file
> and my resulting log file, and listed below are the commands I used
to get
> a sensible run.
>
> Note that I used awk to append the accumulation hour to the APCP
> observation variable name. And then I referenced that in the Point-
Stat
> config file. I couldn't immediately figure out how to make that 2
digits
> by adding a leading zero. So that's why the fcst references
RAINNC_02
> (which pcp_combine wrote out) and the obs references APCP_2.
>
> Thanks,
> John
>
> # Run pcp_combine
> /usr/local/met-8.1.1/bin/pcp_combine -subtract \
> wrfout_d05_2018020200_f002.nc 'name="RAINNC"; level="(0,*,*)";
> file_type=NETCDF_PINT;' \
> wrfout_d05_2018020200_f000.nc 'name="RAINNC"; level="(0,*,*)";
> file_type=NETCDF_PINT;' \
> wrfout_d05_2018020200_f000_to_f002.nc
>
> # Plot the result
> /usr/local/met-8.1.1/bin/plot_data_plane \
> wrfout_d05_2018020200_f000_to_f002.nc \
> wrfout_d05_2018020200_f000_to_f002.ps \
> 'name="RAINNC_02"; level="(*,*)";'
>
> # Prepare point observations
> cat WLLY_ascii.csv | awk '{print
> $1,$2,$3,$4,$5,$6,"APCP_"$8,$8,$9,$10,$11}' > WLLY_ascii_APCP.csv
> /usr/local/met-8.1.1/bin/ascii2nc WLLY_ascii_APCP.csv
WLLY_ascii_APCP.nc
>
> # Run point_stat
> mkdir -p out
> /usr/local/met-8.1.1/bin/point_stat \
> wrfout_d05_2018020200_f000_to_f002.nc \
> WLLY_ascii_APCP.nc \
> PointStatConfig_APCP \
> -outdir out -v 3 -log run_ps.log
>
>
>
>
>
>
> On Mon, Dec 16, 2019 at 1:14 PM lave at ucar.edu via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338 >
> >
> > Yes, thank you for checking in. I did receive the invitation and
accepted
> > it. I'll see you at 3.
> >
> > Joshua
> >
> > On Mon, Dec 16, 2019 at 9:41 AM Julie Prestopnik via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > > Hi Joshua.
> > >
> > > The delay in response is no problem. It's a busy time of year.
We're
> > > happy to help you get started using MET.
> > >
> > > My colleague, John, set up a meeting for us today at 3pm in FL3-
1029.
> I
> > > just wanted to make sure you received the invitation and check
in to
> see
> > if
> > > you'll be able to meet at that time? I'll be able to stay for
about
> the
> > > first 20 minutes of the meeting.
> > >
> > > Thanks,
> > > Julie
> > >
> > > On Fri, Dec 13, 2019 at 3:04 PM lave at ucar.edu via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338
>
> > > >
> > > > Julie,
> > > >
> > > > Sorry it has taken me so long to reply, but thank you for your
> detailed
> > > > response. This is my first time working with MET and I am
doing so at
> > the
> > > > suggestion of another scientist. Most of my struggles so far
have
> > related
> > > > to properly formatting my datasets. The GRIB data that I used
in
> > > > pcp_combine is a set that I converted from netCDF via wgrib
and cdo
> as
> > I
> > > > could not get pcp_combine to accept the nc files.
> > > >
> > > > I would like to accept your offer to meet with me to discuss
MET and
> > how
> > > I
> > > > plan to use it. I am available most of next week (12/16-
12:20),
> > including
> > > > all day Monday, Wednesday, and Friday.
> > > >
> > > > Joshua
> > > >
> > > > On Thu, Dec 5, 2019 at 2:54 PM Julie Prestopnik via RT <
> > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Joshua.
> > > > >
> > > > > I see that you are having some difficult running point_stat
with
> > NetCDF
> > > > > data.
> > > > >
> > > > > In your config file, I noticed that you had set the
"output_flag"
> for
> > > > > "pstd"
> > > > > to "BOTH", but had left all other values as "NONE". "pstd"
is for
> > > > > probabilistic data. Is your intention to use this for
verification
> > of
> > > > > probabilistic data or for traditional unit verification?
> > > > >
> > > > > I made some changes to your config file and have attached a
new
> > config
> > > > > file. I'll outline the differences below:
> > > > >
> > > > > - I removed your entries for init_time, valid_time, and
> lead_time.
> > > > > Those fields are typically used for file that have many
lead
> times
> > > > > specified. In this case, the file only has one time, so
it is
> not
> > > > > necessary to specify those values. You only want to add
those
> > > values
> > > > if
> > > > > they are needed. MET will be able to parse the file for
those
> > > values
> > > > in
> > > > > this case.
> > > > > - I removed "file_type = "NETCDF_MET";" as it is also not
> > necessary
> > > in
> > > > > this case.
> > > > > - I changed the values of the "output_flag" section,
assuming
> that
> > > you
> > > > > wanted to do traditional unit verification. So, I
changed
> "pstd"
> > to
> > > > > "NONE". I changed "ctc" (contingency table counts) and
"cts"
> > > > > (statistics
> > > > > on those contingency table counts) to "BOTH". The
"cat_thresh",
> > or
> > > > > categorical threshold, is used to define what constitutes
an
> > "event"
> > > > > when
> > > > > computing a 2x2 contingency table. I also changed "cnt",
> > continuous
> > > > > statistics to "BOTH", because a lot of folks like to use
that
> > > > > information
> > > > > as well.
> > > > > - I also changed the "obs = fcst;" field to give "obs"
its own
> > > value.
> > > > > In this case, obs=fcst won't work because "APCP_02" does
not
> exist
> > > in
> > > > > the
> > > > > obs file, so MET would not find any matches. In order to
have
> MET
> > > be
> > > > > backward compatible, we are stuck with the GRIB naming
> convention.
> > > > The
> > > > > obs file contains a GRIB value for "obs_gc". This
triggers
> logic
> > in
> > > > > MET to
> > > > > do a table lookup, but MET would find "APCP" and not
> "APCP_02". I
> > > can
> > > > > see
> > > > > that the obs file BATG_ob1 was derived from using
ascii2nc. If
> > > > > "APCP_02" had been listed in the ascii file as the value
for the
> > > > > "Var_Name(or GRIB_Code)" field, and the NetCDF file then
> contained
> > > > > "APCP_02", you may have likely been able to use "obs =
fcst;".
> > > > However,
> > > > > that brings us to another issue in that the message type,
aka
> > > > > "hdr_typ_table" (hdr_typ_table:long_name = "message type"
;),
> > value
> > > in
> > > > > the
> > > > > obs file is "default" so MET will not find any matches
there
> > either,
> > > > as
> > > > > you have set "message_type = [ "ADPSFC" ];" in the
config file
> > and
> > > > > "default" wouldn't match "ADPSFC". You could edit the
ascii
> data
> > > and
> > > > > send
> > > > > it through ascii2nc again. It is recommended to change
> "default"
> > to
> > > > > something that is more meaningful for your data. So, MET
will
> not
> > > > find
> > > > > any
> > > > > matched pairs until that issue is resolved.
> > > > >
> > > > > If I now run your data using point_stat with the changes and
a
> higher
> > > > > verbosity level:
> > > > >
> > > > > > /usr/local/met-8.1.1/bin/point_stat
> > > > > > APCP_wrfout_d05_2017120912_f007_f005.nc BATG_ob1.nc
psc_new -v 3
> > > > > >
> > > > >
> > > > > I do get a successful run, however, there are still no
matched
> pairs
> > > for
> > > > a
> > > > > different reason. The unmatched message_type problem would
still
> > show
> > > up
> > > > > at some point, but the first problem I encountered when
running
> with
> > > the
> > > > > new config file is:
> > > > >
> > > > > > DEBUG 2: Processing APCP_02A2 versus APCP/Z0, for
observation
> type
> > > > > ADPSFC,
> > > > > > over region FULL, for interpolation method NEAREST(1),
using 0
> > > matched
> > > > > > pairs.
> > > > > > DEBUG 3: Number of matched pairs = 0
> > > > > > DEBUG 3: Observations processed = 1
> > > > > > DEBUG 3: Rejected: SID exclusion = 0
> > > > > > DEBUG 3: Rejected: obs type = 0
> > > > > > DEBUG 3: Rejected: valid time = 0
> > > > > > DEBUG 3: Rejected: bad obs value = 0
> > > > > > DEBUG 3: Rejected: off the grid = 1
> > > > > > DEBUG 3: Rejected: topography = 0
> > > > > > DEBUG 3: Rejected: level mismatch = 0
> > > > > > DEBUG 3: Rejected: quality marker = 0
> > > > > > DEBUG 3: Rejected: message type = 0
> > > > > > DEBUG 3: Rejected: masking region = 0
> > > > > > DEBUG 3: Rejected: bad fcst value = 0
> > > > > > DEBUG 3: Rejected: duplicates = 0
> > > > > >
> > > > >
> > > > > We can see that there are no matched pairs and that they
were
> > rejected
> > > > > because they are "off the grid".
> > > > >
> > > > > We can use plot_point_obs to visualize where the observation
lies:
> > > > >
> > > > > > /usr/local/met-8.1.1/bin/plot_point_obs BATG_ob1.nc
BATG_ob1.ps
> > > > -dotsize
> > > > > 10
> > > > > >
> > > > >
> > > > > I have attached BATG_ob1.ps for you. We can see the
observation is
> > > near
> > > > > Antarctica, and hopefully that is what you expect.
> > > > >
> > > > > We can use plot_data_plane to visualize the gridded model
output:
> > > > >
> > > > > > /usr/local/met-8.1.1/bin/plot_data_plane
> > > > > > APCP_wrfout_d05_2017120912_f007_f005.nc
> > > > > > APCP_wrfout_d05_2017120912_f007_f005.ps 'name="APCP_02";
> > > level="A2";'
> > > > > >
> > > > >
> > > > > However, there seems to be a problem with the projection
> > information. I
> > > > > have also attached APCP_wrfout_d05_2017120912_f007_f005.ps
for
> you.
> > > > > Looking in APCP_wrfout_d05_2017120912_f007_f005.nc, I can
see the
> > > > > following delta values:
> > > > >
> > > > > > :delta_lat = "0.005932 degrees" ;
> > > > > > :delta_lon = "0.418030 degrees" ;
> > > > > >
> > > > > which don't look like typical values. Usually they are the
same or
> > > close
> > > > > to the same.
> > > > >
> > > > > We can use the regrid_data_plane tool to regrid to NCEP grid
4 to
> see
> > > > where
> > > > > MET thinks the data is:
> > > > >
> > > > > > /usr/local/met-8.1.1/bin/regrid_data_plane
> > > > > > APCP_wrfout_d05_2017120912_f007_f005.nc G004 global.nc
-field
> > > > > > 'name="APCP_02"; level="A2";'
> > > > > >
> > > > > We can view this data using ncview:
> > > > >
> > > > > > /usr/local/ncview/bin/ncview global.nc &
> > > > > >
> > > > >
> > > > > I took a screenshot of that image and attached it as
global.nc.png.
> > > You
> > > > > can see the data is smooshed with a chuck missing as well.
This
> > seems
> > > > > problematic. If you'd like to pass on the GRIB data that
you used
> in
> > > > > running pcp_combine, we'd be happy to take a look at that.
> > > > >
> > > > > Also, since you're at NCAR, we could also get together to
discuss
> how
> > > > you'd
> > > > > like to use MET and how we can help you with it. My
colleague,
> John
> > > > Halley
> > > > > Gotway, helped me through this work. He is our MET expert.
Please
> > let
> > > > us
> > > > > know if you'd like to arrange a time to meet with us.
> > > > >
> > > > > I hope this is helpful. Please let us know what other
questions
> you
> > > > have.
> > > > >
> > > > > Thanks,
> > > > > Julie
> > > > >
> > > > > On Wed, Dec 4, 2019 at 2:48 PM lave at ucar.edu via RT <
> > met_help at ucar.edu
> > > >
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > Wed Dec 04 14:48:26 2019: Request 93338 was acted upon.
> > > > > > Transaction: Ticket created by lave at ucar.edu
> > > > > > Queue: met_help
> > > > > > Subject: point_stat "unrecognized GRIB1 field";
> > > > > > file_type="NETCDF_MET" in config file
> > > > > > Owner: Nobody
> > > > > > Requestors: lave at ucar.edu
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338
> > > > >
> > > > > >
> > > > > >
> > > > > > Good afternoon,
> > > > > >
> > > > > > I am having trouble running MET on netCDF data. I used
> pcp_combine
> > > > > > -subtract to successfully create a netCDF file
> > > > > > (APCP_wrfout_d05_2017120912_f007_f005.nc) containing
> accumulation
> > > data
> > > > > for
> > > > > > domain five of the AMPS evaluation area. I then used
ascii2nc to
> > > > create a
> > > > > > netCDF file (BATG_ob1.nc) containing an accumulation
observation.
> > > "psc"
> > > > > is
> > > > > > my configuration file, in which I specify
> 'field_type="NETCDF_MET"'
> > > in
> > > > > the
> > > > > > fcst dictionary. With 'name="APCP"', running point_stat
yields
> the
> > > > > > following:
> > > > > >
> > > > > > >point_stat APCP_wrfout_d05_2017120912_f007_f005.nc
BATG_ob1.nc
> > psc
> > > > > >
> > > > > > DEBUG 1: Default Config File:
> > > > > > /usr/local/met-
8.1.1/share/met/config/PointStatConfig_default
> > > > > >
> > > > > > DEBUG 1: User Config File: psc
> > > > > >
> > > > > > GSL_RNG_TYPE=mt19937
> > > > > >
> > > > > > GSL_RNG_SEED=18446744072084449373
> > > > > >
> > > > > > DEBUG 1: Forecast File:
APCP_wrfout_d05_2017120912_f007_f005.nc
> > > > > >
> > > > > > DEBUG 1: Observation File: BATG_ob1.nc
> > > > > >
> > > > > > DEBUG 2:
> > > > > >
> > > > > > DEBUG 2:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > > >
> > > > > > DEBUG 2:
> > > > > >
> > > > > > DEBUG 2: Reading data for APCPA2.
> > > > > >
> > > > > > WARNING:
> > > > > >
> > > > > > WARNING: process_fcst_climo_files() -> no fields matching
APCPA2
> > > found
> > > > in
> > > > > > file: APCP_wrfout_d05_2017120912_f007_f005.nc
> > > > > >
> > > > > > WARNING:
> > > > > >
> > > > > > ERROR :
> > > > > >
> > > > > > ERROR : process_fcst_climo_files() -> no requested
forecast data
> > > > found!
> > > > > > Exiting...
> > > > > >
> > > > > > ERROR :
> > > > > >
> > > > > > yar:/d1/lave/antarctica> vi psc
> > > > > >
> > > > > > yar:/d1/lave/antarctica> point_stat
APCP_wrfout_d05_2017120912_
> > > > > > f007_f005.nc
> > > > > > BATG_ob1.nc psc
> > > > > >
> > > > > > DEBUG 1: Default Config File:
> > > > > > /usr/local/met-
8.1.1/share/met/config/PointStatConfig_default
> > > > > >
> > > > > > DEBUG 1: User Config File: psc
> > > > > >
> > > > > > ERROR :
> > > > > >
> > > > > > ERROR : VarInfoGrib::add_grib_code() -> unrecognized
GRIB1 field
> > > > > > abbreviation 'APCP_02' for table version 2
> > > > > >
> > > > > > ERROR :
> > > > > >
> > > > > > So, following the answer given in [Met_help]
[rt.rap.ucar.edu
> > > > #85872], I
> > > > > > adjusted my config file to include 'name="APCP_02"'.
Running
> > > point_stat
> > > > > > then yielded:
> > > > > >
> > > > > > >point_stat APCP_wrfout_d05_2017120912_f007_f005.nc
BATG_ob1.nc
> > psc
> > > > > >
> > > > > > DEBUG 1: Default Config File:
> > > > > > /usr/local/met-
8.1.1/share/met/config/PointStatConfig_default
> > > > > >
> > > > > > DEBUG 1: User Config File: psc
> > > > > >
> > > > > > ERROR :
> > > > > >
> > > > > > ERROR : VarInfoGrib::add_grib_code() -> unrecognized
GRIB1 field
> > > > > > abbreviation 'APCP_02' for table version 2
> > > > > > ERROR :
> > > > > >
> > > > > > What is going on here? It seems like point_stat is
expecting a
> > GRIB1
> > > > > file,
> > > > > > though I specified the input to be a MET-generated netCDF
file. I
> > > > > uploaded
> > > > > > the three files to your ftp site.
> > > > > >
> > > > > > Sincerely,
> > > > > > Joshua Lave
> > > > > > --
> > > > > > Joshua Lave
> > > > > >
> > > > > > *Associate Scientist INational Center for Atmospheric
Research*
> > > > > > *(303) 497-8258*
> > > > > > *lave at ucar.edu <lave at ucar.edu>*
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Julie Prestopnik
> > > > > Software Engineer
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > Phone: 303.497.8399
> > > > > Email: jpresto at ucar.edu
> > > > >
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > > --
> > > > Joshua Lave
> > > >
> > > > *Associate Scientist INational Center for Atmospheric
Research*
> > > > *(303) 497-8258*
> > > > *lave at ucar.edu <lave at ucar.edu>*
> > > >
> > > >
> > >
> > > --
> > > Julie Prestopnik
> > > Software Engineer
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > Phone: 303.497.8399
> > > Email: jpresto at ucar.edu
> > >
> > > My working day may not be your working day. Please do not feel
obliged
> > to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> > --
> > Joshua Lave
> >
> > *Associate Scientist INational Center for Atmospheric Research*
> > *(303) 497-8258*
> > *lave at ucar.edu <lave at ucar.edu>*
> >
> >
>
>
--
Joshua Lave
*Associate Scientist INational Center for Atmospheric Research*
*(303) 497-8258*
*lave at ucar.edu <lave at ucar.edu>*
------------------------------------------------
Subject: point_stat "unrecognized GRIB1 field"; file_type="NETCDF_MET" in config file
From: John Halley Gotway
Time: Mon Dec 16 20:04:31 2019
Joshua,
Yes, that’s why I sent the updated Point-Stat config file. You’re no
longer using a wrfout formatted file. You’re using the output from
pcp_combine, which follows the format used by MET NetCDF files. So
you
need to remove the file_type = NETCDF_PINT; option from the point-stat
config file.
John
On Mon, Dec 16, 2019 at 6:09 PM lave at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338 >
>
> John,
>
> Including 'Times' in the subsetted file takes care of the valid time
issue.
> However, I still run into the following error:
>
> point_stat diff.nc WLLY_ascii_RAIN.nc PointStatConfig_APCP -v 3
> -obs_valid_beg 20180202_02 -obs_valid_end 20180202_02
>
>
> DEBUG 1: Default Config File:
> /usr/local/met-8.1.1/share/met/config/PointStatConfig_default
>
> DEBUG 1: User Config File: PointStatConfig_APCP
>
> ERROR :
>
> ERROR : MetNcPinterpDataFile::open(const char *) -> unable to open
NetCDF
> file "diff.nc"
>
> ERROR :
>
> ERROR :
>
> ERROR : Met2dDataFileFactory::new_met_2d_data_file() -> error
opening file
> "diff.nc"
>
> ERROR :
>
> What could be causing this?
>
> Joshua
>
> On Mon, Dec 16, 2019 at 5:10 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Joshua,
> >
> > OK, I'm looking at the issues we identified while meeting today.
> >
> > MET is not interpreting the time info correctly...
> >
> > Running pcp_combine on your subsetted APCP file yields weird
timing info:
> > */usr/local/met-8.1.1/bin/pcp_combine -add
APCP_d05_2018020200_f002.nc
> > 'name="RAINNC"; level="(0,*,*)"; file_type=NETCDF_PINT;' met.nc
> > <http://met.nc>*
> >
> > But running the same command on the wrfout files does not:
> > */usr/local/met-8.1.1/bin/pcp_combine -add
wrfout_d05_2018020200_f002.nc
> > <http://wrfout_d05_2018020200_f002.nc> 'name="RAINNC";
level="(0,*,*)";
> > file_type=NETCDF_PINT;' met.nc <http://met.nc>*
> >
> > So there is something in your subsetting process causing this
behavior.
> > Diffing the headers, I see that the "wrfout" file includes a
variable
> named
> > "Times" that the subset does not. Please try re-subsetting but
include:
> >
> >
> > *variables: char Times(Time, DateStrLen) ;*
> >
> > Hopefully that'll yield better results. These weird timestamps
also led
> to
> > the runtime error we saw when trying to read the output of
pcp_combine.
> >
> > Regarding not matching "APCP" observations, you were right. It
was
> looking
> > for obs named "APCP_02". I've attached an updated Point-Stat
config file
> > and my resulting log file, and listed below are the commands I
used to
> get
> > a sensible run.
> >
> > Note that I used awk to append the accumulation hour to the APCP
> > observation variable name. And then I referenced that in the
Point-Stat
> > config file. I couldn't immediately figure out how to make that 2
digits
> > by adding a leading zero. So that's why the fcst references
RAINNC_02
> > (which pcp_combine wrote out) and the obs references APCP_2.
> >
> > Thanks,
> > John
> >
> > # Run pcp_combine
> > /usr/local/met-8.1.1/bin/pcp_combine -subtract \
> > wrfout_d05_2018020200_f002.nc 'name="RAINNC"; level="(0,*,*)";
> > file_type=NETCDF_PINT;' \
> > wrfout_d05_2018020200_f000.nc 'name="RAINNC"; level="(0,*,*)";
> > file_type=NETCDF_PINT;' \
> > wrfout_d05_2018020200_f000_to_f002.nc
> >
> > # Plot the result
> > /usr/local/met-8.1.1/bin/plot_data_plane \
> > wrfout_d05_2018020200_f000_to_f002.nc \
> > wrfout_d05_2018020200_f000_to_f002.ps \
> > 'name="RAINNC_02"; level="(*,*)";'
> >
> > # Prepare point observations
> > cat WLLY_ascii.csv | awk '{print
> > $1,$2,$3,$4,$5,$6,"APCP_"$8,$8,$9,$10,$11}' > WLLY_ascii_APCP.csv
> > /usr/local/met-8.1.1/bin/ascii2nc WLLY_ascii_APCP.csv
WLLY_ascii_APCP.nc
> >
> > # Run point_stat
> > mkdir -p out
> > /usr/local/met-8.1.1/bin/point_stat \
> > wrfout_d05_2018020200_f000_to_f002.nc \
> > WLLY_ascii_APCP.nc \
> > PointStatConfig_APCP \
> > -outdir out -v 3 -log run_ps.log
> >
> >
> >
> >
> >
> >
> > On Mon, Dec 16, 2019 at 1:14 PM lave at ucar.edu via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338 >
> > >
> > > Yes, thank you for checking in. I did receive the invitation and
> accepted
> > > it. I'll see you at 3.
> > >
> > > Joshua
> > >
> > > On Mon, Dec 16, 2019 at 9:41 AM Julie Prestopnik via RT <
> > met_help at ucar.edu
> > > >
> > > wrote:
> > >
> > > > Hi Joshua.
> > > >
> > > > The delay in response is no problem. It's a busy time of
year.
> We're
> > > > happy to help you get started using MET.
> > > >
> > > > My colleague, John, set up a meeting for us today at 3pm in
FL3-1029.
> > I
> > > > just wanted to make sure you received the invitation and check
in to
> > see
> > > if
> > > > you'll be able to meet at that time? I'll be able to stay for
about
> > the
> > > > first 20 minutes of the meeting.
> > > >
> > > > Thanks,
> > > > Julie
> > > >
> > > > On Fri, Dec 13, 2019 at 3:04 PM lave at ucar.edu via RT <
> > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338 >
> > > > >
> > > > > Julie,
> > > > >
> > > > > Sorry it has taken me so long to reply, but thank you for
your
> > detailed
> > > > > response. This is my first time working with MET and I am
doing so
> at
> > > the
> > > > > suggestion of another scientist. Most of my struggles so far
have
> > > related
> > > > > to properly formatting my datasets. The GRIB data that I
used in
> > > > > pcp_combine is a set that I converted from netCDF via wgrib
and cdo
> > as
> > > I
> > > > > could not get pcp_combine to accept the nc files.
> > > > >
> > > > > I would like to accept your offer to meet with me to discuss
MET
> and
> > > how
> > > > I
> > > > > plan to use it. I am available most of next week (12/16-
12:20),
> > > including
> > > > > all day Monday, Wednesday, and Friday.
> > > > >
> > > > > Joshua
> > > > >
> > > > > On Thu, Dec 5, 2019 at 2:54 PM Julie Prestopnik via RT <
> > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Hi Joshua.
> > > > > >
> > > > > > I see that you are having some difficult running
point_stat with
> > > NetCDF
> > > > > > data.
> > > > > >
> > > > > > In your config file, I noticed that you had set the
"output_flag"
> > for
> > > > > > "pstd"
> > > > > > to "BOTH", but had left all other values as "NONE". "pstd"
is for
> > > > > > probabilistic data. Is your intention to use this for
> verification
> > > of
> > > > > > probabilistic data or for traditional unit verification?
> > > > > >
> > > > > > I made some changes to your config file and have attached
a new
> > > config
> > > > > > file. I'll outline the differences below:
> > > > > >
> > > > > > - I removed your entries for init_time, valid_time, and
> > lead_time.
> > > > > > Those fields are typically used for file that have many
lead
> > times
> > > > > > specified. In this case, the file only has one time,
so it is
> > not
> > > > > > necessary to specify those values. You only want to
add those
> > > > values
> > > > > if
> > > > > > they are needed. MET will be able to parse the file
for those
> > > > values
> > > > > in
> > > > > > this case.
> > > > > > - I removed "file_type = "NETCDF_MET";" as it is also
not
> > > necessary
> > > > in
> > > > > > this case.
> > > > > > - I changed the values of the "output_flag" section,
assuming
> > that
> > > > you
> > > > > > wanted to do traditional unit verification. So, I
changed
> > "pstd"
> > > to
> > > > > > "NONE". I changed "ctc" (contingency table counts) and
"cts"
> > > > > > (statistics
> > > > > > on those contingency table counts) to "BOTH". The
> "cat_thresh",
> > > or
> > > > > > categorical threshold, is used to define what
constitutes an
> > > "event"
> > > > > > when
> > > > > > computing a 2x2 contingency table. I also changed
"cnt",
> > > continuous
> > > > > > statistics to "BOTH", because a lot of folks like to
use that
> > > > > > information
> > > > > > as well.
> > > > > > - I also changed the "obs = fcst;" field to give "obs"
its own
> > > > value.
> > > > > > In this case, obs=fcst won't work because "APCP_02"
does not
> > exist
> > > > in
> > > > > > the
> > > > > > obs file, so MET would not find any matches. In order
to have
> > MET
> > > > be
> > > > > > backward compatible, we are stuck with the GRIB naming
> > convention.
> > > > > The
> > > > > > obs file contains a GRIB value for "obs_gc". This
triggers
> > logic
> > > in
> > > > > > MET to
> > > > > > do a table lookup, but MET would find "APCP" and not
> > "APCP_02". I
> > > > can
> > > > > > see
> > > > > > that the obs file BATG_ob1 was derived from using
ascii2nc.
> If
> > > > > > "APCP_02" had been listed in the ascii file as the
value for
> the
> > > > > > "Var_Name(or GRIB_Code)" field, and the NetCDF file
then
> > contained
> > > > > > "APCP_02", you may have likely been able to use "obs =
fcst;".
> > > > > However,
> > > > > > that brings us to another issue in that the message
type, aka
> > > > > > "hdr_typ_table" (hdr_typ_table:long_name = "message
type" ;),
> > > value
> > > > in
> > > > > > the
> > > > > > obs file is "default" so MET will not find any matches
there
> > > either,
> > > > > as
> > > > > > you have set "message_type = [ "ADPSFC" ];" in the
config
> file
> > > and
> > > > > > "default" wouldn't match "ADPSFC". You could edit the
ascii
> > data
> > > > and
> > > > > > send
> > > > > > it through ascii2nc again. It is recommended to change
> > "default"
> > > to
> > > > > > something that is more meaningful for your data. So,
MET will
> > not
> > > > > find
> > > > > > any
> > > > > > matched pairs until that issue is resolved.
> > > > > >
> > > > > > If I now run your data using point_stat with the changes
and a
> > higher
> > > > > > verbosity level:
> > > > > >
> > > > > > > /usr/local/met-8.1.1/bin/point_stat
> > > > > > > APCP_wrfout_d05_2017120912_f007_f005.nc BATG_ob1.nc
psc_new
> -v 3
> > > > > > >
> > > > > >
> > > > > > I do get a successful run, however, there are still no
matched
> > pairs
> > > > for
> > > > > a
> > > > > > different reason. The unmatched message_type problem
would still
> > > show
> > > > up
> > > > > > at some point, but the first problem I encountered when
running
> > with
> > > > the
> > > > > > new config file is:
> > > > > >
> > > > > > > DEBUG 2: Processing APCP_02A2 versus APCP/Z0, for
observation
> > type
> > > > > > ADPSFC,
> > > > > > > over region FULL, for interpolation method NEAREST(1),
using 0
> > > > matched
> > > > > > > pairs.
> > > > > > > DEBUG 3: Number of matched pairs = 0
> > > > > > > DEBUG 3: Observations processed = 1
> > > > > > > DEBUG 3: Rejected: SID exclusion = 0
> > > > > > > DEBUG 3: Rejected: obs type = 0
> > > > > > > DEBUG 3: Rejected: valid time = 0
> > > > > > > DEBUG 3: Rejected: bad obs value = 0
> > > > > > > DEBUG 3: Rejected: off the grid = 1
> > > > > > > DEBUG 3: Rejected: topography = 0
> > > > > > > DEBUG 3: Rejected: level mismatch = 0
> > > > > > > DEBUG 3: Rejected: quality marker = 0
> > > > > > > DEBUG 3: Rejected: message type = 0
> > > > > > > DEBUG 3: Rejected: masking region = 0
> > > > > > > DEBUG 3: Rejected: bad fcst value = 0
> > > > > > > DEBUG 3: Rejected: duplicates = 0
> > > > > > >
> > > > > >
> > > > > > We can see that there are no matched pairs and that they
were
> > > rejected
> > > > > > because they are "off the grid".
> > > > > >
> > > > > > We can use plot_point_obs to visualize where the
observation
> lies:
> > > > > >
> > > > > > > /usr/local/met-8.1.1/bin/plot_point_obs BATG_ob1.nc
BATG_ob1.ps
> > > > > -dotsize
> > > > > > 10
> > > > > > >
> > > > > >
> > > > > > I have attached BATG_ob1.ps for you. We can see the
observation
> is
> > > > near
> > > > > > Antarctica, and hopefully that is what you expect.
> > > > > >
> > > > > > We can use plot_data_plane to visualize the gridded model
output:
> > > > > >
> > > > > > > /usr/local/met-8.1.1/bin/plot_data_plane
> > > > > > > APCP_wrfout_d05_2017120912_f007_f005.nc
> > > > > > > APCP_wrfout_d05_2017120912_f007_f005.ps 'name="APCP_02";
> > > > level="A2";'
> > > > > > >
> > > > > >
> > > > > > However, there seems to be a problem with the projection
> > > information. I
> > > > > > have also attached APCP_wrfout_d05_2017120912_f007_f005.ps
for
> > you.
> > > > > > Looking in APCP_wrfout_d05_2017120912_f007_f005.nc, I can
see
> the
> > > > > > following delta values:
> > > > > >
> > > > > > > :delta_lat = "0.005932 degrees" ;
> > > > > > > :delta_lon = "0.418030 degrees" ;
> > > > > > >
> > > > > > which don't look like typical values. Usually they are
the same
> or
> > > > close
> > > > > > to the same.
> > > > > >
> > > > > > We can use the regrid_data_plane tool to regrid to NCEP
grid 4 to
> > see
> > > > > where
> > > > > > MET thinks the data is:
> > > > > >
> > > > > > > /usr/local/met-8.1.1/bin/regrid_data_plane
> > > > > > > APCP_wrfout_d05_2017120912_f007_f005.nc G004 global.nc
-field
> > > > > > > 'name="APCP_02"; level="A2";'
> > > > > > >
> > > > > > We can view this data using ncview:
> > > > > >
> > > > > > > /usr/local/ncview/bin/ncview global.nc &
> > > > > > >
> > > > > >
> > > > > > I took a screenshot of that image and attached it as
> global.nc.png.
> > > > You
> > > > > > can see the data is smooshed with a chuck missing as well.
This
> > > seems
> > > > > > problematic. If you'd like to pass on the GRIB data that
you
> used
> > in
> > > > > > running pcp_combine, we'd be happy to take a look at that.
> > > > > >
> > > > > > Also, since you're at NCAR, we could also get together to
discuss
> > how
> > > > > you'd
> > > > > > like to use MET and how we can help you with it. My
colleague,
> > John
> > > > > Halley
> > > > > > Gotway, helped me through this work. He is our MET
expert.
> Please
> > > let
> > > > > us
> > > > > > know if you'd like to arrange a time to meet with us.
> > > > > >
> > > > > > I hope this is helpful. Please let us know what other
questions
> > you
> > > > > have.
> > > > > >
> > > > > > Thanks,
> > > > > > Julie
> > > > > >
> > > > > > On Wed, Dec 4, 2019 at 2:48 PM lave at ucar.edu via RT <
> > > met_help at ucar.edu
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > Wed Dec 04 14:48:26 2019: Request 93338 was acted upon.
> > > > > > > Transaction: Ticket created by lave at ucar.edu
> > > > > > > Queue: met_help
> > > > > > > Subject: point_stat "unrecognized GRIB1 field";
> > > > > > > file_type="NETCDF_MET" in config file
> > > > > > > Owner: Nobody
> > > > > > > Requestors: lave at ucar.edu
> > > > > > > Status: new
> > > > > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Good afternoon,
> > > > > > >
> > > > > > > I am having trouble running MET on netCDF data. I used
> > pcp_combine
> > > > > > > -subtract to successfully create a netCDF file
> > > > > > > (APCP_wrfout_d05_2017120912_f007_f005.nc) containing
> > accumulation
> > > > data
> > > > > > for
> > > > > > > domain five of the AMPS evaluation area. I then used
ascii2nc
> to
> > > > > create a
> > > > > > > netCDF file (BATG_ob1.nc) containing an accumulation
> observation.
> > > > "psc"
> > > > > > is
> > > > > > > my configuration file, in which I specify
> > 'field_type="NETCDF_MET"'
> > > > in
> > > > > > the
> > > > > > > fcst dictionary. With 'name="APCP"', running point_stat
yields
> > the
> > > > > > > following:
> > > > > > >
> > > > > > > >point_stat APCP_wrfout_d05_2017120912_f007_f005.nc
> BATG_ob1.nc
> > > psc
> > > > > > >
> > > > > > > DEBUG 1: Default Config File:
> > > > > > > /usr/local/met-
8.1.1/share/met/config/PointStatConfig_default
> > > > > > >
> > > > > > > DEBUG 1: User Config File: psc
> > > > > > >
> > > > > > > GSL_RNG_TYPE=mt19937
> > > > > > >
> > > > > > > GSL_RNG_SEED=18446744072084449373
> > > > > > >
> > > > > > > DEBUG 1: Forecast File: APCP_wrfout_d05_2017120912_
> f007_f005.nc
> > > > > > >
> > > > > > > DEBUG 1: Observation File: BATG_ob1.nc
> > > > > > >
> > > > > > > DEBUG 2:
> > > > > > >
> > > > > > > DEBUG 2:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > > > >
> > > > > > > DEBUG 2:
> > > > > > >
> > > > > > > DEBUG 2: Reading data for APCPA2.
> > > > > > >
> > > > > > > WARNING:
> > > > > > >
> > > > > > > WARNING: process_fcst_climo_files() -> no fields
matching
> APCPA2
> > > > found
> > > > > in
> > > > > > > file: APCP_wrfout_d05_2017120912_f007_f005.nc
> > > > > > >
> > > > > > > WARNING:
> > > > > > >
> > > > > > > ERROR :
> > > > > > >
> > > > > > > ERROR : process_fcst_climo_files() -> no requested
forecast
> data
> > > > > found!
> > > > > > > Exiting...
> > > > > > >
> > > > > > > ERROR :
> > > > > > >
> > > > > > > yar:/d1/lave/antarctica> vi psc
> > > > > > >
> > > > > > > yar:/d1/lave/antarctica> point_stat
APCP_wrfout_d05_2017120912_
> > > > > > > f007_f005.nc
> > > > > > > BATG_ob1.nc psc
> > > > > > >
> > > > > > > DEBUG 1: Default Config File:
> > > > > > > /usr/local/met-
8.1.1/share/met/config/PointStatConfig_default
> > > > > > >
> > > > > > > DEBUG 1: User Config File: psc
> > > > > > >
> > > > > > > ERROR :
> > > > > > >
> > > > > > > ERROR : VarInfoGrib::add_grib_code() -> unrecognized
GRIB1
> field
> > > > > > > abbreviation 'APCP_02' for table version 2
> > > > > > >
> > > > > > > ERROR :
> > > > > > >
> > > > > > > So, following the answer given in [Met_help]
[rt.rap.ucar.edu
> > > > > #85872], I
> > > > > > > adjusted my config file to include 'name="APCP_02"'.
Running
> > > > point_stat
> > > > > > > then yielded:
> > > > > > >
> > > > > > > >point_stat APCP_wrfout_d05_2017120912_f007_f005.nc
> BATG_ob1.nc
> > > psc
> > > > > > >
> > > > > > > DEBUG 1: Default Config File:
> > > > > > > /usr/local/met-
8.1.1/share/met/config/PointStatConfig_default
> > > > > > >
> > > > > > > DEBUG 1: User Config File: psc
> > > > > > >
> > > > > > > ERROR :
> > > > > > >
> > > > > > > ERROR : VarInfoGrib::add_grib_code() -> unrecognized
GRIB1
> field
> > > > > > > abbreviation 'APCP_02' for table version 2
> > > > > > > ERROR :
> > > > > > >
> > > > > > > What is going on here? It seems like point_stat is
expecting a
> > > GRIB1
> > > > > > file,
> > > > > > > though I specified the input to be a MET-generated
netCDF
> file. I
> > > > > > uploaded
> > > > > > > the three files to your ftp site.
> > > > > > >
> > > > > > > Sincerely,
> > > > > > > Joshua Lave
> > > > > > > --
> > > > > > > Joshua Lave
> > > > > > >
> > > > > > > *Associate Scientist INational Center for Atmospheric
Research*
> > > > > > > *(303) 497-8258*
> > > > > > > *lave at ucar.edu <lave at ucar.edu>*
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Julie Prestopnik
> > > > > > Software Engineer
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > Phone: 303.497.8399
> > > > > > Email: jpresto at ucar.edu
> > > > > >
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Joshua Lave
> > > > >
> > > > > *Associate Scientist INational Center for Atmospheric
Research*
> > > > > *(303) 497-8258*
> > > > > *lave at ucar.edu <lave at ucar.edu>*
> > > > >
> > > > >
> > > >
> > > > --
> > > > Julie Prestopnik
> > > > Software Engineer
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > Phone: 303.497.8399
> > > > Email: jpresto at ucar.edu
> > > >
> > > > My working day may not be your working day. Please do not
feel
> obliged
> > > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > > --
> > > Joshua Lave
> > >
> > > *Associate Scientist INational Center for Atmospheric Research*
> > > *(303) 497-8258*
> > > *lave at ucar.edu <lave at ucar.edu>*
> > >
> > >
> >
> >
>
> --
> Joshua Lave
>
> *Associate Scientist INational Center for Atmospheric Research*
> *(303) 497-8258*
> *lave at ucar.edu <lave at ucar.edu>*
>
>
------------------------------------------------
Subject: point_stat "unrecognized GRIB1 field"; file_type="NETCDF_MET" in config file
From: lave at ucar.edu
Time: Tue Dec 17 10:13:45 2019
John,
Ah, I see. Great, everything seems to be in order as far as I can
tell.
Thank you again!
Joshua
On Mon, Dec 16, 2019 at 8:04 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Joshua,
>
> Yes, that’s why I sent the updated Point-Stat config file. You’re
no
> longer using a wrfout formatted file. You’re using the output from
> pcp_combine, which follows the format used by MET NetCDF files. So
you
> need to remove the file_type = NETCDF_PINT; option from the point-
stat
> config file.
>
> John
>
> On Mon, Dec 16, 2019 at 6:09 PM lave at ucar.edu via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338 >
> >
> > John,
> >
> > Including 'Times' in the subsetted file takes care of the valid
time
> issue.
> > However, I still run into the following error:
> >
> > point_stat diff.nc WLLY_ascii_RAIN.nc PointStatConfig_APCP -v 3
> > -obs_valid_beg 20180202_02 -obs_valid_end 20180202_02
> >
> >
> > DEBUG 1: Default Config File:
> > /usr/local/met-8.1.1/share/met/config/PointStatConfig_default
> >
> > DEBUG 1: User Config File: PointStatConfig_APCP
> >
> > ERROR :
> >
> > ERROR : MetNcPinterpDataFile::open(const char *) -> unable to
open
> NetCDF
> > file "diff.nc"
> >
> > ERROR :
> >
> > ERROR :
> >
> > ERROR : Met2dDataFileFactory::new_met_2d_data_file() -> error
opening
> file
> > "diff.nc"
> >
> > ERROR :
> >
> > What could be causing this?
> >
> > Joshua
> >
> > On Mon, Dec 16, 2019 at 5:10 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Joshua,
> > >
> > > OK, I'm looking at the issues we identified while meeting today.
> > >
> > > MET is not interpreting the time info correctly...
> > >
> > > Running pcp_combine on your subsetted APCP file yields weird
timing
> info:
> > > */usr/local/met-8.1.1/bin/pcp_combine -add
APCP_d05_2018020200_f002.nc
> > > 'name="RAINNC"; level="(0,*,*)"; file_type=NETCDF_PINT;' met.nc
> > > <http://met.nc>*
> > >
> > > But running the same command on the wrfout files does not:
> > > */usr/local/met-8.1.1/bin/pcp_combine -add
> wrfout_d05_2018020200_f002.nc
> > > <http://wrfout_d05_2018020200_f002.nc> 'name="RAINNC";
> level="(0,*,*)";
> > > file_type=NETCDF_PINT;' met.nc <http://met.nc>*
> > >
> > > So there is something in your subsetting process causing this
behavior.
> > > Diffing the headers, I see that the "wrfout" file includes a
variable
> > named
> > > "Times" that the subset does not. Please try re-subsetting but
> include:
> > >
> > >
> > > *variables: char Times(Time, DateStrLen) ;*
> > >
> > > Hopefully that'll yield better results. These weird timestamps
also
> led
> > to
> > > the runtime error we saw when trying to read the output of
pcp_combine.
> > >
> > > Regarding not matching "APCP" observations, you were right. It
was
> > looking
> > > for obs named "APCP_02". I've attached an updated Point-Stat
config
> file
> > > and my resulting log file, and listed below are the commands I
used to
> > get
> > > a sensible run.
> > >
> > > Note that I used awk to append the accumulation hour to the APCP
> > > observation variable name. And then I referenced that in the
> Point-Stat
> > > config file. I couldn't immediately figure out how to make that
2
> digits
> > > by adding a leading zero. So that's why the fcst references
RAINNC_02
> > > (which pcp_combine wrote out) and the obs references APCP_2.
> > >
> > > Thanks,
> > > John
> > >
> > > # Run pcp_combine
> > > /usr/local/met-8.1.1/bin/pcp_combine -subtract \
> > > wrfout_d05_2018020200_f002.nc 'name="RAINNC"; level="(0,*,*)";
> > > file_type=NETCDF_PINT;' \
> > > wrfout_d05_2018020200_f000.nc 'name="RAINNC"; level="(0,*,*)";
> > > file_type=NETCDF_PINT;' \
> > > wrfout_d05_2018020200_f000_to_f002.nc
> > >
> > > # Plot the result
> > > /usr/local/met-8.1.1/bin/plot_data_plane \
> > > wrfout_d05_2018020200_f000_to_f002.nc \
> > > wrfout_d05_2018020200_f000_to_f002.ps \
> > > 'name="RAINNC_02"; level="(*,*)";'
> > >
> > > # Prepare point observations
> > > cat WLLY_ascii.csv | awk '{print
> > > $1,$2,$3,$4,$5,$6,"APCP_"$8,$8,$9,$10,$11}' >
WLLY_ascii_APCP.csv
> > > /usr/local/met-8.1.1/bin/ascii2nc WLLY_ascii_APCP.csv
> WLLY_ascii_APCP.nc
> > >
> > > # Run point_stat
> > > mkdir -p out
> > > /usr/local/met-8.1.1/bin/point_stat \
> > > wrfout_d05_2018020200_f000_to_f002.nc \
> > > WLLY_ascii_APCP.nc \
> > > PointStatConfig_APCP \
> > > -outdir out -v 3 -log run_ps.log
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Dec 16, 2019 at 1:14 PM lave at ucar.edu via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338
>
> > > >
> > > > Yes, thank you for checking in. I did receive the invitation
and
> > accepted
> > > > it. I'll see you at 3.
> > > >
> > > > Joshua
> > > >
> > > > On Mon, Dec 16, 2019 at 9:41 AM Julie Prestopnik via RT <
> > > met_help at ucar.edu
> > > > >
> > > > wrote:
> > > >
> > > > > Hi Joshua.
> > > > >
> > > > > The delay in response is no problem. It's a busy time of
year.
> > We're
> > > > > happy to help you get started using MET.
> > > > >
> > > > > My colleague, John, set up a meeting for us today at 3pm in
> FL3-1029.
> > > I
> > > > > just wanted to make sure you received the invitation and
check in
> to
> > > see
> > > > if
> > > > > you'll be able to meet at that time? I'll be able to stay
for
> about
> > > the
> > > > > first 20 minutes of the meeting.
> > > > >
> > > > > Thanks,
> > > > > Julie
> > > > >
> > > > > On Fri, Dec 13, 2019 at 3:04 PM lave at ucar.edu via RT <
> > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338 >
> > > > > >
> > > > > > Julie,
> > > > > >
> > > > > > Sorry it has taken me so long to reply, but thank you for
your
> > > detailed
> > > > > > response. This is my first time working with MET and I am
doing
> so
> > at
> > > > the
> > > > > > suggestion of another scientist. Most of my struggles so
far have
> > > > related
> > > > > > to properly formatting my datasets. The GRIB data that I
used in
> > > > > > pcp_combine is a set that I converted from netCDF via
wgrib and
> cdo
> > > as
> > > > I
> > > > > > could not get pcp_combine to accept the nc files.
> > > > > >
> > > > > > I would like to accept your offer to meet with me to
discuss MET
> > and
> > > > how
> > > > > I
> > > > > > plan to use it. I am available most of next week (12/16-
12:20),
> > > > including
> > > > > > all day Monday, Wednesday, and Friday.
> > > > > >
> > > > > > Joshua
> > > > > >
> > > > > > On Thu, Dec 5, 2019 at 2:54 PM Julie Prestopnik via RT <
> > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Joshua.
> > > > > > >
> > > > > > > I see that you are having some difficult running
point_stat
> with
> > > > NetCDF
> > > > > > > data.
> > > > > > >
> > > > > > > In your config file, I noticed that you had set the
> "output_flag"
> > > for
> > > > > > > "pstd"
> > > > > > > to "BOTH", but had left all other values as "NONE".
"pstd" is
> for
> > > > > > > probabilistic data. Is your intention to use this for
> > verification
> > > > of
> > > > > > > probabilistic data or for traditional unit verification?
> > > > > > >
> > > > > > > I made some changes to your config file and have
attached a new
> > > > config
> > > > > > > file. I'll outline the differences below:
> > > > > > >
> > > > > > > - I removed your entries for init_time, valid_time,
and
> > > lead_time.
> > > > > > > Those fields are typically used for file that have
many lead
> > > times
> > > > > > > specified. In this case, the file only has one time,
so it
> is
> > > not
> > > > > > > necessary to specify those values. You only want to
add
> those
> > > > > values
> > > > > > if
> > > > > > > they are needed. MET will be able to parse the file
for
> those
> > > > > values
> > > > > > in
> > > > > > > this case.
> > > > > > > - I removed "file_type = "NETCDF_MET";" as it is also
not
> > > > necessary
> > > > > in
> > > > > > > this case.
> > > > > > > - I changed the values of the "output_flag" section,
> assuming
> > > that
> > > > > you
> > > > > > > wanted to do traditional unit verification. So, I
changed
> > > "pstd"
> > > > to
> > > > > > > "NONE". I changed "ctc" (contingency table counts)
and
> "cts"
> > > > > > > (statistics
> > > > > > > on those contingency table counts) to "BOTH". The
> > "cat_thresh",
> > > > or
> > > > > > > categorical threshold, is used to define what
constitutes an
> > > > "event"
> > > > > > > when
> > > > > > > computing a 2x2 contingency table. I also changed
"cnt",
> > > > continuous
> > > > > > > statistics to "BOTH", because a lot of folks like to
use
> that
> > > > > > > information
> > > > > > > as well.
> > > > > > > - I also changed the "obs = fcst;" field to give
"obs" its
> own
> > > > > value.
> > > > > > > In this case, obs=fcst won't work because "APCP_02"
does not
> > > exist
> > > > > in
> > > > > > > the
> > > > > > > obs file, so MET would not find any matches. In
order to
> have
> > > MET
> > > > > be
> > > > > > > backward compatible, we are stuck with the GRIB
naming
> > > convention.
> > > > > > The
> > > > > > > obs file contains a GRIB value for "obs_gc". This
triggers
> > > logic
> > > > in
> > > > > > > MET to
> > > > > > > do a table lookup, but MET would find "APCP" and not
> > > "APCP_02". I
> > > > > can
> > > > > > > see
> > > > > > > that the obs file BATG_ob1 was derived from using
ascii2nc.
> > If
> > > > > > > "APCP_02" had been listed in the ascii file as the
value for
> > the
> > > > > > > "Var_Name(or GRIB_Code)" field, and the NetCDF file
then
> > > contained
> > > > > > > "APCP_02", you may have likely been able to use "obs
=
> fcst;".
> > > > > > However,
> > > > > > > that brings us to another issue in that the message
type,
> aka
> > > > > > > "hdr_typ_table" (hdr_typ_table:long_name = "message
type"
> ;),
> > > > value
> > > > > in
> > > > > > > the
> > > > > > > obs file is "default" so MET will not find any
matches there
> > > > either,
> > > > > > as
> > > > > > > you have set "message_type = [ "ADPSFC" ];" in the
config
> > file
> > > > and
> > > > > > > "default" wouldn't match "ADPSFC". You could edit
the ascii
> > > data
> > > > > and
> > > > > > > send
> > > > > > > it through ascii2nc again. It is recommended to
change
> > > "default"
> > > > to
> > > > > > > something that is more meaningful for your data. So,
MET
> will
> > > not
> > > > > > find
> > > > > > > any
> > > > > > > matched pairs until that issue is resolved.
> > > > > > >
> > > > > > > If I now run your data using point_stat with the changes
and a
> > > higher
> > > > > > > verbosity level:
> > > > > > >
> > > > > > > > /usr/local/met-8.1.1/bin/point_stat
> > > > > > > > APCP_wrfout_d05_2017120912_f007_f005.nc BATG_ob1.nc
psc_new
> > -v 3
> > > > > > > >
> > > > > > >
> > > > > > > I do get a successful run, however, there are still no
matched
> > > pairs
> > > > > for
> > > > > > a
> > > > > > > different reason. The unmatched message_type problem
would
> still
> > > > show
> > > > > up
> > > > > > > at some point, but the first problem I encountered when
running
> > > with
> > > > > the
> > > > > > > new config file is:
> > > > > > >
> > > > > > > > DEBUG 2: Processing APCP_02A2 versus APCP/Z0, for
observation
> > > type
> > > > > > > ADPSFC,
> > > > > > > > over region FULL, for interpolation method NEAREST(1),
using
> 0
> > > > > matched
> > > > > > > > pairs.
> > > > > > > > DEBUG 3: Number of matched pairs = 0
> > > > > > > > DEBUG 3: Observations processed = 1
> > > > > > > > DEBUG 3: Rejected: SID exclusion = 0
> > > > > > > > DEBUG 3: Rejected: obs type = 0
> > > > > > > > DEBUG 3: Rejected: valid time = 0
> > > > > > > > DEBUG 3: Rejected: bad obs value = 0
> > > > > > > > DEBUG 3: Rejected: off the grid = 1
> > > > > > > > DEBUG 3: Rejected: topography = 0
> > > > > > > > DEBUG 3: Rejected: level mismatch = 0
> > > > > > > > DEBUG 3: Rejected: quality marker = 0
> > > > > > > > DEBUG 3: Rejected: message type = 0
> > > > > > > > DEBUG 3: Rejected: masking region = 0
> > > > > > > > DEBUG 3: Rejected: bad fcst value = 0
> > > > > > > > DEBUG 3: Rejected: duplicates = 0
> > > > > > > >
> > > > > > >
> > > > > > > We can see that there are no matched pairs and that they
were
> > > > rejected
> > > > > > > because they are "off the grid".
> > > > > > >
> > > > > > > We can use plot_point_obs to visualize where the
observation
> > lies:
> > > > > > >
> > > > > > > > /usr/local/met-8.1.1/bin/plot_point_obs BATG_ob1.nc
> BATG_ob1.ps
> > > > > > -dotsize
> > > > > > > 10
> > > > > > > >
> > > > > > >
> > > > > > > I have attached BATG_ob1.ps for you. We can see the
> observation
> > is
> > > > > near
> > > > > > > Antarctica, and hopefully that is what you expect.
> > > > > > >
> > > > > > > We can use plot_data_plane to visualize the gridded
model
> output:
> > > > > > >
> > > > > > > > /usr/local/met-8.1.1/bin/plot_data_plane
> > > > > > > > APCP_wrfout_d05_2017120912_f007_f005.nc
> > > > > > > > APCP_wrfout_d05_2017120912_f007_f005.ps
'name="APCP_02";
> > > > > level="A2";'
> > > > > > > >
> > > > > > >
> > > > > > > However, there seems to be a problem with the projection
> > > > information. I
> > > > > > > have also attached
APCP_wrfout_d05_2017120912_f007_f005.ps for
> > > you.
> > > > > > > Looking in APCP_wrfout_d05_2017120912_f007_f005.nc, I
can see
> > the
> > > > > > > following delta values:
> > > > > > >
> > > > > > > > :delta_lat = "0.005932 degrees" ;
> > > > > > > > :delta_lon = "0.418030 degrees" ;
> > > > > > > >
> > > > > > > which don't look like typical values. Usually they are
the
> same
> > or
> > > > > close
> > > > > > > to the same.
> > > > > > >
> > > > > > > We can use the regrid_data_plane tool to regrid to NCEP
grid 4
> to
> > > see
> > > > > > where
> > > > > > > MET thinks the data is:
> > > > > > >
> > > > > > > > /usr/local/met-8.1.1/bin/regrid_data_plane
> > > > > > > > APCP_wrfout_d05_2017120912_f007_f005.nc G004 global.nc
> -field
> > > > > > > > 'name="APCP_02"; level="A2";'
> > > > > > > >
> > > > > > > We can view this data using ncview:
> > > > > > >
> > > > > > > > /usr/local/ncview/bin/ncview global.nc &
> > > > > > > >
> > > > > > >
> > > > > > > I took a screenshot of that image and attached it as
> > global.nc.png.
> > > > > You
> > > > > > > can see the data is smooshed with a chuck missing as
well.
> This
> > > > seems
> > > > > > > problematic. If you'd like to pass on the GRIB data
that you
> > used
> > > in
> > > > > > > running pcp_combine, we'd be happy to take a look at
that.
> > > > > > >
> > > > > > > Also, since you're at NCAR, we could also get together
to
> discuss
> > > how
> > > > > > you'd
> > > > > > > like to use MET and how we can help you with it. My
colleague,
> > > John
> > > > > > Halley
> > > > > > > Gotway, helped me through this work. He is our MET
expert.
> > Please
> > > > let
> > > > > > us
> > > > > > > know if you'd like to arrange a time to meet with us.
> > > > > > >
> > > > > > > I hope this is helpful. Please let us know what other
> questions
> > > you
> > > > > > have.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Julie
> > > > > > >
> > > > > > > On Wed, Dec 4, 2019 at 2:48 PM lave at ucar.edu via RT <
> > > > met_help at ucar.edu
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Wed Dec 04 14:48:26 2019: Request 93338 was acted
upon.
> > > > > > > > Transaction: Ticket created by lave at ucar.edu
> > > > > > > > Queue: met_help
> > > > > > > > Subject: point_stat "unrecognized GRIB1 field";
> > > > > > > > file_type="NETCDF_MET" in config file
> > > > > > > > Owner: Nobody
> > > > > > > > Requestors: lave at ucar.edu
> > > > > > > > Status: new
> > > > > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Good afternoon,
> > > > > > > >
> > > > > > > > I am having trouble running MET on netCDF data. I used
> > > pcp_combine
> > > > > > > > -subtract to successfully create a netCDF file
> > > > > > > > (APCP_wrfout_d05_2017120912_f007_f005.nc) containing
> > > accumulation
> > > > > data
> > > > > > > for
> > > > > > > > domain five of the AMPS evaluation area. I then used
ascii2nc
> > to
> > > > > > create a
> > > > > > > > netCDF file (BATG_ob1.nc) containing an accumulation
> > observation.
> > > > > "psc"
> > > > > > > is
> > > > > > > > my configuration file, in which I specify
> > > 'field_type="NETCDF_MET"'
> > > > > in
> > > > > > > the
> > > > > > > > fcst dictionary. With 'name="APCP"', running
point_stat
> yields
> > > the
> > > > > > > > following:
> > > > > > > >
> > > > > > > > >point_stat APCP_wrfout_d05_2017120912_f007_f005.nc
> > BATG_ob1.nc
> > > > psc
> > > > > > > >
> > > > > > > > DEBUG 1: Default Config File:
> > > > > > > > /usr/local/met-
8.1.1/share/met/config/PointStatConfig_default
> > > > > > > >
> > > > > > > > DEBUG 1: User Config File: psc
> > > > > > > >
> > > > > > > > GSL_RNG_TYPE=mt19937
> > > > > > > >
> > > > > > > > GSL_RNG_SEED=18446744072084449373
> > > > > > > >
> > > > > > > > DEBUG 1: Forecast File: APCP_wrfout_d05_2017120912_
> > f007_f005.nc
> > > > > > > >
> > > > > > > > DEBUG 1: Observation File: BATG_ob1.nc
> > > > > > > >
> > > > > > > > DEBUG 2:
> > > > > > > >
> > > > > > > > DEBUG 2:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > > > > >
> > > > > > > > DEBUG 2:
> > > > > > > >
> > > > > > > > DEBUG 2: Reading data for APCPA2.
> > > > > > > >
> > > > > > > > WARNING:
> > > > > > > >
> > > > > > > > WARNING: process_fcst_climo_files() -> no fields
matching
> > APCPA2
> > > > > found
> > > > > > in
> > > > > > > > file: APCP_wrfout_d05_2017120912_f007_f005.nc
> > > > > > > >
> > > > > > > > WARNING:
> > > > > > > >
> > > > > > > > ERROR :
> > > > > > > >
> > > > > > > > ERROR : process_fcst_climo_files() -> no requested
forecast
> > data
> > > > > > found!
> > > > > > > > Exiting...
> > > > > > > >
> > > > > > > > ERROR :
> > > > > > > >
> > > > > > > > yar:/d1/lave/antarctica> vi psc
> > > > > > > >
> > > > > > > > yar:/d1/lave/antarctica> point_stat
> APCP_wrfout_d05_2017120912_
> > > > > > > > f007_f005.nc
> > > > > > > > BATG_ob1.nc psc
> > > > > > > >
> > > > > > > > DEBUG 1: Default Config File:
> > > > > > > > /usr/local/met-
8.1.1/share/met/config/PointStatConfig_default
> > > > > > > >
> > > > > > > > DEBUG 1: User Config File: psc
> > > > > > > >
> > > > > > > > ERROR :
> > > > > > > >
> > > > > > > > ERROR : VarInfoGrib::add_grib_code() -> unrecognized
GRIB1
> > field
> > > > > > > > abbreviation 'APCP_02' for table version 2
> > > > > > > >
> > > > > > > > ERROR :
> > > > > > > >
> > > > > > > > So, following the answer given in [Met_help] [
> rt.rap.ucar.edu
> > > > > > #85872], I
> > > > > > > > adjusted my config file to include 'name="APCP_02"'.
Running
> > > > > point_stat
> > > > > > > > then yielded:
> > > > > > > >
> > > > > > > > >point_stat APCP_wrfout_d05_2017120912_f007_f005.nc
> > BATG_ob1.nc
> > > > psc
> > > > > > > >
> > > > > > > > DEBUG 1: Default Config File:
> > > > > > > > /usr/local/met-
8.1.1/share/met/config/PointStatConfig_default
> > > > > > > >
> > > > > > > > DEBUG 1: User Config File: psc
> > > > > > > >
> > > > > > > > ERROR :
> > > > > > > >
> > > > > > > > ERROR : VarInfoGrib::add_grib_code() -> unrecognized
GRIB1
> > field
> > > > > > > > abbreviation 'APCP_02' for table version 2
> > > > > > > > ERROR :
> > > > > > > >
> > > > > > > > What is going on here? It seems like point_stat is
expecting
> a
> > > > GRIB1
> > > > > > > file,
> > > > > > > > though I specified the input to be a MET-generated
netCDF
> > file. I
> > > > > > > uploaded
> > > > > > > > the three files to your ftp site.
> > > > > > > >
> > > > > > > > Sincerely,
> > > > > > > > Joshua Lave
> > > > > > > > --
> > > > > > > > Joshua Lave
> > > > > > > >
> > > > > > > > *Associate Scientist INational Center for Atmospheric
> Research*
> > > > > > > > *(303) 497-8258*
> > > > > > > > *lave at ucar.edu <lave at ucar.edu>*
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Julie Prestopnik
> > > > > > > Software Engineer
> > > > > > > National Center for Atmospheric Research
> > > > > > > Research Applications Laboratory
> > > > > > > Phone: 303.497.8399
> > > > > > > Email: jpresto at ucar.edu
> > > > > > >
> > > > > > > My working day may not be your working day. Please do
not feel
> > > > obliged
> > > > > > to
> > > > > > > reply to this email outside of your normal working
hours.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Joshua Lave
> > > > > >
> > > > > > *Associate Scientist INational Center for Atmospheric
Research*
> > > > > > *(303) 497-8258*
> > > > > > *lave at ucar.edu <lave at ucar.edu>*
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Julie Prestopnik
> > > > > Software Engineer
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > Phone: 303.497.8399
> > > > > Email: jpresto at ucar.edu
> > > > >
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > > --
> > > > Joshua Lave
> > > >
> > > > *Associate Scientist INational Center for Atmospheric
Research*
> > > > *(303) 497-8258*
> > > > *lave at ucar.edu <lave at ucar.edu>*
> > > >
> > > >
> > >
> > >
> >
> > --
> > Joshua Lave
> >
> > *Associate Scientist INational Center for Atmospheric Research*
> > *(303) 497-8258*
> > *lave at ucar.edu <lave at ucar.edu>*
> >
> >
>
>
--
Joshua Lave
*Associate Scientist INational Center for Atmospheric Research*
*(303) 497-8258*
*lave at ucar.edu <lave at ucar.edu>*
------------------------------------------------
Subject: point_stat "unrecognized GRIB1 field"; file_type="NETCDF_MET" in config file
From: Julie Prestopnik
Time: Tue Dec 17 10:19:15 2019
Glad to hear it, Joshua! I'll go ahead and close this ticket. Please
feel
free to submit a new ticket with any other questions you have or go
ahead
and reopen this one if you encounter other related issues.
Julie
On Tue, Dec 17, 2019 at 10:14 AM lave at ucar.edu via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338 >
>
> John,
>
> Ah, I see. Great, everything seems to be in order as far as I can
tell.
> Thank you again!
>
> Joshua
>
> On Mon, Dec 16, 2019 at 8:04 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Joshua,
> >
> > Yes, that’s why I sent the updated Point-Stat config file. You’re
no
> > longer using a wrfout formatted file. You’re using the output
from
> > pcp_combine, which follows the format used by MET NetCDF files.
So you
> > need to remove the file_type = NETCDF_PINT; option from the point-
stat
> > config file.
> >
> > John
> >
> > On Mon, Dec 16, 2019 at 6:09 PM lave at ucar.edu via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338 >
> > >
> > > John,
> > >
> > > Including 'Times' in the subsetted file takes care of the valid
time
> > issue.
> > > However, I still run into the following error:
> > >
> > > point_stat diff.nc WLLY_ascii_RAIN.nc PointStatConfig_APCP -v 3
> > > -obs_valid_beg 20180202_02 -obs_valid_end 20180202_02
> > >
> > >
> > > DEBUG 1: Default Config File:
> > > /usr/local/met-8.1.1/share/met/config/PointStatConfig_default
> > >
> > > DEBUG 1: User Config File: PointStatConfig_APCP
> > >
> > > ERROR :
> > >
> > > ERROR : MetNcPinterpDataFile::open(const char *) -> unable to
open
> > NetCDF
> > > file "diff.nc"
> > >
> > > ERROR :
> > >
> > > ERROR :
> > >
> > > ERROR : Met2dDataFileFactory::new_met_2d_data_file() -> error
opening
> > file
> > > "diff.nc"
> > >
> > > ERROR :
> > >
> > > What could be causing this?
> > >
> > > Joshua
> > >
> > > On Mon, Dec 16, 2019 at 5:10 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Joshua,
> > > >
> > > > OK, I'm looking at the issues we identified while meeting
today.
> > > >
> > > > MET is not interpreting the time info correctly...
> > > >
> > > > Running pcp_combine on your subsetted APCP file yields weird
timing
> > info:
> > > > */usr/local/met-8.1.1/bin/pcp_combine -add
> APCP_d05_2018020200_f002.nc
> > > > 'name="RAINNC"; level="(0,*,*)"; file_type=NETCDF_PINT;'
met.nc
> > > > <http://met.nc>*
> > > >
> > > > But running the same command on the wrfout files does not:
> > > > */usr/local/met-8.1.1/bin/pcp_combine -add
> > wrfout_d05_2018020200_f002.nc
> > > > <http://wrfout_d05_2018020200_f002.nc> 'name="RAINNC";
> > level="(0,*,*)";
> > > > file_type=NETCDF_PINT;' met.nc <http://met.nc>*
> > > >
> > > > So there is something in your subsetting process causing this
> behavior.
> > > > Diffing the headers, I see that the "wrfout" file includes a
variable
> > > named
> > > > "Times" that the subset does not. Please try re-subsetting
but
> > include:
> > > >
> > > >
> > > > *variables: char Times(Time, DateStrLen) ;*
> > > >
> > > > Hopefully that'll yield better results. These weird
timestamps also
> > led
> > > to
> > > > the runtime error we saw when trying to read the output of
> pcp_combine.
> > > >
> > > > Regarding not matching "APCP" observations, you were right.
It was
> > > looking
> > > > for obs named "APCP_02". I've attached an updated Point-Stat
config
> > file
> > > > and my resulting log file, and listed below are the commands I
used
> to
> > > get
> > > > a sensible run.
> > > >
> > > > Note that I used awk to append the accumulation hour to the
APCP
> > > > observation variable name. And then I referenced that in the
> > Point-Stat
> > > > config file. I couldn't immediately figure out how to make
that 2
> > digits
> > > > by adding a leading zero. So that's why the fcst references
> RAINNC_02
> > > > (which pcp_combine wrote out) and the obs references APCP_2.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > # Run pcp_combine
> > > > /usr/local/met-8.1.1/bin/pcp_combine -subtract \
> > > > wrfout_d05_2018020200_f002.nc 'name="RAINNC"; level="(0,*,*)";
> > > > file_type=NETCDF_PINT;' \
> > > > wrfout_d05_2018020200_f000.nc 'name="RAINNC"; level="(0,*,*)";
> > > > file_type=NETCDF_PINT;' \
> > > > wrfout_d05_2018020200_f000_to_f002.nc
> > > >
> > > > # Plot the result
> > > > /usr/local/met-8.1.1/bin/plot_data_plane \
> > > > wrfout_d05_2018020200_f000_to_f002.nc \
> > > > wrfout_d05_2018020200_f000_to_f002.ps \
> > > > 'name="RAINNC_02"; level="(*,*)";'
> > > >
> > > > # Prepare point observations
> > > > cat WLLY_ascii.csv | awk '{print
> > > > $1,$2,$3,$4,$5,$6,"APCP_"$8,$8,$9,$10,$11}' >
WLLY_ascii_APCP.csv
> > > > /usr/local/met-8.1.1/bin/ascii2nc WLLY_ascii_APCP.csv
> > WLLY_ascii_APCP.nc
> > > >
> > > > # Run point_stat
> > > > mkdir -p out
> > > > /usr/local/met-8.1.1/bin/point_stat \
> > > > wrfout_d05_2018020200_f000_to_f002.nc \
> > > > WLLY_ascii_APCP.nc \
> > > > PointStatConfig_APCP \
> > > > -outdir out -v 3 -log run_ps.log
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Dec 16, 2019 at 1:14 PM lave at ucar.edu via RT <
> > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338 >
> > > > >
> > > > > Yes, thank you for checking in. I did receive the invitation
and
> > > accepted
> > > > > it. I'll see you at 3.
> > > > >
> > > > > Joshua
> > > > >
> > > > > On Mon, Dec 16, 2019 at 9:41 AM Julie Prestopnik via RT <
> > > > met_help at ucar.edu
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Joshua.
> > > > > >
> > > > > > The delay in response is no problem. It's a busy time of
year.
> > > We're
> > > > > > happy to help you get started using MET.
> > > > > >
> > > > > > My colleague, John, set up a meeting for us today at 3pm
in
> > FL3-1029.
> > > > I
> > > > > > just wanted to make sure you received the invitation and
check in
> > to
> > > > see
> > > > > if
> > > > > > you'll be able to meet at that time? I'll be able to stay
for
> > about
> > > > the
> > > > > > first 20 minutes of the meeting.
> > > > > >
> > > > > > Thanks,
> > > > > > Julie
> > > > > >
> > > > > > On Fri, Dec 13, 2019 at 3:04 PM lave at ucar.edu via RT <
> > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338
> >
> > > > > > >
> > > > > > > Julie,
> > > > > > >
> > > > > > > Sorry it has taken me so long to reply, but thank you
for your
> > > > detailed
> > > > > > > response. This is my first time working with MET and I
am doing
> > so
> > > at
> > > > > the
> > > > > > > suggestion of another scientist. Most of my struggles so
far
> have
> > > > > related
> > > > > > > to properly formatting my datasets. The GRIB data that I
used
> in
> > > > > > > pcp_combine is a set that I converted from netCDF via
wgrib and
> > cdo
> > > > as
> > > > > I
> > > > > > > could not get pcp_combine to accept the nc files.
> > > > > > >
> > > > > > > I would like to accept your offer to meet with me to
discuss
> MET
> > > and
> > > > > how
> > > > > > I
> > > > > > > plan to use it. I am available most of next week (12/16-
12:20),
> > > > > including
> > > > > > > all day Monday, Wednesday, and Friday.
> > > > > > >
> > > > > > > Joshua
> > > > > > >
> > > > > > > On Thu, Dec 5, 2019 at 2:54 PM Julie Prestopnik via RT <
> > > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Joshua.
> > > > > > > >
> > > > > > > > I see that you are having some difficult running
point_stat
> > with
> > > > > NetCDF
> > > > > > > > data.
> > > > > > > >
> > > > > > > > In your config file, I noticed that you had set the
> > "output_flag"
> > > > for
> > > > > > > > "pstd"
> > > > > > > > to "BOTH", but had left all other values as "NONE".
"pstd" is
> > for
> > > > > > > > probabilistic data. Is your intention to use this for
> > > verification
> > > > > of
> > > > > > > > probabilistic data or for traditional unit
verification?
> > > > > > > >
> > > > > > > > I made some changes to your config file and have
attached a
> new
> > > > > config
> > > > > > > > file. I'll outline the differences below:
> > > > > > > >
> > > > > > > > - I removed your entries for init_time, valid_time,
and
> > > > lead_time.
> > > > > > > > Those fields are typically used for file that have
many
> lead
> > > > times
> > > > > > > > specified. In this case, the file only has one
time, so
> it
> > is
> > > > not
> > > > > > > > necessary to specify those values. You only want
to add
> > those
> > > > > > values
> > > > > > > if
> > > > > > > > they are needed. MET will be able to parse the
file for
> > those
> > > > > > values
> > > > > > > in
> > > > > > > > this case.
> > > > > > > > - I removed "file_type = "NETCDF_MET";" as it is
also not
> > > > > necessary
> > > > > > in
> > > > > > > > this case.
> > > > > > > > - I changed the values of the "output_flag"
section,
> > assuming
> > > > that
> > > > > > you
> > > > > > > > wanted to do traditional unit verification. So, I
changed
> > > > "pstd"
> > > > > to
> > > > > > > > "NONE". I changed "ctc" (contingency table counts)
and
> > "cts"
> > > > > > > > (statistics
> > > > > > > > on those contingency table counts) to "BOTH". The
> > > "cat_thresh",
> > > > > or
> > > > > > > > categorical threshold, is used to define what
constitutes
> an
> > > > > "event"
> > > > > > > > when
> > > > > > > > computing a 2x2 contingency table. I also changed
"cnt",
> > > > > continuous
> > > > > > > > statistics to "BOTH", because a lot of folks like
to use
> > that
> > > > > > > > information
> > > > > > > > as well.
> > > > > > > > - I also changed the "obs = fcst;" field to give
"obs" its
> > own
> > > > > > value.
> > > > > > > > In this case, obs=fcst won't work because "APCP_02"
does
> not
> > > > exist
> > > > > > in
> > > > > > > > the
> > > > > > > > obs file, so MET would not find any matches. In
order to
> > have
> > > > MET
> > > > > > be
> > > > > > > > backward compatible, we are stuck with the GRIB
naming
> > > > convention.
> > > > > > > The
> > > > > > > > obs file contains a GRIB value for "obs_gc". This
> triggers
> > > > logic
> > > > > in
> > > > > > > > MET to
> > > > > > > > do a table lookup, but MET would find "APCP" and
not
> > > > "APCP_02". I
> > > > > > can
> > > > > > > > see
> > > > > > > > that the obs file BATG_ob1 was derived from using
> ascii2nc.
> > > If
> > > > > > > > "APCP_02" had been listed in the ascii file as the
value
> for
> > > the
> > > > > > > > "Var_Name(or GRIB_Code)" field, and the NetCDF file
then
> > > > contained
> > > > > > > > "APCP_02", you may have likely been able to use
"obs =
> > fcst;".
> > > > > > > However,
> > > > > > > > that brings us to another issue in that the message
type,
> > aka
> > > > > > > > "hdr_typ_table" (hdr_typ_table:long_name = "message
type"
> > ;),
> > > > > value
> > > > > > in
> > > > > > > > the
> > > > > > > > obs file is "default" so MET will not find any
matches
> there
> > > > > either,
> > > > > > > as
> > > > > > > > you have set "message_type = [ "ADPSFC" ];" in the
config
> > > file
> > > > > and
> > > > > > > > "default" wouldn't match "ADPSFC". You could edit
the
> ascii
> > > > data
> > > > > > and
> > > > > > > > send
> > > > > > > > it through ascii2nc again. It is recommended to
change
> > > > "default"
> > > > > to
> > > > > > > > something that is more meaningful for your data.
So, MET
> > will
> > > > not
> > > > > > > find
> > > > > > > > any
> > > > > > > > matched pairs until that issue is resolved.
> > > > > > > >
> > > > > > > > If I now run your data using point_stat with the
changes and
> a
> > > > higher
> > > > > > > > verbosity level:
> > > > > > > >
> > > > > > > > > /usr/local/met-8.1.1/bin/point_stat
> > > > > > > > > APCP_wrfout_d05_2017120912_f007_f005.nc BATG_ob1.nc
> psc_new
> > > -v 3
> > > > > > > > >
> > > > > > > >
> > > > > > > > I do get a successful run, however, there are still no
> matched
> > > > pairs
> > > > > > for
> > > > > > > a
> > > > > > > > different reason. The unmatched message_type problem
would
> > still
> > > > > show
> > > > > > up
> > > > > > > > at some point, but the first problem I encountered
when
> running
> > > > with
> > > > > > the
> > > > > > > > new config file is:
> > > > > > > >
> > > > > > > > > DEBUG 2: Processing APCP_02A2 versus APCP/Z0, for
> observation
> > > > type
> > > > > > > > ADPSFC,
> > > > > > > > > over region FULL, for interpolation method
NEAREST(1),
> using
> > 0
> > > > > > matched
> > > > > > > > > pairs.
> > > > > > > > > DEBUG 3: Number of matched pairs = 0
> > > > > > > > > DEBUG 3: Observations processed = 1
> > > > > > > > > DEBUG 3: Rejected: SID exclusion = 0
> > > > > > > > > DEBUG 3: Rejected: obs type = 0
> > > > > > > > > DEBUG 3: Rejected: valid time = 0
> > > > > > > > > DEBUG 3: Rejected: bad obs value = 0
> > > > > > > > > DEBUG 3: Rejected: off the grid = 1
> > > > > > > > > DEBUG 3: Rejected: topography = 0
> > > > > > > > > DEBUG 3: Rejected: level mismatch = 0
> > > > > > > > > DEBUG 3: Rejected: quality marker = 0
> > > > > > > > > DEBUG 3: Rejected: message type = 0
> > > > > > > > > DEBUG 3: Rejected: masking region = 0
> > > > > > > > > DEBUG 3: Rejected: bad fcst value = 0
> > > > > > > > > DEBUG 3: Rejected: duplicates = 0
> > > > > > > > >
> > > > > > > >
> > > > > > > > We can see that there are no matched pairs and that
they were
> > > > > rejected
> > > > > > > > because they are "off the grid".
> > > > > > > >
> > > > > > > > We can use plot_point_obs to visualize where the
observation
> > > lies:
> > > > > > > >
> > > > > > > > > /usr/local/met-8.1.1/bin/plot_point_obs BATG_ob1.nc
> > BATG_ob1.ps
> > > > > > > -dotsize
> > > > > > > > 10
> > > > > > > > >
> > > > > > > >
> > > > > > > > I have attached BATG_ob1.ps for you. We can see the
> > observation
> > > is
> > > > > > near
> > > > > > > > Antarctica, and hopefully that is what you expect.
> > > > > > > >
> > > > > > > > We can use plot_data_plane to visualize the gridded
model
> > output:
> > > > > > > >
> > > > > > > > > /usr/local/met-8.1.1/bin/plot_data_plane
> > > > > > > > > APCP_wrfout_d05_2017120912_f007_f005.nc
> > > > > > > > > APCP_wrfout_d05_2017120912_f007_f005.ps
'name="APCP_02";
> > > > > > level="A2";'
> > > > > > > > >
> > > > > > > >
> > > > > > > > However, there seems to be a problem with the
projection
> > > > > information. I
> > > > > > > > have also attached
APCP_wrfout_d05_2017120912_f007_f005.ps
> for
> > > > you.
> > > > > > > > Looking in APCP_wrfout_d05_2017120912_f007_f005.nc, I
can
> see
> > > the
> > > > > > > > following delta values:
> > > > > > > >
> > > > > > > > > :delta_lat = "0.005932 degrees" ;
> > > > > > > > > :delta_lon = "0.418030 degrees" ;
> > > > > > > > >
> > > > > > > > which don't look like typical values. Usually they
are the
> > same
> > > or
> > > > > > close
> > > > > > > > to the same.
> > > > > > > >
> > > > > > > > We can use the regrid_data_plane tool to regrid to
NCEP grid
> 4
> > to
> > > > see
> > > > > > > where
> > > > > > > > MET thinks the data is:
> > > > > > > >
> > > > > > > > > /usr/local/met-8.1.1/bin/regrid_data_plane
> > > > > > > > > APCP_wrfout_d05_2017120912_f007_f005.nc G004
global.nc
> > -field
> > > > > > > > > 'name="APCP_02"; level="A2";'
> > > > > > > > >
> > > > > > > > We can view this data using ncview:
> > > > > > > >
> > > > > > > > > /usr/local/ncview/bin/ncview global.nc &
> > > > > > > > >
> > > > > > > >
> > > > > > > > I took a screenshot of that image and attached it as
> > > global.nc.png.
> > > > > > You
> > > > > > > > can see the data is smooshed with a chuck missing as
well.
> > This
> > > > > seems
> > > > > > > > problematic. If you'd like to pass on the GRIB data
that you
> > > used
> > > > in
> > > > > > > > running pcp_combine, we'd be happy to take a look at
that.
> > > > > > > >
> > > > > > > > Also, since you're at NCAR, we could also get together
to
> > discuss
> > > > how
> > > > > > > you'd
> > > > > > > > like to use MET and how we can help you with it. My
> colleague,
> > > > John
> > > > > > > Halley
> > > > > > > > Gotway, helped me through this work. He is our MET
expert.
> > > Please
> > > > > let
> > > > > > > us
> > > > > > > > know if you'd like to arrange a time to meet with us.
> > > > > > > >
> > > > > > > > I hope this is helpful. Please let us know what other
> > questions
> > > > you
> > > > > > > have.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Julie
> > > > > > > >
> > > > > > > > On Wed, Dec 4, 2019 at 2:48 PM lave at ucar.edu via RT <
> > > > > met_help at ucar.edu
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Wed Dec 04 14:48:26 2019: Request 93338 was acted
upon.
> > > > > > > > > Transaction: Ticket created by lave at ucar.edu
> > > > > > > > > Queue: met_help
> > > > > > > > > Subject: point_stat "unrecognized GRIB1 field";
> > > > > > > > > file_type="NETCDF_MET" in config file
> > > > > > > > > Owner: Nobody
> > > > > > > > > Requestors: lave at ucar.edu
> > > > > > > > > Status: new
> > > > > > > > > Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93338
> > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Good afternoon,
> > > > > > > > >
> > > > > > > > > I am having trouble running MET on netCDF data. I
used
> > > > pcp_combine
> > > > > > > > > -subtract to successfully create a netCDF file
> > > > > > > > > (APCP_wrfout_d05_2017120912_f007_f005.nc) containing
> > > > accumulation
> > > > > > data
> > > > > > > > for
> > > > > > > > > domain five of the AMPS evaluation area. I then used
> ascii2nc
> > > to
> > > > > > > create a
> > > > > > > > > netCDF file (BATG_ob1.nc) containing an accumulation
> > > observation.
> > > > > > "psc"
> > > > > > > > is
> > > > > > > > > my configuration file, in which I specify
> > > > 'field_type="NETCDF_MET"'
> > > > > > in
> > > > > > > > the
> > > > > > > > > fcst dictionary. With 'name="APCP"', running
point_stat
> > yields
> > > > the
> > > > > > > > > following:
> > > > > > > > >
> > > > > > > > > >point_stat APCP_wrfout_d05_2017120912_f007_f005.nc
> > > BATG_ob1.nc
> > > > > psc
> > > > > > > > >
> > > > > > > > > DEBUG 1: Default Config File:
> > > > > > > > >
> /usr/local/met-8.1.1/share/met/config/PointStatConfig_default
> > > > > > > > >
> > > > > > > > > DEBUG 1: User Config File: psc
> > > > > > > > >
> > > > > > > > > GSL_RNG_TYPE=mt19937
> > > > > > > > >
> > > > > > > > > GSL_RNG_SEED=18446744072084449373
> > > > > > > > >
> > > > > > > > > DEBUG 1: Forecast File: APCP_wrfout_d05_2017120912_
> > > f007_f005.nc
> > > > > > > > >
> > > > > > > > > DEBUG 1: Observation File: BATG_ob1.nc
> > > > > > > > >
> > > > > > > > > DEBUG 2:
> > > > > > > > >
> > > > > > > > > DEBUG 2:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--------------------------------------------------------------------------------
> > > > > > > > >
> > > > > > > > > DEBUG 2:
> > > > > > > > >
> > > > > > > > > DEBUG 2: Reading data for APCPA2.
> > > > > > > > >
> > > > > > > > > WARNING:
> > > > > > > > >
> > > > > > > > > WARNING: process_fcst_climo_files() -> no fields
matching
> > > APCPA2
> > > > > > found
> > > > > > > in
> > > > > > > > > file: APCP_wrfout_d05_2017120912_f007_f005.nc
> > > > > > > > >
> > > > > > > > > WARNING:
> > > > > > > > >
> > > > > > > > > ERROR :
> > > > > > > > >
> > > > > > > > > ERROR : process_fcst_climo_files() -> no requested
> forecast
> > > data
> > > > > > > found!
> > > > > > > > > Exiting...
> > > > > > > > >
> > > > > > > > > ERROR :
> > > > > > > > >
> > > > > > > > > yar:/d1/lave/antarctica> vi psc
> > > > > > > > >
> > > > > > > > > yar:/d1/lave/antarctica> point_stat
> > APCP_wrfout_d05_2017120912_
> > > > > > > > > f007_f005.nc
> > > > > > > > > BATG_ob1.nc psc
> > > > > > > > >
> > > > > > > > > DEBUG 1: Default Config File:
> > > > > > > > >
> /usr/local/met-8.1.1/share/met/config/PointStatConfig_default
> > > > > > > > >
> > > > > > > > > DEBUG 1: User Config File: psc
> > > > > > > > >
> > > > > > > > > ERROR :
> > > > > > > > >
> > > > > > > > > ERROR : VarInfoGrib::add_grib_code() ->
unrecognized GRIB1
> > > field
> > > > > > > > > abbreviation 'APCP_02' for table version 2
> > > > > > > > >
> > > > > > > > > ERROR :
> > > > > > > > >
> > > > > > > > > So, following the answer given in [Met_help] [
> > rt.rap.ucar.edu
> > > > > > > #85872], I
> > > > > > > > > adjusted my config file to include 'name="APCP_02"'.
> Running
> > > > > > point_stat
> > > > > > > > > then yielded:
> > > > > > > > >
> > > > > > > > > >point_stat APCP_wrfout_d05_2017120912_f007_f005.nc
> > > BATG_ob1.nc
> > > > > psc
> > > > > > > > >
> > > > > > > > > DEBUG 1: Default Config File:
> > > > > > > > >
> /usr/local/met-8.1.1/share/met/config/PointStatConfig_default
> > > > > > > > >
> > > > > > > > > DEBUG 1: User Config File: psc
> > > > > > > > >
> > > > > > > > > ERROR :
> > > > > > > > >
> > > > > > > > > ERROR : VarInfoGrib::add_grib_code() ->
unrecognized GRIB1
> > > field
> > > > > > > > > abbreviation 'APCP_02' for table version 2
> > > > > > > > > ERROR :
> > > > > > > > >
> > > > > > > > > What is going on here? It seems like point_stat is
> expecting
> > a
> > > > > GRIB1
> > > > > > > > file,
> > > > > > > > > though I specified the input to be a MET-generated
netCDF
> > > file. I
> > > > > > > > uploaded
> > > > > > > > > the three files to your ftp site.
> > > > > > > > >
> > > > > > > > > Sincerely,
> > > > > > > > > Joshua Lave
> > > > > > > > > --
> > > > > > > > > Joshua Lave
> > > > > > > > >
> > > > > > > > > *Associate Scientist INational Center for
Atmospheric
> > Research*
> > > > > > > > > *(303) 497-8258*
> > > > > > > > > *lave at ucar.edu <lave at ucar.edu>*
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Julie Prestopnik
> > > > > > > > Software Engineer
> > > > > > > > National Center for Atmospheric Research
> > > > > > > > Research Applications Laboratory
> > > > > > > > Phone: 303.497.8399
> > > > > > > > Email: jpresto at ucar.edu
> > > > > > > >
> > > > > > > > My working day may not be your working day. Please do
not
> feel
> > > > > obliged
> > > > > > > to
> > > > > > > > reply to this email outside of your normal working
hours.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Joshua Lave
> > > > > > >
> > > > > > > *Associate Scientist INational Center for Atmospheric
Research*
> > > > > > > *(303) 497-8258*
> > > > > > > *lave at ucar.edu <lave at ucar.edu>*
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Julie Prestopnik
> > > > > > Software Engineer
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > Phone: 303.497.8399
> > > > > > Email: jpresto at ucar.edu
> > > > > >
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Joshua Lave
> > > > >
> > > > > *Associate Scientist INational Center for Atmospheric
Research*
> > > > > *(303) 497-8258*
> > > > > *lave at ucar.edu <lave at ucar.edu>*
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > Joshua Lave
> > >
> > > *Associate Scientist INational Center for Atmospheric Research*
> > > *(303) 497-8258*
> > > *lave at ucar.edu <lave at ucar.edu>*
> > >
> > >
> >
> >
>
> --
> Joshua Lave
>
> *Associate Scientist INational Center for Atmospheric Research*
> *(303) 497-8258*
> *lave at ucar.edu <lave at ucar.edu>*
>
>
--
Julie Prestopnik
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
More information about the Met_help
mailing list