[Met_help] [rt.rap.ucar.edu #95731] History for Using MET to evaluate WRF with MADIS NETCDF data
Julie Prestopnik via RT
met_help at ucar.edu
Fri Jul 10 09:33:14 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Good morning MET help people!
I’m trying to use MET to evaluate precipitation skill in a bunch of model data we have from WRF and GFS runs. Initially, I’m focused on a high resolution WRF domain, at 1-hourly intervals in GriB2 format to compare with MADIS NETCDF output I have (also in 1-hourly files). I’ve read through the MET users manual to try to figure out what steps I have to perform and how to set up the configuration files, but I am either not understanding the manual or missing something.
1st question: do I really need to run the MADIS2NC tool on the NETCDF files? I’ve done so, but the output of MADIS2NC removes most of the data only leaving information about the station in the resulting .nc file.
Since I’m comparing 1-hourly WRF output to 1-hourly MADIS/METAR (or mesonet) data, I don’t need to run PCP_COMBINE (I will when looking at other 3-hourly WRF or GFS output).
Leading me to my second question: I cannot figure out a proper Grid_stat configuration file to compare the WRF data against the MADIS data. I have searched your website, googled the topic, and asked a number of others (with no luck). Is there a sample configuration file you can provide for comparing WRF 1-hourly files to MADIS NetCDF data?
3rd question: Am I doing this all wrong?
John Eylander
--
John Eylander
Hydrologic Systems Branch
Engineer Research and Development Center
US Army Corp of Engineers
John.B.Eylander at usace.army.mil<mailto:John.B.Eylander at usace.army.mil> (U)
John.B.Eylander at erdc.dren.mil<mailto:John.B.Eylander at erdc.dren.mil> (U – RDE)
John.B.Eylander.civ at mail.smil.mil<mailto:John.B.Eylander.civ at mail.smil.mil> (S)
John.B.Eylander.civ at army.ic.gov<mailto:John.B.Eylander.civ at army.ic.gov> (JWICS)
PH: 603-646-4188
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Using MET to evaluate WRF with MADIS NETCDF data
From: George McCabe
Time: Thu Jun 25 12:09:03 2020
Hi John,
I see you have questions about processing MADIS data using the MET
tools. I
have assigned this ticket to John Halley Gotway, a software engineer
familiar with these tools. Please allow a few business days for a
response.
Thanks,
George
On Thu, Jun 25, 2020 at 5:46 AM Eylander, John B ERDC-RDE-CRREL-NH CIV
via
RT <met_help at ucar.edu> wrote:
>
> Thu Jun 25 05:46:04 2020: Request 95731 was acted upon.
> Transaction: Ticket created by John.B.Eylander at erdc.dren.mil
> Queue: met_help
> Subject: Using MET to evaluate WRF with MADIS NETCDF data
> Owner: Nobody
> Requestors: John.B.Eylander at erdc.dren.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731 >
>
>
> Good morning MET help people!
>
> I’m trying to use MET to evaluate precipitation skill in a bunch of
model
> data we have from WRF and GFS runs. Initially, I’m focused on a
high
> resolution WRF domain, at 1-hourly intervals in GriB2 format to
compare
> with MADIS NETCDF output I have (also in 1-hourly files). I’ve read
> through the MET users manual to try to figure out what steps I have
to
> perform and how to set up the configuration files, but I am either
not
> understanding the manual or missing something.
>
> 1st question: do I really need to run the MADIS2NC tool on the
NETCDF
> files? I’ve done so, but the output of MADIS2NC removes most of the
data
> only leaving information about the station in the resulting .nc
file.
>
> Since I’m comparing 1-hourly WRF output to 1-hourly MADIS/METAR (or
> mesonet) data, I don’t need to run PCP_COMBINE (I will when looking
at
> other 3-hourly WRF or GFS output).
>
> Leading me to my second question: I cannot figure out a proper
Grid_stat
> configuration file to compare the WRF data against the MADIS data.
I have
> searched your website, googled the topic, and asked a number of
others
> (with no luck). Is there a sample configuration file you can
provide for
> comparing WRF 1-hourly files to MADIS NetCDF data?
>
> 3rd question: Am I doing this all wrong?
>
> John Eylander
>
>
> --
>
> John Eylander
> Hydrologic Systems Branch
> Engineer Research and Development Center
> US Army Corp of Engineers
>
>
John.B.Eylander at usace.army.mil<mailto:John.B.Eylander at usace.army.mil>
(U)
> John.B.Eylander at erdc.dren.mil<mailto:John.B.Eylander at erdc.dren.mil>
(U –
> RDE)
>
>
John.B.Eylander.civ at mail.smil.mil<mailto:John.B.Eylander.civ at mail.smil.mil>
> (S)
>
John.B.Eylander.civ at army.ic.gov<mailto:John.B.Eylander.civ at army.ic.gov>
> (JWICS)
>
> PH: 603-646-4188
>
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #95731] Using MET to evaluate WRF with MADIS NETCDF data
From: Eylander, John B ERDC-RDE-CRREL-NH CIV
Time: Thu Jun 25 12:19:37 2020
Thanks!
On 6/25/20, 2:09 PM, "George McCabe via RT" <met_help at ucar.edu>
wrote:
Hi John,
I see you have questions about processing MADIS data using the MET
tools. I
have assigned this ticket to John Halley Gotway, a software
engineer
familiar with these tools. Please allow a few business days for a
response.
Thanks,
George
On Thu, Jun 25, 2020 at 5:46 AM Eylander, John B ERDC-RDE-CRREL-NH
CIV via
RT <met_help at ucar.edu> wrote:
>
> Thu Jun 25 05:46:04 2020: Request 95731 was acted upon.
> Transaction: Ticket created by John.B.Eylander at erdc.dren.mil
> Queue: met_help
> Subject: Using MET to evaluate WRF with MADIS NETCDF data
> Owner: Nobody
> Requestors: John.B.Eylander at erdc.dren.mil
> Status: new
> Ticket <URL:
Blockedhttps://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked
>
>
>
> Good morning MET help people!
>
> I’m trying to use MET to evaluate precipitation skill in a bunch
of model
> data we have from WRF and GFS runs. Initially, I’m focused on a
high
> resolution WRF domain, at 1-hourly intervals in GriB2 format to
compare
> with MADIS NETCDF output I have (also in 1-hourly files). I’ve
read
> through the MET users manual to try to figure out what steps I
have to
> perform and how to set up the configuration files, but I am
either not
> understanding the manual or missing something.
>
> 1st question: do I really need to run the MADIS2NC tool on the
NETCDF
> files? I’ve done so, but the output of MADIS2NC removes most of
the data
> only leaving information about the station in the resulting .nc
file.
>
> Since I’m comparing 1-hourly WRF output to 1-hourly MADIS/METAR
(or
> mesonet) data, I don’t need to run PCP_COMBINE (I will when
looking at
> other 3-hourly WRF or GFS output).
>
> Leading me to my second question: I cannot figure out a proper
Grid_stat
> configuration file to compare the WRF data against the MADIS
data. I have
> searched your website, googled the topic, and asked a number of
others
> (with no luck). Is there a sample configuration file you can
provide for
> comparing WRF 1-hourly files to MADIS NetCDF data?
>
> 3rd question: Am I doing this all wrong?
>
> John Eylander
>
>
> --
>
> John Eylander
> Hydrologic Systems Branch
> Engineer Research and Development Center
> US Army Corp of Engineers
>
>
John.B.Eylander at usace.army.mil<mailto:John.B.Eylander at usace.army.mil>
(U)
>
John.B.Eylander at erdc.dren.mil<mailto:John.B.Eylander at erdc.dren.mil> (U
–
> RDE)
>
>
John.B.Eylander.civ at mail.smil.mil<mailto:John.B.Eylander.civ at mail.smil.mil>
> (S)
>
John.B.Eylander.civ at army.ic.gov<mailto:John.B.Eylander.civ at army.ic.gov>
> (JWICS)
>
> PH: 603-646-4188
>
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Using MET to evaluate WRF with MADIS NETCDF data
From: John Halley Gotway
Time: Thu Jun 25 15:24:22 2020
Hello John,
I see you have a few questions about the MET software.
(1) Yes, you really need to run the madis2nc tool on the MADIS files.
I
realize that this seems silly since the MADIS data is already in
NetCDF
format. The madis2nc tool is essentially a reformatter. It can also
filter
out obs based on QC flags and can compute time summaries of the point
observations. But the main purpose is to reformat the point
observations
into a common NetCDF format that the other MET tools (point-stat and
ensemble-stat) expect.
We have done a nice job in MET supporting different gridded data files
in
their native format, but we decided not to do that with point
observations.
Instead of reading them directly, we reformat them into a common
format
using pb2nc, madis2nc, ascii2nc, lidar2nc, and so on. However we may
redesign that in the future and support the point obs in their native
formats. But for now madis2nc is necessary.
If your madis2nc output doesn't contain the expected obs, please send
me
the command you're running, along with a sample data file.
(2) You should check your WRF output files. You state that you have
hourly
WRF output, but WRF often outputs a runtime accumulation. So that
output
file for a 6-hour forecast contains 0 to 6 hours of accumulation. At
7-hours you'd get 0 to 7 hours of accumulation. However, WRF really
can be
configured to dump an hourly precip bucket, but that's not the
default.
If you actually have a runtime accumulation, you'd run pcp_combine to
compute the difference. For example, 7-hour accum - 6-hour accum = 1-
hr
accum between 6 and 7 hours.
(3) WRF and GFS are gridded forecasts. MADIS are point observations.
You
run the Point-Stat tool to verify against point observations, not the
Grid-Stat tool.
Two other things to mention...
We typically recommend that users post-process their WRF output using
the
Unified Post-Processor (
https://dtcenter.org/community-code/unified-post-processor-upp). That
reads
WRF NetCDF output, de-staggers the staggered variables (winds),
derives
requested variables, interpolates from WRF's native vertical
coordinate to
pressure, and writes GRIB1 or GRIB2 output files. If you really are
only
interested in precip at the surface... and not winds or any variables
on
pressure levels... then technically you do not have to run UPP. But I
would
still recommend it.
METplus is a suite of python wrappers intended to make it easier to
automate calls to the MET tools. The intention was to make getting up
and
running with MET easier, but it's still relatively new. I don't know
whether or not we have an existing use case which verifies WRF output
against MADIS obs, but I'll let George chime in about that.
Thanks,
John Halley Gotway
On Thu, Jun 25, 2020 at 12:19 PM Eylander, John B ERDC-RDE-CRREL-NH
CIV via
RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731 >
>
> Thanks!
>
> On 6/25/20, 2:09 PM, "George McCabe via RT" <met_help at ucar.edu>
wrote:
>
> Hi John,
>
> I see you have questions about processing MADIS data using the
MET
> tools. I
> have assigned this ticket to John Halley Gotway, a software
engineer
> familiar with these tools. Please allow a few business days for
a
> response.
>
> Thanks,
> George
>
> On Thu, Jun 25, 2020 at 5:46 AM Eylander, John B ERDC-RDE-CRREL-
NH CIV
> via
> RT <met_help at ucar.edu> wrote:
>
> >
> > Thu Jun 25 05:46:04 2020: Request 95731 was acted upon.
> > Transaction: Ticket created by John.B.Eylander at erdc.dren.mil
> > Queue: met_help
> > Subject: Using MET to evaluate WRF with MADIS NETCDF data
> > Owner: Nobody
> > Requestors: John.B.Eylander at erdc.dren.mil
> > Status: new
> > Ticket <URL: Blockedhttps://
> rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> >
> >
> > Good morning MET help people!
> >
> > I’m trying to use MET to evaluate precipitation skill in a
bunch of
> model
> > data we have from WRF and GFS runs. Initially, I’m focused on
a high
> > resolution WRF domain, at 1-hourly intervals in GriB2 format
to
> compare
> > with MADIS NETCDF output I have (also in 1-hourly files).
I’ve read
> > through the MET users manual to try to figure out what steps I
have
> to
> > perform and how to set up the configuration files, but I am
either
> not
> > understanding the manual or missing something.
> >
> > 1st question: do I really need to run the MADIS2NC tool on
the
> NETCDF
> > files? I’ve done so, but the output of MADIS2NC removes most
of the
> data
> > only leaving information about the station in the resulting
.nc file.
> >
> > Since I’m comparing 1-hourly WRF output to 1-hourly
MADIS/METAR (or
> > mesonet) data, I don’t need to run PCP_COMBINE (I will when
looking
> at
> > other 3-hourly WRF or GFS output).
> >
> > Leading me to my second question: I cannot figure out a
proper
> Grid_stat
> > configuration file to compare the WRF data against the MADIS
data.
> I have
> > searched your website, googled the topic, and asked a number
of
> others
> > (with no luck). Is there a sample configuration file you can
> provide for
> > comparing WRF 1-hourly files to MADIS NetCDF data?
> >
> > 3rd question: Am I doing this all wrong?
> >
> > John Eylander
> >
> >
> > --
> >
> > John Eylander
> > Hydrologic Systems Branch
> > Engineer Research and Development Center
> > US Army Corp of Engineers
> >
> >
John.B.Eylander at usace.army.mil<mailto:John.B.Eylander at usace.army.mil>
> (U)
> >
John.B.Eylander at erdc.dren.mil<mailto:John.B.Eylander at erdc.dren.mil>
> (U –
> > RDE)
> >
> > John.B.Eylander.civ at mail.smil.mil<mailto:
> John.B.Eylander.civ at mail.smil.mil>
> > (S)
> > John.B.Eylander.civ at army.ic.gov<mailto:
> John.B.Eylander.civ at army.ic.gov>
> > (JWICS)
> >
> > PH: 603-646-4188
> >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged
> to
> reply to this email outside of your normal working hours.
>
>
>
>
>
>
------------------------------------------------
Subject: Using MET to evaluate WRF with MADIS NETCDF data
From: George McCabe
Time: Thu Jun 25 17:32:32 2020
MADIS2NC is not currently included in the METplus wrappers, so I don't
think there is a use case using that data unless it was previously run
through MADIS2NC before added to the use cases in the repository.
On Thu, Jun 25, 2020 at 3:24 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731 >
>
> Hello John,
>
> I see you have a few questions about the MET software.
>
> (1) Yes, you really need to run the madis2nc tool on the MADIS
files. I
> realize that this seems silly since the MADIS data is already in
NetCDF
> format. The madis2nc tool is essentially a reformatter. It can also
filter
> out obs based on QC flags and can compute time summaries of the
point
> observations. But the main purpose is to reformat the point
observations
> into a common NetCDF format that the other MET tools (point-stat and
> ensemble-stat) expect.
>
> We have done a nice job in MET supporting different gridded data
files in
> their native format, but we decided not to do that with point
observations.
> Instead of reading them directly, we reformat them into a common
format
> using pb2nc, madis2nc, ascii2nc, lidar2nc, and so on. However we may
> redesign that in the future and support the point obs in their
native
> formats. But for now madis2nc is necessary.
>
> If your madis2nc output doesn't contain the expected obs, please
send me
> the command you're running, along with a sample data file.
>
> (2) You should check your WRF output files. You state that you have
hourly
> WRF output, but WRF often outputs a runtime accumulation. So that
output
> file for a 6-hour forecast contains 0 to 6 hours of accumulation. At
> 7-hours you'd get 0 to 7 hours of accumulation. However, WRF really
can be
> configured to dump an hourly precip bucket, but that's not the
default.
>
> If you actually have a runtime accumulation, you'd run pcp_combine
to
> compute the difference. For example, 7-hour accum - 6-hour accum =
1-hr
> accum between 6 and 7 hours.
>
> (3) WRF and GFS are gridded forecasts. MADIS are point observations.
You
> run the Point-Stat tool to verify against point observations, not
the
> Grid-Stat tool.
>
> Two other things to mention...
>
> We typically recommend that users post-process their WRF output
using the
> Unified Post-Processor (
> https://dtcenter.org/community-code/unified-post-processor-upp).
That
> reads
> WRF NetCDF output, de-staggers the staggered variables (winds),
derives
> requested variables, interpolates from WRF's native vertical
coordinate to
> pressure, and writes GRIB1 or GRIB2 output files. If you really are
only
> interested in precip at the surface... and not winds or any
variables on
> pressure levels... then technically you do not have to run UPP. But
I would
> still recommend it.
>
> METplus is a suite of python wrappers intended to make it easier to
> automate calls to the MET tools. The intention was to make getting
up and
> running with MET easier, but it's still relatively new. I don't know
> whether or not we have an existing use case which verifies WRF
output
> against MADIS obs, but I'll let George chime in about that.
>
> Thanks,
> John Halley Gotway
>
> On Thu, Jun 25, 2020 at 12:19 PM Eylander, John B ERDC-RDE-CRREL-NH
CIV via
> RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731 >
> >
> > Thanks!
> >
> > On 6/25/20, 2:09 PM, "George McCabe via RT" <met_help at ucar.edu>
wrote:
> >
> > Hi John,
> >
> > I see you have questions about processing MADIS data using the
MET
> > tools. I
> > have assigned this ticket to John Halley Gotway, a software
engineer
> > familiar with these tools. Please allow a few business days
for a
> > response.
> >
> > Thanks,
> > George
> >
> > On Thu, Jun 25, 2020 at 5:46 AM Eylander, John B ERDC-RDE-
CRREL-NH
> CIV
> > via
> > RT <met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Jun 25 05:46:04 2020: Request 95731 was acted upon.
> > > Transaction: Ticket created by John.B.Eylander at erdc.dren.mil
> > > Queue: met_help
> > > Subject: Using MET to evaluate WRF with MADIS NETCDF
data
> > > Owner: Nobody
> > > Requestors: John.B.Eylander at erdc.dren.mil
> > > Status: new
> > > Ticket <URL: Blockedhttps://
> > rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> > >
> > >
> > > Good morning MET help people!
> > >
> > > I’m trying to use MET to evaluate precipitation skill in a
bunch of
> > model
> > > data we have from WRF and GFS runs. Initially, I’m focused
on a
> high
> > > resolution WRF domain, at 1-hourly intervals in GriB2 format
to
> > compare
> > > with MADIS NETCDF output I have (also in 1-hourly files).
I’ve
> read
> > > through the MET users manual to try to figure out what steps
I have
> > to
> > > perform and how to set up the configuration files, but I am
either
> > not
> > > understanding the manual or missing something.
> > >
> > > 1st question: do I really need to run the MADIS2NC tool on
the
> > NETCDF
> > > files? I’ve done so, but the output of MADIS2NC removes
most of
> the
> > data
> > > only leaving information about the station in the resulting
.nc
> file.
> > >
> > > Since I’m comparing 1-hourly WRF output to 1-hourly
MADIS/METAR (or
> > > mesonet) data, I don’t need to run PCP_COMBINE (I will when
looking
> > at
> > > other 3-hourly WRF or GFS output).
> > >
> > > Leading me to my second question: I cannot figure out a
proper
> > Grid_stat
> > > configuration file to compare the WRF data against the MADIS
data.
> > I have
> > > searched your website, googled the topic, and asked a number
of
> > others
> > > (with no luck). Is there a sample configuration file you
can
> > provide for
> > > comparing WRF 1-hourly files to MADIS NetCDF data?
> > >
> > > 3rd question: Am I doing this all wrong?
> > >
> > > John Eylander
> > >
> > >
> > > --
> > >
> > > John Eylander
> > > Hydrologic Systems Branch
> > > Engineer Research and Development Center
> > > US Army Corp of Engineers
> > >
> > > John.B.Eylander at usace.army.mil<mailto:
> John.B.Eylander at usace.army.mil>
> > (U)
> > >
John.B.Eylander at erdc.dren.mil<mailto:John.B.Eylander at erdc.dren.mil
> >
> > (U –
> > > RDE)
> > >
> > > John.B.Eylander.civ at mail.smil.mil<mailto:
> > John.B.Eylander.civ at mail.smil.mil>
> > > (S)
> > > John.B.Eylander.civ at army.ic.gov<mailto:
> > John.B.Eylander.civ at army.ic.gov>
> > > (JWICS)
> > >
> > > PH: 603-646-4188
> > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
> obliged
> > to
> > reply to this email outside of your normal working hours.
> >
> >
> >
> >
> >
> >
>
>
--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Using MET to evaluate WRF with MADIS NETCDF data
From: Eylander, John B ERDC-RDE-CRREL-NH CIV
Time: Fri Jun 26 06:25:20 2020
Thanks for getting back to me.
I ran MADIS2NC with the following command (within a script).
$(${MET_BIN}/madis2nc ${infile} ${outfile} -type metar -log
${infile}.log -v 3)
I've included a sample input and output. The .nc file is the output.
Regarding WRF accumulation, I hadn't actually looked at the WRF
accumulated precip results yet but acknowledge I should before going
further to see if I should run PCP_combine to generate the 1-hourly
fields. Thanks for pointing that out.
I will run the point-ob tool. The flow-chart diagram I found online
suggested that I run grid-stat, but after finding a newer diagram I
see that it states I should run point-stat. Maybe initially I was
looking at an older/out of date diagram? Or maybe I'm crazy. I
thought it strange to run grid-stat since MADIS is a point-ob, but I
was just trying to go according to what I found in the online
tutorial. Thanks.
John
On 6/25/20, 5:24 PM, "John Halley Gotway via RT" <met_help at ucar.edu>
wrote:
Hello John,
I see you have a few questions about the MET software.
(1) Yes, you really need to run the madis2nc tool on the MADIS
files. I
realize that this seems silly since the MADIS data is already in
NetCDF
format. The madis2nc tool is essentially a reformatter. It can
also filter
out obs based on QC flags and can compute time summaries of the
point
observations. But the main purpose is to reformat the point
observations
into a common NetCDF format that the other MET tools (point-stat
and
ensemble-stat) expect.
We have done a nice job in MET supporting different gridded data
files in
their native format, but we decided not to do that with point
observations.
Instead of reading them directly, we reformat them into a common
format
using pb2nc, madis2nc, ascii2nc, lidar2nc, and so on. However we
may
redesign that in the future and support the point obs in their
native
formats. But for now madis2nc is necessary.
If your madis2nc output doesn't contain the expected obs, please
send me
the command you're running, along with a sample data file.
(2) You should check your WRF output files. You state that you
have hourly
WRF output, but WRF often outputs a runtime accumulation. So that
output
file for a 6-hour forecast contains 0 to 6 hours of accumulation.
At
7-hours you'd get 0 to 7 hours of accumulation. However, WRF
really can be
configured to dump an hourly precip bucket, but that's not the
default.
If you actually have a runtime accumulation, you'd run pcp_combine
to
compute the difference. For example, 7-hour accum - 6-hour accum =
1-hr
accum between 6 and 7 hours.
(3) WRF and GFS are gridded forecasts. MADIS are point
observations. You
run the Point-Stat tool to verify against point observations, not
the
Grid-Stat tool.
Two other things to mention...
We typically recommend that users post-process their WRF output
using the
Unified Post-Processor (
Blockedhttps://dtcenter.org/community-code/unified-post-processor-
upp)Blocked. That reads
WRF NetCDF output, de-staggers the staggered variables (winds),
derives
requested variables, interpolates from WRF's native vertical
coordinate to
pressure, and writes GRIB1 or GRIB2 output files. If you really
are only
interested in precip at the surface... and not winds or any
variables on
pressure levels... then technically you do not have to run UPP.
But I would
still recommend it.
METplus is a suite of python wrappers intended to make it easier
to
automate calls to the MET tools. The intention was to make getting
up and
running with MET easier, but it's still relatively new. I don't
know
whether or not we have an existing use case which verifies WRF
output
against MADIS obs, but I'll let George chime in about that.
Thanks,
John Halley Gotway
On Thu, Jun 25, 2020 at 12:19 PM Eylander, John B ERDC-RDE-CRREL-
NH CIV via
RT <met_help at ucar.edu> wrote:
>
> <URL:
Blockedhttps://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked
>
>
> Thanks!
>
> On 6/25/20, 2:09 PM, "George McCabe via RT" <met_help at ucar.edu>
wrote:
>
> Hi John,
>
> I see you have questions about processing MADIS data using
the MET
> tools. I
> have assigned this ticket to John Halley Gotway, a software
engineer
> familiar with these tools. Please allow a few business days
for a
> response.
>
> Thanks,
> George
>
> On Thu, Jun 25, 2020 at 5:46 AM Eylander, John B ERDC-RDE-
CRREL-NH CIV
> via
> RT <met_help at ucar.edu> wrote:
>
> >
> > Thu Jun 25 05:46:04 2020: Request 95731 was acted upon.
> > Transaction: Ticket created by
John.B.Eylander at erdc.dren.mil
> > Queue: met_help
> > Subject: Using MET to evaluate WRF with MADIS NETCDF
data
> > Owner: Nobody
> > Requestors: John.B.Eylander at erdc.dren.mil
> > Status: new
> > Ticket <URL: Blockedhttps://
> rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> >
> >
> > Good morning MET help people!
> >
> > I’m trying to use MET to evaluate precipitation skill in a
bunch of
> model
> > data we have from WRF and GFS runs. Initially, I’m
focused on a high
> > resolution WRF domain, at 1-hourly intervals in GriB2
format to
> compare
> > with MADIS NETCDF output I have (also in 1-hourly files).
I’ve read
> > through the MET users manual to try to figure out what
steps I have
> to
> > perform and how to set up the configuration files, but I
am either
> not
> > understanding the manual or missing something.
> >
> > 1st question: do I really need to run the MADIS2NC tool
on the
> NETCDF
> > files? I’ve done so, but the output of MADIS2NC removes
most of the
> data
> > only leaving information about the station in the
resulting .nc file.
> >
> > Since I’m comparing 1-hourly WRF output to 1-hourly
MADIS/METAR (or
> > mesonet) data, I don’t need to run PCP_COMBINE (I will
when looking
> at
> > other 3-hourly WRF or GFS output).
> >
> > Leading me to my second question: I cannot figure out a
proper
> Grid_stat
> > configuration file to compare the WRF data against the
MADIS data.
> I have
> > searched your website, googled the topic, and asked a
number of
> others
> > (with no luck). Is there a sample configuration file you
can
> provide for
> > comparing WRF 1-hourly files to MADIS NetCDF data?
> >
> > 3rd question: Am I doing this all wrong?
> >
> > John Eylander
> >
> >
> > --
> >
> > John Eylander
> > Hydrologic Systems Branch
> > Engineer Research and Development Center
> > US Army Corp of Engineers
> >
> >
John.B.Eylander at usace.army.mil<mailto:John.B.Eylander at usace.army.mil>
> (U)
> >
John.B.Eylander at erdc.dren.mil<mailto:John.B.Eylander at erdc.dren.mil>
> (U –
> > RDE)
> >
> > John.B.Eylander.civ at mail.smil.mil<mailto:
> John.B.Eylander.civ at mail.smil.mil>
> > (S)
> > John.B.Eylander.civ at army.ic.gov<mailto:
> John.B.Eylander.civ at army.ic.gov>
> > (JWICS)
> >
> > PH: 603-646-4188
> >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not
feel obliged
> to
> reply to this email outside of your normal working hours.
>
>
>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #95731] Using MET to evaluate WRF with MADIS NETCDF data
From: Eylander, John B ERDC-RDE-CRREL-NH CIV
Time: Tue Jun 30 09:06:46 2020
I am now trying to execute PCP_COMBINE on the WRF GriB2 (already post-
processed) data to separate the ACPC fields into 1-hour accumulation
field. Based on my reading of the MET manual and after exploring
several online scenarios I found, I thought the execution of
PCP_COMBINE should look like so:
${MET_BIN}/pcp_combine -subtract
nam.t18z.awphys.f84.tm00.20200412.grib2 84
nam.t18z.awphys.f81.tm00.20200412.grib2 81 $HOME/test.out -v 3
However, I get the following error/diagnostic output when trying to
run pcp_combine:
DEBUG 2: Since the "-field" command line option was not used, parsing
the command line arguments a list of files, each followed by a
configuration string.
DEBUG 1: Reading input file: nam.t18z.awphys.f84.tm00.20200412.grib2
DEBUG 1: Reading data (name="APCP"; level="A84";) from input file:
nam.t18z.awphys.f84.tm00.20200412.grib2
WARNING:
WARNING: MetGrib2DataFile::data_plane() -> No matching record found
for 'APCP/A84'
WARNING:
ERROR :
ERROR : get_field() -> can't get data plane from file
"nam.t18z.awphys.f84.tm00.20200412.grib2"
ERROR :
What I can't understand is why the "level" field is set to "A84".
>From what I understand of the manual, the APCP field should be
automatically able to find APCP in the GriB2 output. However, this is
not happening because the level of APCP should be "surface", not
"A84".
When I try to explicitly set the configuration to set the level to
"surface", I still get an error:
$MET_BIN/pcp_combine -subtract nam.t18z.awphys.f81.tm00.20200412.grib2
'name="APCP"; level="surface";'
nam.t18z.awphys.f84.tm00.20200412.grib2 'name="APCP";
level="surface";' $HOME/test.out -v 3
DEBUG 2: Since the "-field" command line option was not used, parsing
the command line arguments a list of files, each followed by a
configuration string.
DEBUG 1: Reading input file: nam.t18z.awphys.f81.tm00.20200412.grib2
DEBUG 1: Reading data (name="APCP"; level="surface";) from input file:
nam.t18z.awphys.f81.tm00.20200412.grib2
ERROR :
ERROR : VarInfo::set_level_info_grib() - failed to parse level string
'surface'
ERROR :
Thanks in advance for any help you can provide.
John Eylander
On 6/25/20, 5:24 PM, "John Halley Gotway via RT" <met_help at ucar.edu>
wrote:
Hello John,
I see you have a few questions about the MET software.
(1) Yes, you really need to run the madis2nc tool on the MADIS
files. I
realize that this seems silly since the MADIS data is already in
NetCDF
format. The madis2nc tool is essentially a reformatter. It can
also filter
out obs based on QC flags and can compute time summaries of the
point
observations. But the main purpose is to reformat the point
observations
into a common NetCDF format that the other MET tools (point-stat
and
ensemble-stat) expect.
We have done a nice job in MET supporting different gridded data
files in
their native format, but we decided not to do that with point
observations.
Instead of reading them directly, we reformat them into a common
format
using pb2nc, madis2nc, ascii2nc, lidar2nc, and so on. However we
may
redesign that in the future and support the point obs in their
native
formats. But for now madis2nc is necessary.
If your madis2nc output doesn't contain the expected obs, please
send me
the command you're running, along with a sample data file.
(2) You should check your WRF output files. You state that you
have hourly
WRF output, but WRF often outputs a runtime accumulation. So that
output
file for a 6-hour forecast contains 0 to 6 hours of accumulation.
At
7-hours you'd get 0 to 7 hours of accumulation. However, WRF
really can be
configured to dump an hourly precip bucket, but that's not the
default.
If you actually have a runtime accumulation, you'd run pcp_combine
to
compute the difference. For example, 7-hour accum - 6-hour accum =
1-hr
accum between 6 and 7 hours.
(3) WRF and GFS are gridded forecasts. MADIS are point
observations. You
run the Point-Stat tool to verify against point observations, not
the
Grid-Stat tool.
Two other things to mention...
We typically recommend that users post-process their WRF output
using the
Unified Post-Processor (
Blockedhttps://dtcenter.org/community-code/unified-post-processor-
upp)Blocked. That reads
WRF NetCDF output, de-staggers the staggered variables (winds),
derives
requested variables, interpolates from WRF's native vertical
coordinate to
pressure, and writes GRIB1 or GRIB2 output files. If you really
are only
interested in precip at the surface... and not winds or any
variables on
pressure levels... then technically you do not have to run UPP.
But I would
still recommend it.
METplus is a suite of python wrappers intended to make it easier
to
automate calls to the MET tools. The intention was to make getting
up and
running with MET easier, but it's still relatively new. I don't
know
whether or not we have an existing use case which verifies WRF
output
against MADIS obs, but I'll let George chime in about that.
Thanks,
John Halley Gotway
On Thu, Jun 25, 2020 at 12:19 PM Eylander, John B ERDC-RDE-CRREL-
NH CIV via
RT <met_help at ucar.edu> wrote:
>
> <URL:
Blockedhttps://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked
>
>
> Thanks!
>
> On 6/25/20, 2:09 PM, "George McCabe via RT" <met_help at ucar.edu>
wrote:
>
> Hi John,
>
> I see you have questions about processing MADIS data using
the MET
> tools. I
> have assigned this ticket to John Halley Gotway, a software
engineer
> familiar with these tools. Please allow a few business days
for a
> response.
>
> Thanks,
> George
>
> On Thu, Jun 25, 2020 at 5:46 AM Eylander, John B ERDC-RDE-
CRREL-NH CIV
> via
> RT <met_help at ucar.edu> wrote:
>
> >
> > Thu Jun 25 05:46:04 2020: Request 95731 was acted upon.
> > Transaction: Ticket created by
John.B.Eylander at erdc.dren.mil
> > Queue: met_help
> > Subject: Using MET to evaluate WRF with MADIS NETCDF
data
> > Owner: Nobody
> > Requestors: John.B.Eylander at erdc.dren.mil
> > Status: new
> > Ticket <URL: Blockedhttps://
> rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> >
> >
> > Good morning MET help people!
> >
> > I’m trying to use MET to evaluate precipitation skill in a
bunch of
> model
> > data we have from WRF and GFS runs. Initially, I’m
focused on a high
> > resolution WRF domain, at 1-hourly intervals in GriB2
format to
> compare
> > with MADIS NETCDF output I have (also in 1-hourly files).
I’ve read
> > through the MET users manual to try to figure out what
steps I have
> to
> > perform and how to set up the configuration files, but I
am either
> not
> > understanding the manual or missing something.
> >
> > 1st question: do I really need to run the MADIS2NC tool
on the
> NETCDF
> > files? I’ve done so, but the output of MADIS2NC removes
most of the
> data
> > only leaving information about the station in the
resulting .nc file.
> >
> > Since I’m comparing 1-hourly WRF output to 1-hourly
MADIS/METAR (or
> > mesonet) data, I don’t need to run PCP_COMBINE (I will
when looking
> at
> > other 3-hourly WRF or GFS output).
> >
> > Leading me to my second question: I cannot figure out a
proper
> Grid_stat
> > configuration file to compare the WRF data against the
MADIS data.
> I have
> > searched your website, googled the topic, and asked a
number of
> others
> > (with no luck). Is there a sample configuration file you
can
> provide for
> > comparing WRF 1-hourly files to MADIS NetCDF data?
> >
> > 3rd question: Am I doing this all wrong?
> >
> > John Eylander
> >
> >
> > --
> >
> > John Eylander
> > Hydrologic Systems Branch
> > Engineer Research and Development Center
> > US Army Corp of Engineers
> >
> >
John.B.Eylander at usace.army.mil<mailto:John.B.Eylander at usace.army.mil>
> (U)
> >
John.B.Eylander at erdc.dren.mil<mailto:John.B.Eylander at erdc.dren.mil>
> (U –
> > RDE)
> >
> > John.B.Eylander.civ at mail.smil.mil<mailto:
> John.B.Eylander.civ at mail.smil.mil>
> > (S)
> > John.B.Eylander.civ at army.ic.gov<mailto:
> John.B.Eylander.civ at army.ic.gov>
> > (JWICS)
> >
> > PH: 603-646-4188
> >
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not
feel obliged
> to
> reply to this email outside of your normal working hours.
>
>
>
>
>
>
------------------------------------------------
Subject: Using MET to evaluate WRF with MADIS NETCDF data
From: John Halley Gotway
Time: Tue Jun 30 09:55:13 2020
John,
When working with GRIB1 or GRIB2 files, you request data from them by
specifying a "name" and "level". Just set to the "name" to the GRIB
abbreviation for the data... e.g. name = "APCP" for accumulated
precip. For
GRIB data, the "level" string starts with A, P, Z, L, or R. A is for
accumulation interval, P is for pressure level, Z is for height level,
L is
for a generic level type, and R is for a specific record number.
So name="APCP"; level="A84"; would request a field of 84-hour
accumulated
precip.
Just run wgrib2 on your data file to see what precip accumulation is
present there:
wgrib2 nam.t18z.awphys.f84.tm00.20200412.grib2 | grep APCP
And take a look at line 609 of this README file for some more info on
GRIB
levels.
Thanks,
John
On Tue, Jun 30, 2020 at 9:07 AM Eylander, John B ERDC-RDE-CRREL-NH CIV
via
RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731 >
>
> I am now trying to execute PCP_COMBINE on the WRF GriB2 (already
> post-processed) data to separate the ACPC fields into 1-hour
accumulation
> field. Based on my reading of the MET manual and after exploring
several
> online scenarios I found, I thought the execution of PCP_COMBINE
should
> look like so:
>
> ${MET_BIN}/pcp_combine -subtract
nam.t18z.awphys.f84.tm00.20200412.grib2
> 84 nam.t18z.awphys.f81.tm00.20200412.grib2 81 $HOME/test.out -v
3
>
> However, I get the following error/diagnostic output when trying to
run
> pcp_combine:
>
> DEBUG 2: Since the "-field" command line option was not used,
parsing the
> command line arguments a list of files, each followed by a
configuration
> string.
> DEBUG 1: Reading input file: nam.t18z.awphys.f84.tm00.20200412.grib2
> DEBUG 1: Reading data (name="APCP"; level="A84";) from input file:
> nam.t18z.awphys.f84.tm00.20200412.grib2
> WARNING:
> WARNING: MetGrib2DataFile::data_plane() -> No matching record found
for
> 'APCP/A84'
> WARNING:
> ERROR :
> ERROR : get_field() -> can't get data plane from file
> "nam.t18z.awphys.f84.tm00.20200412.grib2"
> ERROR :
>
> What I can't understand is why the "level" field is set to "A84".
From
> what I understand of the manual, the APCP field should be
automatically
> able to find APCP in the GriB2 output. However, this is not
happening
> because the level of APCP should be "surface", not "A84".
>
> When I try to explicitly set the configuration to set the level to
> "surface", I still get an error:
>
> $MET_BIN/pcp_combine -subtract
nam.t18z.awphys.f81.tm00.20200412.grib2
> 'name="APCP"; level="surface";'
nam.t18z.awphys.f84.tm00.20200412.grib2
> 'name="APCP"; level="surface";' $HOME/test.out -v 3
>
> DEBUG 2: Since the "-field" command line option was not used,
parsing the
> command line arguments a list of files, each followed by a
configuration
> string.
> DEBUG 1: Reading input file: nam.t18z.awphys.f81.tm00.20200412.grib2
> DEBUG 1: Reading data (name="APCP"; level="surface";) from input
file:
> nam.t18z.awphys.f81.tm00.20200412.grib2
> ERROR :
> ERROR : VarInfo::set_level_info_grib() - failed to parse level
string
> 'surface'
> ERROR :
>
> Thanks in advance for any help you can provide.
>
> John Eylander
>
>
> On 6/25/20, 5:24 PM, "John Halley Gotway via RT"
<met_help at ucar.edu>
> wrote:
>
> Hello John,
>
> I see you have a few questions about the MET software.
>
> (1) Yes, you really need to run the madis2nc tool on the MADIS
files. I
> realize that this seems silly since the MADIS data is already in
NetCDF
> format. The madis2nc tool is essentially a reformatter. It can
also
> filter
> out obs based on QC flags and can compute time summaries of the
point
> observations. But the main purpose is to reformat the point
> observations
> into a common NetCDF format that the other MET tools (point-stat
and
> ensemble-stat) expect.
>
> We have done a nice job in MET supporting different gridded data
files
> in
> their native format, but we decided not to do that with point
> observations.
> Instead of reading them directly, we reformat them into a common
format
> using pb2nc, madis2nc, ascii2nc, lidar2nc, and so on. However we
may
> redesign that in the future and support the point obs in their
native
> formats. But for now madis2nc is necessary.
>
> If your madis2nc output doesn't contain the expected obs, please
send
> me
> the command you're running, along with a sample data file.
>
> (2) You should check your WRF output files. You state that you
have
> hourly
> WRF output, but WRF often outputs a runtime accumulation. So
that
> output
> file for a 6-hour forecast contains 0 to 6 hours of
accumulation. At
> 7-hours you'd get 0 to 7 hours of accumulation. However, WRF
really
> can be
> configured to dump an hourly precip bucket, but that's not the
default.
>
> If you actually have a runtime accumulation, you'd run
pcp_combine to
> compute the difference. For example, 7-hour accum - 6-hour accum
= 1-hr
> accum between 6 and 7 hours.
>
> (3) WRF and GFS are gridded forecasts. MADIS are point
observations.
> You
> run the Point-Stat tool to verify against point observations,
not the
> Grid-Stat tool.
>
> Two other things to mention...
>
> We typically recommend that users post-process their WRF output
using
> the
> Unified Post-Processor (
> Blockedhttps://
> dtcenter.org/community-code/unified-post-processor-upp)Blocked. That
reads
> WRF NetCDF output, de-staggers the staggered variables (winds),
derives
> requested variables, interpolates from WRF's native vertical
> coordinate to
> pressure, and writes GRIB1 or GRIB2 output files. If you really
are
> only
> interested in precip at the surface... and not winds or any
variables
> on
> pressure levels... then technically you do not have to run UPP.
But I
> would
> still recommend it.
>
> METplus is a suite of python wrappers intended to make it easier
to
> automate calls to the MET tools. The intention was to make
getting up
> and
> running with MET easier, but it's still relatively new. I don't
know
> whether or not we have an existing use case which verifies WRF
output
> against MADIS obs, but I'll let George chime in about that.
>
> Thanks,
> John Halley Gotway
>
> On Thu, Jun 25, 2020 at 12:19 PM Eylander, John B ERDC-RDE-
CRREL-NH
> CIV via
> RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: Blockedhttps://
> rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> >
> > Thanks!
> >
> > On 6/25/20, 2:09 PM, "George McCabe via RT"
<met_help at ucar.edu>
> wrote:
> >
> > Hi John,
> >
> > I see you have questions about processing MADIS data using
the
> MET
> > tools. I
> > have assigned this ticket to John Halley Gotway, a
software
> engineer
> > familiar with these tools. Please allow a few business
days for a
> > response.
> >
> > Thanks,
> > George
> >
> > On Thu, Jun 25, 2020 at 5:46 AM Eylander, John B
> ERDC-RDE-CRREL-NH CIV
> > via
> > RT <met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Jun 25 05:46:04 2020: Request 95731 was acted upon.
> > > Transaction: Ticket created by
John.B.Eylander at erdc.dren.mil
> > > Queue: met_help
> > > Subject: Using MET to evaluate WRF with MADIS
NETCDF data
> > > Owner: Nobody
> > > Requestors: John.B.Eylander at erdc.dren.mil
> > > Status: new
> > > Ticket <URL: Blockedhttps://
> > rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> > >
> > >
> > > Good morning MET help people!
> > >
> > > I’m trying to use MET to evaluate precipitation skill in
a
> bunch of
> > model
> > > data we have from WRF and GFS runs. Initially, I’m
focused on
> a high
> > > resolution WRF domain, at 1-hourly intervals in GriB2
format to
> > compare
> > > with MADIS NETCDF output I have (also in 1-hourly
files).
> I’ve read
> > > through the MET users manual to try to figure out what
steps I
> have
> > to
> > > perform and how to set up the configuration files, but I
am
> either
> > not
> > > understanding the manual or missing something.
> > >
> > > 1st question: do I really need to run the MADIS2NC tool
on the
> > NETCDF
> > > files? I’ve done so, but the output of MADIS2NC removes
most
> of the
> > data
> > > only leaving information about the station in the
resulting
> .nc file.
> > >
> > > Since I’m comparing 1-hourly WRF output to 1-hourly
> MADIS/METAR (or
> > > mesonet) data, I don’t need to run PCP_COMBINE (I will
when
> looking
> > at
> > > other 3-hourly WRF or GFS output).
> > >
> > > Leading me to my second question: I cannot figure out a
proper
> > Grid_stat
> > > configuration file to compare the WRF data against the
MADIS
> data.
> > I have
> > > searched your website, googled the topic, and asked a
number of
> > others
> > > (with no luck). Is there a sample configuration file
you can
> > provide for
> > > comparing WRF 1-hourly files to MADIS NetCDF data?
> > >
> > > 3rd question: Am I doing this all wrong?
> > >
> > > John Eylander
> > >
> > >
> > > --
> > >
> > > John Eylander
> > > Hydrologic Systems Branch
> > > Engineer Research and Development Center
> > > US Army Corp of Engineers
> > >
> > > John.B.Eylander at usace.army.mil<mailto:
> John.B.Eylander at usace.army.mil>
> > (U)
> > > John.B.Eylander at erdc.dren.mil<mailto:
> John.B.Eylander at erdc.dren.mil>
> > (U –
> > > RDE)
> > >
> > > John.B.Eylander.civ at mail.smil.mil<mailto:
> > John.B.Eylander.civ at mail.smil.mil>
> > > (S)
> > > John.B.Eylander.civ at army.ic.gov<mailto:
> > John.B.Eylander.civ at army.ic.gov>
> > > (JWICS)
> > >
> > > PH: 603-646-4188
> > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not
feel
> obliged
> > to
> > reply to this email outside of your normal working hours.
> >
> >
> >
> >
> >
> >
>
>
>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #95731] Using MET to evaluate WRF with MADIS NETCDF data
From: Eylander, John B ERDC-RDE-CRREL-NH CIV
Time: Tue Jun 30 10:00:15 2020
The README either didn't get attached or it didn't make it here. Can
you resend?
On 6/30/20, 11:55 AM, "John Halley Gotway via RT" <met_help at ucar.edu>
wrote:
John,
When working with GRIB1 or GRIB2 files, you request data from them
by
specifying a "name" and "level". Just set to the "name" to the
GRIB
abbreviation for the data... e.g. name = "APCP" for accumulated
precip. For
GRIB data, the "level" string starts with A, P, Z, L, or R. A is
for
accumulation interval, P is for pressure level, Z is for height
level, L is
for a generic level type, and R is for a specific record number.
So name="APCP"; level="A84"; would request a field of 84-hour
accumulated
precip.
Just run wgrib2 on your data file to see what precip accumulation
is
present there:
wgrib2 nam.t18z.awphys.f84.tm00.20200412.grib2 | grep APCP
And take a look at line 609 of this README file for some more info
on GRIB
levels.
Thanks,
John
On Tue, Jun 30, 2020 at 9:07 AM Eylander, John B ERDC-RDE-CRREL-NH
CIV via
RT <met_help at ucar.edu> wrote:
>
> <URL:
Blockedhttps://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked
>
>
> I am now trying to execute PCP_COMBINE on the WRF GriB2 (already
> post-processed) data to separate the ACPC fields into 1-hour
accumulation
> field. Based on my reading of the MET manual and after
exploring several
> online scenarios I found, I thought the execution of PCP_COMBINE
should
> look like so:
>
> ${MET_BIN}/pcp_combine -subtract
nam.t18z.awphys.f84.tm00.20200412.grib2
> 84 nam.t18z.awphys.f81.tm00.20200412.grib2 81 $HOME/test.out
-v 3
>
> However, I get the following error/diagnostic output when trying
to run
> pcp_combine:
>
> DEBUG 2: Since the "-field" command line option was not used,
parsing the
> command line arguments a list of files, each followed by a
configuration
> string.
> DEBUG 1: Reading input file:
nam.t18z.awphys.f84.tm00.20200412.grib2
> DEBUG 1: Reading data (name="APCP"; level="A84";) from input
file:
> nam.t18z.awphys.f84.tm00.20200412.grib2
> WARNING:
> WARNING: MetGrib2DataFile::data_plane() -> No matching record
found for
> 'APCP/A84'
> WARNING:
> ERROR :
> ERROR : get_field() -> can't get data plane from file
> "nam.t18z.awphys.f84.tm00.20200412.grib2"
> ERROR :
>
> What I can't understand is why the "level" field is set to
"A84". From
> what I understand of the manual, the APCP field should be
automatically
> able to find APCP in the GriB2 output. However, this is not
happening
> because the level of APCP should be "surface", not "A84".
>
> When I try to explicitly set the configuration to set the level
to
> "surface", I still get an error:
>
> $MET_BIN/pcp_combine -subtract
nam.t18z.awphys.f81.tm00.20200412.grib2
> 'name="APCP"; level="surface";'
nam.t18z.awphys.f84.tm00.20200412.grib2
> 'name="APCP"; level="surface";' $HOME/test.out -v 3
>
> DEBUG 2: Since the "-field" command line option was not used,
parsing the
> command line arguments a list of files, each followed by a
configuration
> string.
> DEBUG 1: Reading input file:
nam.t18z.awphys.f81.tm00.20200412.grib2
> DEBUG 1: Reading data (name="APCP"; level="surface";) from input
file:
> nam.t18z.awphys.f81.tm00.20200412.grib2
> ERROR :
> ERROR : VarInfo::set_level_info_grib() - failed to parse level
string
> 'surface'
> ERROR :
>
> Thanks in advance for any help you can provide.
>
> John Eylander
>
>
> On 6/25/20, 5:24 PM, "John Halley Gotway via RT"
<met_help at ucar.edu>
> wrote:
>
> Hello John,
>
> I see you have a few questions about the MET software.
>
> (1) Yes, you really need to run the madis2nc tool on the
MADIS files. I
> realize that this seems silly since the MADIS data is
already in NetCDF
> format. The madis2nc tool is essentially a reformatter. It
can also
> filter
> out obs based on QC flags and can compute time summaries of
the point
> observations. But the main purpose is to reformat the point
> observations
> into a common NetCDF format that the other MET tools (point-
stat and
> ensemble-stat) expect.
>
> We have done a nice job in MET supporting different gridded
data files
> in
> their native format, but we decided not to do that with
point
> observations.
> Instead of reading them directly, we reformat them into a
common format
> using pb2nc, madis2nc, ascii2nc, lidar2nc, and so on.
However we may
> redesign that in the future and support the point obs in
their native
> formats. But for now madis2nc is necessary.
>
> If your madis2nc output doesn't contain the expected obs,
please send
> me
> the command you're running, along with a sample data file.
>
> (2) You should check your WRF output files. You state that
you have
> hourly
> WRF output, but WRF often outputs a runtime accumulation. So
that
> output
> file for a 6-hour forecast contains 0 to 6 hours of
accumulation. At
> 7-hours you'd get 0 to 7 hours of accumulation. However, WRF
really
> can be
> configured to dump an hourly precip bucket, but that's not
the default.
>
> If you actually have a runtime accumulation, you'd run
pcp_combine to
> compute the difference. For example, 7-hour accum - 6-hour
accum = 1-hr
> accum between 6 and 7 hours.
>
> (3) WRF and GFS are gridded forecasts. MADIS are point
observations.
> You
> run the Point-Stat tool to verify against point
observations, not the
> Grid-Stat tool.
>
> Two other things to mention...
>
> We typically recommend that users post-process their WRF
output using
> the
> Unified Post-Processor (
> Blockedhttps://
> dtcenter.org/community-code/unified-post-processor-upp)Blocked.
That reads
> WRF NetCDF output, de-staggers the staggered variables
(winds), derives
> requested variables, interpolates from WRF's native vertical
> coordinate to
> pressure, and writes GRIB1 or GRIB2 output files. If you
really are
> only
> interested in precip at the surface... and not winds or any
variables
> on
> pressure levels... then technically you do not have to run
UPP. But I
> would
> still recommend it.
>
> METplus is a suite of python wrappers intended to make it
easier to
> automate calls to the MET tools. The intention was to make
getting up
> and
> running with MET easier, but it's still relatively new. I
don't know
> whether or not we have an existing use case which verifies
WRF output
> against MADIS obs, but I'll let George chime in about that.
>
> Thanks,
> John Halley Gotway
>
> On Thu, Jun 25, 2020 at 12:19 PM Eylander, John B ERDC-RDE-
CRREL-NH
> CIV via
> RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: Blockedhttps://
> rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> >
> > Thanks!
> >
> > On 6/25/20, 2:09 PM, "George McCabe via RT"
<met_help at ucar.edu>
> wrote:
> >
> > Hi John,
> >
> > I see you have questions about processing MADIS data
using the
> MET
> > tools. I
> > have assigned this ticket to John Halley Gotway, a
software
> engineer
> > familiar with these tools. Please allow a few business
days for a
> > response.
> >
> > Thanks,
> > George
> >
> > On Thu, Jun 25, 2020 at 5:46 AM Eylander, John B
> ERDC-RDE-CRREL-NH CIV
> > via
> > RT <met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Jun 25 05:46:04 2020: Request 95731 was acted
upon.
> > > Transaction: Ticket created by
John.B.Eylander at erdc.dren.mil
> > > Queue: met_help
> > > Subject: Using MET to evaluate WRF with MADIS
NETCDF data
> > > Owner: Nobody
> > > Requestors: John.B.Eylander at erdc.dren.mil
> > > Status: new
> > > Ticket <URL: Blockedhttps://
> > rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> > >
> > >
> > > Good morning MET help people!
> > >
> > > I’m trying to use MET to evaluate precipitation
skill in a
> bunch of
> > model
> > > data we have from WRF and GFS runs. Initially, I’m
focused on
> a high
> > > resolution WRF domain, at 1-hourly intervals in
GriB2 format to
> > compare
> > > with MADIS NETCDF output I have (also in 1-hourly
files).
> I’ve read
> > > through the MET users manual to try to figure out
what steps I
> have
> > to
> > > perform and how to set up the configuration files,
but I am
> either
> > not
> > > understanding the manual or missing something.
> > >
> > > 1st question: do I really need to run the MADIS2NC
tool on the
> > NETCDF
> > > files? I’ve done so, but the output of MADIS2NC
removes most
> of the
> > data
> > > only leaving information about the station in the
resulting
> .nc file.
> > >
> > > Since I’m comparing 1-hourly WRF output to 1-hourly
> MADIS/METAR (or
> > > mesonet) data, I don’t need to run PCP_COMBINE (I
will when
> looking
> > at
> > > other 3-hourly WRF or GFS output).
> > >
> > > Leading me to my second question: I cannot figure
out a proper
> > Grid_stat
> > > configuration file to compare the WRF data against
the MADIS
> data.
> > I have
> > > searched your website, googled the topic, and asked
a number of
> > others
> > > (with no luck). Is there a sample configuration
file you can
> > provide for
> > > comparing WRF 1-hourly files to MADIS NetCDF data?
> > >
> > > 3rd question: Am I doing this all wrong?
> > >
> > > John Eylander
> > >
> > >
> > > --
> > >
> > > John Eylander
> > > Hydrologic Systems Branch
> > > Engineer Research and Development Center
> > > US Army Corp of Engineers
> > >
> > > John.B.Eylander at usace.army.mil<mailto:
> John.B.Eylander at usace.army.mil>
> > (U)
> > > John.B.Eylander at erdc.dren.mil<mailto:
> John.B.Eylander at erdc.dren.mil>
> > (U –
> > > RDE)
> > >
> > > John.B.Eylander.civ at mail.smil.mil<mailto:
> > John.B.Eylander.civ at mail.smil.mil>
> > > (S)
> > > John.B.Eylander.civ at army.ic.gov<mailto:
> > John.B.Eylander.civ at army.ic.gov>
> > > (JWICS)
> > >
> > > PH: 603-646-4188
> > >
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do
not feel
> obliged
> > to
> > reply to this email outside of your normal working
hours.
> >
> >
> >
> >
> >
> >
>
>
>
>
>
>
------------------------------------------------
Subject: Using MET to evaluate WRF with MADIS NETCDF data
From: John Halley Gotway
Time: Tue Jun 30 10:02:30 2020
Ah forgot to paste the link!
https://github.com/NCAR/MET/blob/master_v9.0/met/data/config/README
Doing too many things at once.
John
On Tue, Jun 30, 2020 at 10:00 AM Eylander, John B ERDC-RDE-CRREL-NH
CIV via
RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731 >
>
> The README either didn't get attached or it didn't make it here.
Can you
> resend?
>
> On 6/30/20, 11:55 AM, "John Halley Gotway via RT"
<met_help at ucar.edu>
> wrote:
>
> John,
>
> When working with GRIB1 or GRIB2 files, you request data from
them by
> specifying a "name" and "level". Just set to the "name" to the
GRIB
> abbreviation for the data... e.g. name = "APCP" for accumulated
> precip. For
> GRIB data, the "level" string starts with A, P, Z, L, or R. A is
for
> accumulation interval, P is for pressure level, Z is for height
level,
> L is
> for a generic level type, and R is for a specific record number.
>
> So name="APCP"; level="A84"; would request a field of 84-hour
> accumulated
> precip.
>
> Just run wgrib2 on your data file to see what precip
accumulation is
> present there:
>
> wgrib2 nam.t18z.awphys.f84.tm00.20200412.grib2 | grep APCP
>
> And take a look at line 609 of this README file for some more
info on
> GRIB
> levels.
>
> Thanks,
> John
>
> On Tue, Jun 30, 2020 at 9:07 AM Eylander, John B ERDC-RDE-CRREL-
NH CIV
> via
> RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: Blockedhttps://
> rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> >
> > I am now trying to execute PCP_COMBINE on the WRF GriB2
(already
> > post-processed) data to separate the ACPC fields into 1-hour
> accumulation
> > field. Based on my reading of the MET manual and after
exploring
> several
> > online scenarios I found, I thought the execution of
PCP_COMBINE
> should
> > look like so:
> >
> > ${MET_BIN}/pcp_combine -subtract
> nam.t18z.awphys.f84.tm00.20200412.grib2
> > 84 nam.t18z.awphys.f81.tm00.20200412.grib2 81
$HOME/test.out -v 3
> >
> > However, I get the following error/diagnostic output when
trying to
> run
> > pcp_combine:
> >
> > DEBUG 2: Since the "-field" command line option was not used,
> parsing the
> > command line arguments a list of files, each followed by a
> configuration
> > string.
> > DEBUG 1: Reading input file:
nam.t18z.awphys.f84.tm00.20200412.grib2
> > DEBUG 1: Reading data (name="APCP"; level="A84";) from input
file:
> > nam.t18z.awphys.f84.tm00.20200412.grib2
> > WARNING:
> > WARNING: MetGrib2DataFile::data_plane() -> No matching record
found
> for
> > 'APCP/A84'
> > WARNING:
> > ERROR :
> > ERROR : get_field() -> can't get data plane from file
> > "nam.t18z.awphys.f84.tm00.20200412.grib2"
> > ERROR :
> >
> > What I can't understand is why the "level" field is set to
"A84".
> From
> > what I understand of the manual, the APCP field should be
> automatically
> > able to find APCP in the GriB2 output. However, this is not
> happening
> > because the level of APCP should be "surface", not "A84".
> >
> > When I try to explicitly set the configuration to set the
level to
> > "surface", I still get an error:
> >
> > $MET_BIN/pcp_combine -subtract
> nam.t18z.awphys.f81.tm00.20200412.grib2
> > 'name="APCP"; level="surface";'
> nam.t18z.awphys.f84.tm00.20200412.grib2
> > 'name="APCP"; level="surface";' $HOME/test.out -v 3
> >
> > DEBUG 2: Since the "-field" command line option was not used,
> parsing the
> > command line arguments a list of files, each followed by a
> configuration
> > string.
> > DEBUG 1: Reading input file:
nam.t18z.awphys.f81.tm00.20200412.grib2
> > DEBUG 1: Reading data (name="APCP"; level="surface";) from
input
> file:
> > nam.t18z.awphys.f81.tm00.20200412.grib2
> > ERROR :
> > ERROR : VarInfo::set_level_info_grib() - failed to parse
level
> string
> > 'surface'
> > ERROR :
> >
> > Thanks in advance for any help you can provide.
> >
> > John Eylander
> >
> >
> > On 6/25/20, 5:24 PM, "John Halley Gotway via RT"
<met_help at ucar.edu>
> > wrote:
> >
> > Hello John,
> >
> > I see you have a few questions about the MET software.
> >
> > (1) Yes, you really need to run the madis2nc tool on the
MADIS
> files. I
> > realize that this seems silly since the MADIS data is
already in
> NetCDF
> > format. The madis2nc tool is essentially a reformatter. It
can
> also
> > filter
> > out obs based on QC flags and can compute time summaries
of the
> point
> > observations. But the main purpose is to reformat the
point
> > observations
> > into a common NetCDF format that the other MET tools
(point-stat
> and
> > ensemble-stat) expect.
> >
> > We have done a nice job in MET supporting different
gridded data
> files
> > in
> > their native format, but we decided not to do that with
point
> > observations.
> > Instead of reading them directly, we reformat them into a
common
> format
> > using pb2nc, madis2nc, ascii2nc, lidar2nc, and so on.
However we
> may
> > redesign that in the future and support the point obs in
their
> native
> > formats. But for now madis2nc is necessary.
> >
> > If your madis2nc output doesn't contain the expected obs,
please
> send
> > me
> > the command you're running, along with a sample data file.
> >
> > (2) You should check your WRF output files. You state that
you
> have
> > hourly
> > WRF output, but WRF often outputs a runtime accumulation.
So that
> > output
> > file for a 6-hour forecast contains 0 to 6 hours of
> accumulation. At
> > 7-hours you'd get 0 to 7 hours of accumulation. However,
WRF
> really
> > can be
> > configured to dump an hourly precip bucket, but that's not
the
> default.
> >
> > If you actually have a runtime accumulation, you'd run
> pcp_combine to
> > compute the difference. For example, 7-hour accum - 6-hour
accum
> = 1-hr
> > accum between 6 and 7 hours.
> >
> > (3) WRF and GFS are gridded forecasts. MADIS are point
> observations.
> > You
> > run the Point-Stat tool to verify against point
observations,
> not the
> > Grid-Stat tool.
> >
> > Two other things to mention...
> >
> > We typically recommend that users post-process their WRF
output
> using
> > the
> > Unified Post-Processor (
> > Blockedhttps://
> > dtcenter.org/community-code/unified-post-processor-
upp)Blocked.
> That reads
> > WRF NetCDF output, de-staggers the staggered variables
(winds),
> derives
> > requested variables, interpolates from WRF's native
vertical
> > coordinate to
> > pressure, and writes GRIB1 or GRIB2 output files. If you
really
> are
> > only
> > interested in precip at the surface... and not winds or
any
> variables
> > on
> > pressure levels... then technically you do not have to run
UPP.
> But I
> > would
> > still recommend it.
> >
> > METplus is a suite of python wrappers intended to make it
easier
> to
> > automate calls to the MET tools. The intention was to make
> getting up
> > and
> > running with MET easier, but it's still relatively new. I
don't
> know
> > whether or not we have an existing use case which verifies
WRF
> output
> > against MADIS obs, but I'll let George chime in about
that.
> >
> > Thanks,
> > John Halley Gotway
> >
> > On Thu, Jun 25, 2020 at 12:19 PM Eylander, John B
> ERDC-RDE-CRREL-NH
> > CIV via
> > RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: Blockedhttps://
> > rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> > >
> > > Thanks!
> > >
> > > On 6/25/20, 2:09 PM, "George McCabe via RT"
<met_help at ucar.edu
> >
> > wrote:
> > >
> > > Hi John,
> > >
> > > I see you have questions about processing MADIS data
using
> the
> > MET
> > > tools. I
> > > have assigned this ticket to John Halley Gotway, a
software
> > engineer
> > > familiar with these tools. Please allow a few
business
> days for a
> > > response.
> > >
> > > Thanks,
> > > George
> > >
> > > On Thu, Jun 25, 2020 at 5:46 AM Eylander, John B
> > ERDC-RDE-CRREL-NH CIV
> > > via
> > > RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Thu Jun 25 05:46:04 2020: Request 95731 was acted
upon.
> > > > Transaction: Ticket created by
> John.B.Eylander at erdc.dren.mil
> > > > Queue: met_help
> > > > Subject: Using MET to evaluate WRF with MADIS
> NETCDF data
> > > > Owner: Nobody
> > > > Requestors: John.B.Eylander at erdc.dren.mil
> > > > Status: new
> > > > Ticket <URL: Blockedhttps://
> > > rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> > > >
> > > >
> > > > Good morning MET help people!
> > > >
> > > > I’m trying to use MET to evaluate precipitation
skill in
> a
> > bunch of
> > > model
> > > > data we have from WRF and GFS runs. Initially,
I’m
> focused on
> > a high
> > > > resolution WRF domain, at 1-hourly intervals in
GriB2
> format to
> > > compare
> > > > with MADIS NETCDF output I have (also in 1-hourly
files).
> > I’ve read
> > > > through the MET users manual to try to figure out
what
> steps I
> > have
> > > to
> > > > perform and how to set up the configuration files,
but I
> am
> > either
> > > not
> > > > understanding the manual or missing something.
> > > >
> > > > 1st question: do I really need to run the
MADIS2NC tool
> on the
> > > NETCDF
> > > > files? I’ve done so, but the output of MADIS2NC
removes
> most
> > of the
> > > data
> > > > only leaving information about the station in the
> resulting
> > .nc file.
> > > >
> > > > Since I’m comparing 1-hourly WRF output to 1-
hourly
> > MADIS/METAR (or
> > > > mesonet) data, I don’t need to run PCP_COMBINE (I
will
> when
> > looking
> > > at
> > > > other 3-hourly WRF or GFS output).
> > > >
> > > > Leading me to my second question: I cannot figure
out a
> proper
> > > Grid_stat
> > > > configuration file to compare the WRF data against
the
> MADIS
> > data.
> > > I have
> > > > searched your website, googled the topic, and
asked a
> number of
> > > others
> > > > (with no luck). Is there a sample configuration
file
> you can
> > > provide for
> > > > comparing WRF 1-hourly files to MADIS NetCDF data?
> > > >
> > > > 3rd question: Am I doing this all wrong?
> > > >
> > > > John Eylander
> > > >
> > > >
> > > > --
> > > >
> > > > John Eylander
> > > > Hydrologic Systems Branch
> > > > Engineer Research and Development Center
> > > > US Army Corp of Engineers
> > > >
> > > > John.B.Eylander at usace.army.mil<mailto:
> > John.B.Eylander at usace.army.mil>
> > > (U)
> > > > John.B.Eylander at erdc.dren.mil<mailto:
> > John.B.Eylander at erdc.dren.mil>
> > > (U –
> > > > RDE)
> > > >
> > > > John.B.Eylander.civ at mail.smil.mil<mailto:
> > > John.B.Eylander.civ at mail.smil.mil>
> > > > (S)
> > > > John.B.Eylander.civ at army.ic.gov<mailto:
> > > John.B.Eylander.civ at army.ic.gov>
> > > > (JWICS)
> > > >
> > > > PH: 603-646-4188
> > > >
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please
do not
> feel
> > obliged
> > > to
> > > reply to this email outside of your normal working
hours.
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
> >
>
>
>
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #95731] Using MET to evaluate WRF with MADIS NETCDF data
From: Eylander, John B ERDC-RDE-CRREL-NH CIV
Time: Mon Jul 06 20:00:06 2020
I'm still struggling to work my way through the comparison of WRF with
MADIS data. Here's where I'm at:
I've confirmed that the MADIS2NC results are good. The tool and my
use of it produce output that matches what the MET documentation
states I should be getting.
I've run PCP_COMBINE on the WRF output to generate 1-hourly APCP
files. The resulting NetCDF header for the PCP_COMBINE output files
is below (result of ncdump -h). I'm now trying to execute the
Point_stat routine, but I get the following error (this is just one
version of the error, I've tried to configure the point-stat tool in
multiple ways but can't see to get the tool to read in the APCP
record. I include my configuration file at the bottom of this.
Again...trying to compare MADIS point obs against WRF data is the
goal.
**********************************************************************************************
DEBUG 1: Default Config File:
/p/app/unsupported/hydrorf/met/9.0.1/share/met/config/PointStatConfig_default
DEBUG 1: User Config File:
/p/home/eylandej/configs/PointStatConfig_MADIS
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=584385375
DEBUG 1: Forecast File:
/p/work/eylandej/MET_DATA/20200412/nam.t18z.awphys.f02.tm00.20200412.MET_PCP_SUB.nc
DEBUG 1: Observation File:
/gpfs/cwfs/eylandej/MADIS/MADIS2NC_Files/20200412/20200412_2000.nc
DEBUG 2:
DEBUG 2:
--------------------------------------------------------------------------------
DEBUG 2:
DEBUG 2: Reading data for APCPA1.
WARNING:
WARNING: process_fcst_climo_files() -> no fields matching APCPA1 found
in file:
/p/work/eylandej/MET_DATA/20200412/nam.t18z.awphys.f02.tm00.20200412.MET_PCP_SUB.nc
WARNING:
ERROR :
ERROR : process_fcst_climo_files() -> no requested forecast data
found! Exiting...
ERROR :
**********************************************************************************************
netcdf nam.t18z.awphys.f06.tm00.20200412.MET_PCP_SUB {
dimensions:
lat = 402 ;
lon = 613 ;
variables:
float lat(lat, lon) ;
lat:long_name = "latitude" ;
lat:units = "degrees_north" ;
lat:standard_name = "latitude" ;
float lon(lat, lon) ;
lon:long_name = "longitude" ;
lon:units = "degrees_east" ;
lon:standard_name = "longitude" ;
float APCP_01(lat, lon) ;
APCP_01:name = "APCP_01" ;
APCP_01:long_name = "Total Precipitation" ;
APCP_01:level = "A1" ;
APCP_01:units = "kg/m^2" ;
APCP_01:_FillValue = -9999.f ;
APCP_01:init_time = "20200412_180000" ;
APCP_01:init_time_ut = "1586714400" ;
APCP_01:valid_time = "20200413_000000" ;
APCP_01:valid_time_ut = "1586736000" ;
APCP_01:accum_time = "010000" ;
APCP_01:accum_time_sec = 3600 ;
// global attributes:
:FileOrigins = "File
/p/work/eylandej/MET_DATA/20200412/nam.t18z.awphys.f06.tm00.20200412.MET_PCP_SUB.nc
generated 20200706_183200 UTC on host onyx09 by the MET pcp_combine
tool" ;
:MET_version = "V9.0.1" ;
:MET_tool = "pcp_combine" ;
:RunCommand = "Subtraction:
/p/work/smatus/HydroRF/Archives/NAM/20200412/18/nam.t18z.awphys.f06.tm00.20200412.grib2
minus
Configuration file:
////////////////////////////////////////////////////////////////////////////////
//
// May be set separately in each "field" entry
//
censor_thresh = [];
censor_val = [];
cat_thresh = [ NA ];
cnt_thresh = [ NA ];
cnt_logic = UNION;
wind_thresh = [ NA ];
wind_logic = UNION;
eclv_points = 0.05;
rank_corr_flag = FALSE;
//
// Forecast and observation fields to be verified
//
fcst = {
field = [
{
name = "APCP";
level = ["A1"];
cat_thresh = [ >=0.1, >=0.2, >=0.3, >=0.4, >=0.5, >=1.0, >=1.5,
>=2.0, >=3.0 ];
}
];
}
obs = fcst;
////////////////////////////////////////////////////////////////////////////////
//
// Point observation filtering options
// May be set separately in each "obs.field" entry
//
message_type = [ "ADPSFC" ];
sid_inc = [];
sid_exc = [];
obs_quality = [];
duplicate_flag = NONE;
obs_summary = NONE;
obs_perc_value = 50;
//
// Mapping of message type group name to comma-separated list of
values.
//
message_type_group_map = [
{ key = "SURFACE"; val = "ADPSFC,SFCSHP,MSONET"; },
{ key = "ANYAIR"; val = "AIRCAR,AIRCFT"; },
{ key = "ANYSFC"; val = "ADPSFC,SFCSHP,ADPUPA,PROFLR,MSONET"; },
{ key = "ONLYSF"; val = "ADPSFC,SFCSHP"; },
{ key = "LANDSF"; val = "ADPSFC,MSONET"; },
{ key = "WATERSF"; val = "SFCSHP"; }
];
////////////////////////////////////////////////////////////////////////////////
John Eylander
/p/work/smatus/HydroRF/Archives/NAM/20200412/18/nam.t18z.awphys.f05.tm00.20200412.grib2"
;
On 6/30/20, 12:02 PM, "John Halley Gotway via RT" <met_help at ucar.edu>
wrote:
Ah forgot to paste the link!
Blockedhttps://github.com/NCAR/MET/blob/master_v9.0/met/data/config/READMEBlocked
Doing too many things at once.
John
On Tue, Jun 30, 2020 at 10:00 AM Eylander, John B ERDC-RDE-CRREL-
NH CIV via
RT <met_help at ucar.edu> wrote:
>
> <URL:
Blockedhttps://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked
>
>
> The README either didn't get attached or it didn't make it here.
Can you
> resend?
>
> On 6/30/20, 11:55 AM, "John Halley Gotway via RT"
<met_help at ucar.edu>
> wrote:
>
> John,
>
> When working with GRIB1 or GRIB2 files, you request data
from them by
> specifying a "name" and "level". Just set to the "name" to
the GRIB
> abbreviation for the data... e.g. name = "APCP" for
accumulated
> precip. For
> GRIB data, the "level" string starts with A, P, Z, L, or R.
A is for
> accumulation interval, P is for pressure level, Z is for
height level,
> L is
> for a generic level type, and R is for a specific record
number.
>
> So name="APCP"; level="A84"; would request a field of 84-
hour
> accumulated
> precip.
>
> Just run wgrib2 on your data file to see what precip
accumulation is
> present there:
>
> wgrib2 nam.t18z.awphys.f84.tm00.20200412.grib2 | grep APCP
>
> And take a look at line 609 of this README file for some
more info on
> GRIB
> levels.
>
> Thanks,
> John
>
> On Tue, Jun 30, 2020 at 9:07 AM Eylander, John B ERDC-RDE-
CRREL-NH CIV
> via
> RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: Blockedhttps://
> rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> >
> > I am now trying to execute PCP_COMBINE on the WRF GriB2
(already
> > post-processed) data to separate the ACPC fields into 1-
hour
> accumulation
> > field. Based on my reading of the MET manual and after
exploring
> several
> > online scenarios I found, I thought the execution of
PCP_COMBINE
> should
> > look like so:
> >
> > ${MET_BIN}/pcp_combine -subtract
> nam.t18z.awphys.f84.tm00.20200412.grib2
> > 84 nam.t18z.awphys.f81.tm00.20200412.grib2 81
$HOME/test.out -v 3
> >
> > However, I get the following error/diagnostic output when
trying to
> run
> > pcp_combine:
> >
> > DEBUG 2: Since the "-field" command line option was not
used,
> parsing the
> > command line arguments a list of files, each followed by a
> configuration
> > string.
> > DEBUG 1: Reading input file:
nam.t18z.awphys.f84.tm00.20200412.grib2
> > DEBUG 1: Reading data (name="APCP"; level="A84";) from
input file:
> > nam.t18z.awphys.f84.tm00.20200412.grib2
> > WARNING:
> > WARNING: MetGrib2DataFile::data_plane() -> No matching
record found
> for
> > 'APCP/A84'
> > WARNING:
> > ERROR :
> > ERROR : get_field() -> can't get data plane from file
> > "nam.t18z.awphys.f84.tm00.20200412.grib2"
> > ERROR :
> >
> > What I can't understand is why the "level" field is set to
"A84".
> From
> > what I understand of the manual, the APCP field should be
> automatically
> > able to find APCP in the GriB2 output. However, this is
not
> happening
> > because the level of APCP should be "surface", not "A84".
> >
> > When I try to explicitly set the configuration to set the
level to
> > "surface", I still get an error:
> >
> > $MET_BIN/pcp_combine -subtract
> nam.t18z.awphys.f81.tm00.20200412.grib2
> > 'name="APCP"; level="surface";'
> nam.t18z.awphys.f84.tm00.20200412.grib2
> > 'name="APCP"; level="surface";' $HOME/test.out -v 3
> >
> > DEBUG 2: Since the "-field" command line option was not
used,
> parsing the
> > command line arguments a list of files, each followed by a
> configuration
> > string.
> > DEBUG 1: Reading input file:
nam.t18z.awphys.f81.tm00.20200412.grib2
> > DEBUG 1: Reading data (name="APCP"; level="surface";) from
input
> file:
> > nam.t18z.awphys.f81.tm00.20200412.grib2
> > ERROR :
> > ERROR : VarInfo::set_level_info_grib() - failed to parse
level
> string
> > 'surface'
> > ERROR :
> >
> > Thanks in advance for any help you can provide.
> >
> > John Eylander
> >
> >
> > On 6/25/20, 5:24 PM, "John Halley Gotway via RT"
<met_help at ucar.edu>
> > wrote:
> >
> > Hello John,
> >
> > I see you have a few questions about the MET software.
> >
> > (1) Yes, you really need to run the madis2nc tool on
the MADIS
> files. I
> > realize that this seems silly since the MADIS data is
already in
> NetCDF
> > format. The madis2nc tool is essentially a
reformatter. It can
> also
> > filter
> > out obs based on QC flags and can compute time
summaries of the
> point
> > observations. But the main purpose is to reformat the
point
> > observations
> > into a common NetCDF format that the other MET tools
(point-stat
> and
> > ensemble-stat) expect.
> >
> > We have done a nice job in MET supporting different
gridded data
> files
> > in
> > their native format, but we decided not to do that
with point
> > observations.
> > Instead of reading them directly, we reformat them
into a common
> format
> > using pb2nc, madis2nc, ascii2nc, lidar2nc, and so on.
However we
> may
> > redesign that in the future and support the point obs
in their
> native
> > formats. But for now madis2nc is necessary.
> >
> > If your madis2nc output doesn't contain the expected
obs, please
> send
> > me
> > the command you're running, along with a sample data
file.
> >
> > (2) You should check your WRF output files. You state
that you
> have
> > hourly
> > WRF output, but WRF often outputs a runtime
accumulation. So that
> > output
> > file for a 6-hour forecast contains 0 to 6 hours of
> accumulation. At
> > 7-hours you'd get 0 to 7 hours of accumulation.
However, WRF
> really
> > can be
> > configured to dump an hourly precip bucket, but that's
not the
> default.
> >
> > If you actually have a runtime accumulation, you'd run
> pcp_combine to
> > compute the difference. For example, 7-hour accum - 6-
hour accum
> = 1-hr
> > accum between 6 and 7 hours.
> >
> > (3) WRF and GFS are gridded forecasts. MADIS are point
> observations.
> > You
> > run the Point-Stat tool to verify against point
observations,
> not the
> > Grid-Stat tool.
> >
> > Two other things to mention...
> >
> > We typically recommend that users post-process their
WRF output
> using
> > the
> > Unified Post-Processor (
> > Blockedhttps://
> > dtcenter.org/community-code/unified-post-processor-
upp)Blocked.
> That reads
> > WRF NetCDF output, de-staggers the staggered variables
(winds),
> derives
> > requested variables, interpolates from WRF's native
vertical
> > coordinate to
> > pressure, and writes GRIB1 or GRIB2 output files. If
you really
> are
> > only
> > interested in precip at the surface... and not winds
or any
> variables
> > on
> > pressure levels... then technically you do not have to
run UPP.
> But I
> > would
> > still recommend it.
> >
> > METplus is a suite of python wrappers intended to make
it easier
> to
> > automate calls to the MET tools. The intention was to
make
> getting up
> > and
> > running with MET easier, but it's still relatively
new. I don't
> know
> > whether or not we have an existing use case which
verifies WRF
> output
> > against MADIS obs, but I'll let George chime in about
that.
> >
> > Thanks,
> > John Halley Gotway
> >
> > On Thu, Jun 25, 2020 at 12:19 PM Eylander, John B
> ERDC-RDE-CRREL-NH
> > CIV via
> > RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: Blockedhttps://
> > rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> > >
> > > Thanks!
> > >
> > > On 6/25/20, 2:09 PM, "George McCabe via RT"
<met_help at ucar.edu
> >
> > wrote:
> > >
> > > Hi John,
> > >
> > > I see you have questions about processing MADIS
data using
> the
> > MET
> > > tools. I
> > > have assigned this ticket to John Halley Gotway,
a software
> > engineer
> > > familiar with these tools. Please allow a few
business
> days for a
> > > response.
> > >
> > > Thanks,
> > > George
> > >
> > > On Thu, Jun 25, 2020 at 5:46 AM Eylander, John B
> > ERDC-RDE-CRREL-NH CIV
> > > via
> > > RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Thu Jun 25 05:46:04 2020: Request 95731 was
acted upon.
> > > > Transaction: Ticket created by
> John.B.Eylander at erdc.dren.mil
> > > > Queue: met_help
> > > > Subject: Using MET to evaluate WRF with
MADIS
> NETCDF data
> > > > Owner: Nobody
> > > > Requestors: John.B.Eylander at erdc.dren.mil
> > > > Status: new
> > > > Ticket <URL: Blockedhttps://
> > >
rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> > > >
> > > >
> > > > Good morning MET help people!
> > > >
> > > > I’m trying to use MET to evaluate
precipitation skill in
> a
> > bunch of
> > > model
> > > > data we have from WRF and GFS runs.
Initially, I’m
> focused on
> > a high
> > > > resolution WRF domain, at 1-hourly intervals
in GriB2
> format to
> > > compare
> > > > with MADIS NETCDF output I have (also in 1-
hourly files).
> > I’ve read
> > > > through the MET users manual to try to figure
out what
> steps I
> > have
> > > to
> > > > perform and how to set up the configuration
files, but I
> am
> > either
> > > not
> > > > understanding the manual or missing something.
> > > >
> > > > 1st question: do I really need to run the
MADIS2NC tool
> on the
> > > NETCDF
> > > > files? I’ve done so, but the output of
MADIS2NC removes
> most
> > of the
> > > data
> > > > only leaving information about the station in
the
> resulting
> > .nc file.
> > > >
> > > > Since I’m comparing 1-hourly WRF output to 1-
hourly
> > MADIS/METAR (or
> > > > mesonet) data, I don’t need to run PCP_COMBINE
(I will
> when
> > looking
> > > at
> > > > other 3-hourly WRF or GFS output).
> > > >
> > > > Leading me to my second question: I cannot
figure out a
> proper
> > > Grid_stat
> > > > configuration file to compare the WRF data
against the
> MADIS
> > data.
> > > I have
> > > > searched your website, googled the topic, and
asked a
> number of
> > > others
> > > > (with no luck). Is there a sample
configuration file
> you can
> > > provide for
> > > > comparing WRF 1-hourly files to MADIS NetCDF
data?
> > > >
> > > > 3rd question: Am I doing this all wrong?
> > > >
> > > > John Eylander
> > > >
> > > >
> > > > --
> > > >
> > > > John Eylander
> > > > Hydrologic Systems Branch
> > > > Engineer Research and Development Center
> > > > US Army Corp of Engineers
> > > >
> > > > John.B.Eylander at usace.army.mil<mailto:
> > John.B.Eylander at usace.army.mil>
> > > (U)
> > > > John.B.Eylander at erdc.dren.mil<mailto:
> > John.B.Eylander at erdc.dren.mil>
> > > (U –
> > > > RDE)
> > > >
> > > > John.B.Eylander.civ at mail.smil.mil<mailto:
> > > John.B.Eylander.civ at mail.smil.mil>
> > > > (S)
> > > > John.B.Eylander.civ at army.ic.gov<mailto:
> > > John.B.Eylander.civ at army.ic.gov>
> > > > (JWICS)
> > > >
> > > > PH: 603-646-4188
> > > >
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day.
Please do not
> feel
> > obliged
> > > to
> > > reply to this email outside of your normal
working hours.
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
> >
>
>
>
>
>
>
------------------------------------------------
Subject: Using MET to evaluate WRF with MADIS NETCDF data
From: Julie Prestopnik
Time: Thu Jul 09 16:34:04 2020
Hi John.
John is out of the office, hence the delayed response. My name is
Julie,
and I work with John. I thought I would jump in and try to help in
his
absence.
I see that you are having trouble with PointStat:
> WARNING: process_fcst_climo_files() -> no fields matching APCPA1
found in
> file: /p/work/eylandej/MET_DATA/20200412/
> nam.t18z.awphys.f02.tm00.20200412.MET_PCP_SUB.nc
MET is having a hard time recognizing the field. From the header dump
that
you sent (thank you!), I can see that the field name is "APCP_01" and
not
"APCP". Setting the "level" field is file-format specific. This is
described starting on page 59 of our most recent MET User's Guide
<https://dtcenter.org/sites/default/files/community-
code/met/docs/user-guide/MET_Users_Guide_v9.0.3.pdf>.
You have specified the field using the syntax for GRIB and GRIB2
files.
For NetCDF files, you'll want to specify the level as "(*,*)", which
specifies the two dimensions for the gridded field. Please try
changing
the "fcst" entries for "name" and "level" as I have below:
fcst = {
> field = [
> {
> name = "APCP_01";
> level = "(*,*)";
> cat_thresh = [ >=0.1, >=0.2, >=0.3, >=0.4, >=0.5, >=1.0,
>=1.5,
> >=2.0, >=3.0 ];
> }
>
> ];
>
> }
>
If your obs values are the same, go ahead and use this as you had it:
> obs = fcst;
>
Otherwise, you'll need to change that as well.
I hope this helps move you along farther. If it does not, please feel
free
to upload your data to our FTP server as describe here under "How To
Send
Us Data":
https://dtcenter.org/community-code/model-evaluation-tools-met/met-
help-desk
and also send the command line you are running, and I can try to get
it
working for you.
Julie
On Mon, Jul 6, 2020 at 8:00 PM Eylander, John B ERDC-RDE-CRREL-NH CIV
via
RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731 >
>
> I'm still struggling to work my way through the comparison of WRF
with
> MADIS data. Here's where I'm at:
>
> I've confirmed that the MADIS2NC results are good. The tool and my
use of
> it produce output that matches what the MET documentation states I
should
> be getting.
>
> I've run PCP_COMBINE on the WRF output to generate 1-hourly APCP
files.
> The resulting NetCDF header for the PCP_COMBINE output files is
below
> (result of ncdump -h). I'm now trying to execute the Point_stat
routine,
> but I get the following error (this is just one version of the
error, I've
> tried to configure the point-stat tool in multiple ways but can't
see to
> get the tool to read in the APCP record. I include my configuration
file
> at the bottom of this. Again...trying to compare MADIS point obs
against
> WRF data is the goal.
>
>
>
**********************************************************************************************
> DEBUG 1: Default Config File:
>
/p/app/unsupported/hydrorf/met/9.0.1/share/met/config/PointStatConfig_default
> DEBUG 1: User Config File:
/p/home/eylandej/configs/PointStatConfig_MADIS
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=584385375
> DEBUG 1: Forecast File: /p/work/eylandej/MET_DATA/20200412/
> nam.t18z.awphys.f02.tm00.20200412.MET_PCP_SUB.nc
> DEBUG 1: Observation File:
> /gpfs/cwfs/eylandej/MADIS/MADIS2NC_Files/20200412/20200412_2000.nc
> DEBUG 2:
> DEBUG 2:
>
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 2: Reading data for APCPA1.
> WARNING:
> WARNING: process_fcst_climo_files() -> no fields matching APCPA1
found in
> file: /p/work/eylandej/MET_DATA/20200412/
> nam.t18z.awphys.f02.tm00.20200412.MET_PCP_SUB.nc
> WARNING:
> ERROR :
> ERROR : process_fcst_climo_files() -> no requested forecast data
found!
> Exiting...
> ERROR :
>
>
**********************************************************************************************
>
> netcdf nam.t18z.awphys.f06.tm00.20200412.MET_PCP_SUB {
> dimensions:
> lat = 402 ;
> lon = 613 ;
> variables:
> float lat(lat, lon) ;
> lat:long_name = "latitude" ;
> lat:units = "degrees_north" ;
> lat:standard_name = "latitude" ;
> float lon(lat, lon) ;
> lon:long_name = "longitude" ;
> lon:units = "degrees_east" ;
> lon:standard_name = "longitude" ;
> float APCP_01(lat, lon) ;
> APCP_01:name = "APCP_01" ;
> APCP_01:long_name = "Total Precipitation" ;
> APCP_01:level = "A1" ;
> APCP_01:units = "kg/m^2" ;
> APCP_01:_FillValue = -9999.f ;
> APCP_01:init_time = "20200412_180000" ;
> APCP_01:init_time_ut = "1586714400" ;
> APCP_01:valid_time = "20200413_000000" ;
> APCP_01:valid_time_ut = "1586736000" ;
> APCP_01:accum_time = "010000" ;
> APCP_01:accum_time_sec = 3600 ;
>
> // global attributes:
> :FileOrigins = "File
/p/work/eylandej/MET_DATA/20200412/
> nam.t18z.awphys.f06.tm00.20200412.MET_PCP_SUB.nc generated
> 20200706_183200 UTC on host onyx09 by the MET pcp_combine tool" ;
> :MET_version = "V9.0.1" ;
> :MET_tool = "pcp_combine" ;
> :RunCommand = "Subtraction:
>
/p/work/smatus/HydroRF/Archives/NAM/20200412/18/nam.t18z.awphys.f06.tm00.20200412.grib2
> minus
>
>
>
> Configuration file:
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // May be set separately in each "field" entry
> //
> censor_thresh = [];
> censor_val = [];
> cat_thresh = [ NA ];
> cnt_thresh = [ NA ];
> cnt_logic = UNION;
> wind_thresh = [ NA ];
> wind_logic = UNION;
> eclv_points = 0.05;
> rank_corr_flag = FALSE;
>
> //
> // Forecast and observation fields to be verified
> //
> fcst = {
> field = [
> {
> name = "APCP";
> level = ["A1"];
> cat_thresh = [ >=0.1, >=0.2, >=0.3, >=0.4, >=0.5, >=1.0,
>=1.5,
> >=2.0, >=3.0 ];
> }
>
> ];
>
> }
> obs = fcst;
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Point observation filtering options
> // May be set separately in each "obs.field" entry
> //
> message_type = [ "ADPSFC" ];
> sid_inc = [];
> sid_exc = [];
> obs_quality = [];
> duplicate_flag = NONE;
> obs_summary = NONE;
> obs_perc_value = 50;
>
> //
> // Mapping of message type group name to comma-separated list of
values.
> //
> message_type_group_map = [
> { key = "SURFACE"; val = "ADPSFC,SFCSHP,MSONET"; },
> { key = "ANYAIR"; val = "AIRCAR,AIRCFT"; },
> { key = "ANYSFC"; val = "ADPSFC,SFCSHP,ADPUPA,PROFLR,MSONET"; },
> { key = "ONLYSF"; val = "ADPSFC,SFCSHP"; },
> { key = "LANDSF"; val = "ADPSFC,MSONET"; },
> { key = "WATERSF"; val = "SFCSHP"; }
> ];
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
>
> John Eylander
>
>
>
>
>
>
>
/p/work/smatus/HydroRF/Archives/NAM/20200412/18/nam.t18z.awphys.f05.tm00.20200412.grib2"
> ;
>
> On 6/30/20, 12:02 PM, "John Halley Gotway via RT"
<met_help at ucar.edu>
> wrote:
>
> Ah forgot to paste the link!
> Blockedhttps://
> github.com/NCAR/MET/blob/master_v9.0/met/data/config/READMEBlocked
>
> Doing too many things at once.
>
> John
>
> On Tue, Jun 30, 2020 at 10:00 AM Eylander, John B ERDC-RDE-
CRREL-NH
> CIV via
> RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: Blockedhttps://
> rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> >
> > The README either didn't get attached or it didn't make it
here.
> Can you
> > resend?
> >
> > On 6/30/20, 11:55 AM, "John Halley Gotway via RT"
<met_help at ucar.edu
> >
> > wrote:
> >
> > John,
> >
> > When working with GRIB1 or GRIB2 files, you request data
from
> them by
> > specifying a "name" and "level". Just set to the "name" to
the
> GRIB
> > abbreviation for the data... e.g. name = "APCP" for
accumulated
> > precip. For
> > GRIB data, the "level" string starts with A, P, Z, L, or
R. A is
> for
> > accumulation interval, P is for pressure level, Z is for
height
> level,
> > L is
> > for a generic level type, and R is for a specific record
number.
> >
> > So name="APCP"; level="A84"; would request a field of 84-
hour
> > accumulated
> > precip.
> >
> > Just run wgrib2 on your data file to see what precip
> accumulation is
> > present there:
> >
> > wgrib2 nam.t18z.awphys.f84.tm00.20200412.grib2 | grep APCP
> >
> > And take a look at line 609 of this README file for some
more
> info on
> > GRIB
> > levels.
> >
> > Thanks,
> > John
> >
> > On Tue, Jun 30, 2020 at 9:07 AM Eylander, John B
> ERDC-RDE-CRREL-NH CIV
> > via
> > RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: Blockedhttps://
> > rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> > >
> > > I am now trying to execute PCP_COMBINE on the WRF GriB2
> (already
> > > post-processed) data to separate the ACPC fields into 1-
hour
> > accumulation
> > > field. Based on my reading of the MET manual and after
> exploring
> > several
> > > online scenarios I found, I thought the execution of
> PCP_COMBINE
> > should
> > > look like so:
> > >
> > > ${MET_BIN}/pcp_combine -subtract
> > nam.t18z.awphys.f84.tm00.20200412.grib2
> > > 84 nam.t18z.awphys.f81.tm00.20200412.grib2 81
> $HOME/test.out -v 3
> > >
> > > However, I get the following error/diagnostic output
when
> trying to
> > run
> > > pcp_combine:
> > >
> > > DEBUG 2: Since the "-field" command line option was not
used,
> > parsing the
> > > command line arguments a list of files, each followed by
a
> > configuration
> > > string.
> > > DEBUG 1: Reading input file:
> nam.t18z.awphys.f84.tm00.20200412.grib2
> > > DEBUG 1: Reading data (name="APCP"; level="A84";) from
input
> file:
> > > nam.t18z.awphys.f84.tm00.20200412.grib2
> > > WARNING:
> > > WARNING: MetGrib2DataFile::data_plane() -> No matching
record
> found
> > for
> > > 'APCP/A84'
> > > WARNING:
> > > ERROR :
> > > ERROR : get_field() -> can't get data plane from file
> > > "nam.t18z.awphys.f84.tm00.20200412.grib2"
> > > ERROR :
> > >
> > > What I can't understand is why the "level" field is set
to
> "A84".
> > From
> > > what I understand of the manual, the APCP field should
be
> > automatically
> > > able to find APCP in the GriB2 output. However, this is
not
> > happening
> > > because the level of APCP should be "surface", not
"A84".
> > >
> > > When I try to explicitly set the configuration to set
the
> level to
> > > "surface", I still get an error:
> > >
> > > $MET_BIN/pcp_combine -subtract
> > nam.t18z.awphys.f81.tm00.20200412.grib2
> > > 'name="APCP"; level="surface";'
> > nam.t18z.awphys.f84.tm00.20200412.grib2
> > > 'name="APCP"; level="surface";' $HOME/test.out -v 3
> > >
> > > DEBUG 2: Since the "-field" command line option was not
used,
> > parsing the
> > > command line arguments a list of files, each followed by
a
> > configuration
> > > string.
> > > DEBUG 1: Reading input file:
> nam.t18z.awphys.f81.tm00.20200412.grib2
> > > DEBUG 1: Reading data (name="APCP"; level="surface";)
from
> input
> > file:
> > > nam.t18z.awphys.f81.tm00.20200412.grib2
> > > ERROR :
> > > ERROR : VarInfo::set_level_info_grib() - failed to
parse level
> > string
> > > 'surface'
> > > ERROR :
> > >
> > > Thanks in advance for any help you can provide.
> > >
> > > John Eylander
> > >
> > >
> > > On 6/25/20, 5:24 PM, "John Halley Gotway via RT" <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > Hello John,
> > >
> > > I see you have a few questions about the MET
software.
> > >
> > > (1) Yes, you really need to run the madis2nc tool on
the
> MADIS
> > files. I
> > > realize that this seems silly since the MADIS data
is
> already in
> > NetCDF
> > > format. The madis2nc tool is essentially a
reformatter. It
> can
> > also
> > > filter
> > > out obs based on QC flags and can compute time
summaries
> of the
> > point
> > > observations. But the main purpose is to reformat
the point
> > > observations
> > > into a common NetCDF format that the other MET tools
> (point-stat
> > and
> > > ensemble-stat) expect.
> > >
> > > We have done a nice job in MET supporting different
> gridded data
> > files
> > > in
> > > their native format, but we decided not to do that
with
> point
> > > observations.
> > > Instead of reading them directly, we reformat them
into a
> common
> > format
> > > using pb2nc, madis2nc, ascii2nc, lidar2nc, and so
on.
> However we
> > may
> > > redesign that in the future and support the point
obs in
> their
> > native
> > > formats. But for now madis2nc is necessary.
> > >
> > > If your madis2nc output doesn't contain the expected
obs,
> please
> > send
> > > me
> > > the command you're running, along with a sample data
file.
> > >
> > > (2) You should check your WRF output files. You
state that
> you
> > have
> > > hourly
> > > WRF output, but WRF often outputs a runtime
accumulation.
> So that
> > > output
> > > file for a 6-hour forecast contains 0 to 6 hours of
> > accumulation. At
> > > 7-hours you'd get 0 to 7 hours of accumulation.
However,
> WRF
> > really
> > > can be
> > > configured to dump an hourly precip bucket, but
that's not
> the
> > default.
> > >
> > > If you actually have a runtime accumulation, you'd
run
> > pcp_combine to
> > > compute the difference. For example, 7-hour accum -
6-hour
> accum
> > = 1-hr
> > > accum between 6 and 7 hours.
> > >
> > > (3) WRF and GFS are gridded forecasts. MADIS are
point
> > observations.
> > > You
> > > run the Point-Stat tool to verify against point
> observations,
> > not the
> > > Grid-Stat tool.
> > >
> > > Two other things to mention...
> > >
> > > We typically recommend that users post-process their
WRF
> output
> > using
> > > the
> > > Unified Post-Processor (
> > > Blockedhttps://
> > > dtcenter.org/community-code/unified-post-processor-
upp)Blocked
> .
> > That reads
> > > WRF NetCDF output, de-staggers the staggered
variables
> (winds),
> > derives
> > > requested variables, interpolates from WRF's native
> vertical
> > > coordinate to
> > > pressure, and writes GRIB1 or GRIB2 output files. If
you
> really
> > are
> > > only
> > > interested in precip at the surface... and not winds
or any
> > variables
> > > on
> > > pressure levels... then technically you do not have
to run
> UPP.
> > But I
> > > would
> > > still recommend it.
> > >
> > > METplus is a suite of python wrappers intended to
make it
> easier
> > to
> > > automate calls to the MET tools. The intention was
to make
> > getting up
> > > and
> > > running with MET easier, but it's still relatively
new. I
> don't
> > know
> > > whether or not we have an existing use case which
verifies
> WRF
> > output
> > > against MADIS obs, but I'll let George chime in
about that.
> > >
> > > Thanks,
> > > John Halley Gotway
> > >
> > > On Thu, Jun 25, 2020 at 12:19 PM Eylander, John B
> > ERDC-RDE-CRREL-NH
> > > CIV via
> > > RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: Blockedhttps://
> > > rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> > > >
> > > > Thanks!
> > > >
> > > > On 6/25/20, 2:09 PM, "George McCabe via RT" <
> met_help at ucar.edu
> > >
> > > wrote:
> > > >
> > > > Hi John,
> > > >
> > > > I see you have questions about processing
MADIS data
> using
> > the
> > > MET
> > > > tools. I
> > > > have assigned this ticket to John Halley
Gotway, a
> software
> > > engineer
> > > > familiar with these tools. Please allow a few
> business
> > days for a
> > > > response.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Thu, Jun 25, 2020 at 5:46 AM Eylander, John
B
> > > ERDC-RDE-CRREL-NH CIV
> > > > via
> > > > RT <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Thu Jun 25 05:46:04 2020: Request 95731 was
acted
> upon.
> > > > > Transaction: Ticket created by
> > John.B.Eylander at erdc.dren.mil
> > > > > Queue: met_help
> > > > > Subject: Using MET to evaluate WRF with
MADIS
> > NETCDF data
> > > > > Owner: Nobody
> > > > > Requestors: John.B.Eylander at erdc.dren.mil
> > > > > Status: new
> > > > > Ticket <URL: Blockedhttps://
> > > >
rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> > > > >
> > > > >
> > > > > Good morning MET help people!
> > > > >
> > > > > I’m trying to use MET to evaluate
precipitation
> skill in
> > a
> > > bunch of
> > > > model
> > > > > data we have from WRF and GFS runs.
Initially, I’m
> > focused on
> > > a high
> > > > > resolution WRF domain, at 1-hourly intervals
in
> GriB2
> > format to
> > > > compare
> > > > > with MADIS NETCDF output I have (also in 1-
hourly
> files).
> > > I’ve read
> > > > > through the MET users manual to try to
figure out
> what
> > steps I
> > > have
> > > > to
> > > > > perform and how to set up the configuration
files,
> but I
> > am
> > > either
> > > > not
> > > > > understanding the manual or missing
something.
> > > > >
> > > > > 1st question: do I really need to run the
> MADIS2NC tool
> > on the
> > > > NETCDF
> > > > > files? I’ve done so, but the output of
MADIS2NC
> removes
> > most
> > > of the
> > > > data
> > > > > only leaving information about the station
in the
> > resulting
> > > .nc file.
> > > > >
> > > > > Since I’m comparing 1-hourly WRF output to
1-hourly
> > > MADIS/METAR (or
> > > > > mesonet) data, I don’t need to run
PCP_COMBINE (I
> will
> > when
> > > looking
> > > > at
> > > > > other 3-hourly WRF or GFS output).
> > > > >
> > > > > Leading me to my second question: I cannot
figure
> out a
> > proper
> > > > Grid_stat
> > > > > configuration file to compare the WRF data
against
> the
> > MADIS
> > > data.
> > > > I have
> > > > > searched your website, googled the topic,
and
> asked a
> > number of
> > > > others
> > > > > (with no luck). Is there a sample
configuration
> file
> > you can
> > > > provide for
> > > > > comparing WRF 1-hourly files to MADIS NetCDF
data?
> > > > >
> > > > > 3rd question: Am I doing this all wrong?
> > > > >
> > > > > John Eylander
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > John Eylander
> > > > > Hydrologic Systems Branch
> > > > > Engineer Research and Development Center
> > > > > US Army Corp of Engineers
> > > > >
> > > > > John.B.Eylander at usace.army.mil<mailto:
> > > John.B.Eylander at usace.army.mil>
> > > > (U)
> > > > > John.B.Eylander at erdc.dren.mil<mailto:
> > > John.B.Eylander at erdc.dren.mil>
> > > > (U –
> > > > > RDE)
> > > > >
> > > > > John.B.Eylander.civ at mail.smil.mil<mailto:
> > > > John.B.Eylander.civ at mail.smil.mil>
> > > > > (S)
> > > > > John.B.Eylander.civ at army.ic.gov<mailto:
> > > > John.B.Eylander.civ at army.ic.gov>
> > > > > (JWICS)
> > > > >
> > > > > PH: 603-646-4188
> > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day.
Please
> do not
> > feel
> > > obliged
> > > > to
> > > > reply to this email outside of your normal
working
> hours.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
> >
>
>
>
>
>
>
--
Julie Prestopnik
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #95731] Using MET to evaluate WRF with MADIS NETCDF data
From: Eylander, John B ERDC-RDE-CRREL-NH CIV
Time: Fri Jul 10 09:13:41 2020
Julie,
You can disregard my previously stated issues. I figured out my
problem. The errors were coming from the fact that I had the
obs=fcst. The error messages didn't really point me in that
direction, but after looking at it I figured out the errors were
really coming from trying to read in the obs files. Once I set
independent values for the observation files, the point tool ran
correctly.
John
On 7/9/20, 6:37 PM, "Julie Prestopnik via RT" <met_help at ucar.edu>
wrote:
Hi John.
John is out of the office, hence the delayed response. My name is
Julie,
and I work with John. I thought I would jump in and try to help
in his
absence.
I see that you are having trouble with PointStat:
> WARNING: process_fcst_climo_files() -> no fields matching APCPA1
found in
> file: /p/work/eylandej/MET_DATA/20200412/
> nam.t18z.awphys.f02.tm00.20200412.MET_PCP_SUB.nc
MET is having a hard time recognizing the field. From the header
dump that
you sent (thank you!), I can see that the field name is "APCP_01"
and not
"APCP". Setting the "level" field is file-format specific. This
is
described starting on page 59 of our most recent MET User's Guide
<Blockedhttps://dtcenter.org/sites/default/files/community-
code/met/docs/user-guide/MET_Users_Guide_v9.0.3.pdfBlocked>.
You have specified the field using the syntax for GRIB and GRIB2
files.
For NetCDF files, you'll want to specify the level as "(*,*)",
which
specifies the two dimensions for the gridded field. Please try
changing
the "fcst" entries for "name" and "level" as I have below:
fcst = {
> field = [
> {
> name = "APCP_01";
> level = "(*,*)";
> cat_thresh = [ >=0.1, >=0.2, >=0.3, >=0.4, >=0.5, >=1.0,
>=1.5,
> >=2.0, >=3.0 ];
> }
>
> ];
>
> }
>
If your obs values are the same, go ahead and use this as you had
it:
> obs = fcst;
>
Otherwise, you'll need to change that as well.
I hope this helps move you along farther. If it does not, please
feel free
to upload your data to our FTP server as describe here under "How
To Send
Us Data":
Blockedhttps://dtcenter.org/community-code/model-evaluation-tools-
met/met-help-deskBlocked
and also send the command line you are running, and I can try to
get it
working for you.
Julie
On Mon, Jul 6, 2020 at 8:00 PM Eylander, John B ERDC-RDE-CRREL-NH
CIV via
RT <met_help at ucar.edu> wrote:
>
> <URL:
Blockedhttps://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked
>
>
> I'm still struggling to work my way through the comparison of
WRF with
> MADIS data. Here's where I'm at:
>
> I've confirmed that the MADIS2NC results are good. The tool and
my use of
> it produce output that matches what the MET documentation
states I should
> be getting.
>
> I've run PCP_COMBINE on the WRF output to generate 1-hourly APCP
files.
> The resulting NetCDF header for the PCP_COMBINE output files is
below
> (result of ncdump -h). I'm now trying to execute the Point_stat
routine,
> but I get the following error (this is just one version of the
error, I've
> tried to configure the point-stat tool in multiple ways but
can't see to
> get the tool to read in the APCP record. I include my
configuration file
> at the bottom of this. Again...trying to compare MADIS point
obs against
> WRF data is the goal.
>
>
>
**********************************************************************************************
> DEBUG 1: Default Config File:
>
/p/app/unsupported/hydrorf/met/9.0.1/share/met/config/PointStatConfig_default
> DEBUG 1: User Config File:
/p/home/eylandej/configs/PointStatConfig_MADIS
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=584385375
> DEBUG 1: Forecast File: /p/work/eylandej/MET_DATA/20200412/
> nam.t18z.awphys.f02.tm00.20200412.MET_PCP_SUB.nc
> DEBUG 1: Observation File:
>
/gpfs/cwfs/eylandej/MADIS/MADIS2NC_Files/20200412/20200412_2000.nc
> DEBUG 2:
> DEBUG 2:
>
--------------------------------------------------------------------------------
> DEBUG 2:
> DEBUG 2: Reading data for APCPA1.
> WARNING:
> WARNING: process_fcst_climo_files() -> no fields matching APCPA1
found in
> file: /p/work/eylandej/MET_DATA/20200412/
> nam.t18z.awphys.f02.tm00.20200412.MET_PCP_SUB.nc
> WARNING:
> ERROR :
> ERROR : process_fcst_climo_files() -> no requested forecast
data found!
> Exiting...
> ERROR :
>
>
**********************************************************************************************
>
> netcdf nam.t18z.awphys.f06.tm00.20200412.MET_PCP_SUB {
> dimensions:
> lat = 402 ;
> lon = 613 ;
> variables:
> float lat(lat, lon) ;
> lat:long_name = "latitude" ;
> lat:units = "degrees_north" ;
> lat:standard_name = "latitude" ;
> float lon(lat, lon) ;
> lon:long_name = "longitude" ;
> lon:units = "degrees_east" ;
> lon:standard_name = "longitude" ;
> float APCP_01(lat, lon) ;
> APCP_01:name = "APCP_01" ;
> APCP_01:long_name = "Total Precipitation" ;
> APCP_01:level = "A1" ;
> APCP_01:units = "kg/m^2" ;
> APCP_01:_FillValue = -9999.f ;
> APCP_01:init_time = "20200412_180000" ;
> APCP_01:init_time_ut = "1586714400" ;
> APCP_01:valid_time = "20200413_000000" ;
> APCP_01:valid_time_ut = "1586736000" ;
> APCP_01:accum_time = "010000" ;
> APCP_01:accum_time_sec = 3600 ;
>
> // global attributes:
> :FileOrigins = "File
/p/work/eylandej/MET_DATA/20200412/
> nam.t18z.awphys.f06.tm00.20200412.MET_PCP_SUB.nc generated
> 20200706_183200 UTC on host onyx09 by the MET pcp_combine tool"
;
> :MET_version = "V9.0.1" ;
> :MET_tool = "pcp_combine" ;
> :RunCommand = "Subtraction:
>
/p/work/smatus/HydroRF/Archives/NAM/20200412/18/nam.t18z.awphys.f06.tm00.20200412.grib2
> minus
>
>
>
> Configuration file:
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // May be set separately in each "field" entry
> //
> censor_thresh = [];
> censor_val = [];
> cat_thresh = [ NA ];
> cnt_thresh = [ NA ];
> cnt_logic = UNION;
> wind_thresh = [ NA ];
> wind_logic = UNION;
> eclv_points = 0.05;
> rank_corr_flag = FALSE;
>
> //
> // Forecast and observation fields to be verified
> //
> fcst = {
> field = [
> {
> name = "APCP";
> level = ["A1"];
> cat_thresh = [ >=0.1, >=0.2, >=0.3, >=0.4, >=0.5, >=1.0,
>=1.5,
> >=2.0, >=3.0 ];
> }
>
> ];
>
> }
> obs = fcst;
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
> //
> // Point observation filtering options
> // May be set separately in each "obs.field" entry
> //
> message_type = [ "ADPSFC" ];
> sid_inc = [];
> sid_exc = [];
> obs_quality = [];
> duplicate_flag = NONE;
> obs_summary = NONE;
> obs_perc_value = 50;
>
> //
> // Mapping of message type group name to comma-separated list of
values.
> //
> message_type_group_map = [
> { key = "SURFACE"; val = "ADPSFC,SFCSHP,MSONET";
},
> { key = "ANYAIR"; val = "AIRCAR,AIRCFT";
},
> { key = "ANYSFC"; val =
"ADPSFC,SFCSHP,ADPUPA,PROFLR,MSONET"; },
> { key = "ONLYSF"; val = "ADPSFC,SFCSHP";
},
> { key = "LANDSF"; val = "ADPSFC,MSONET";
},
> { key = "WATERSF"; val = "SFCSHP";
}
> ];
>
>
>
////////////////////////////////////////////////////////////////////////////////
>
>
> John Eylander
>
>
>
>
>
>
>
/p/work/smatus/HydroRF/Archives/NAM/20200412/18/nam.t18z.awphys.f05.tm00.20200412.grib2"
> ;
>
> On 6/30/20, 12:02 PM, "John Halley Gotway via RT"
<met_help at ucar.edu>
> wrote:
>
> Ah forgot to paste the link!
> Blockedhttps://
>
github.com/NCAR/MET/blob/master_v9.0/met/data/config/READMEBlocked
>
> Doing too many things at once.
>
> John
>
> On Tue, Jun 30, 2020 at 10:00 AM Eylander, John B ERDC-RDE-
CRREL-NH
> CIV via
> RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: Blockedhttps://
> rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> >
> > The README either didn't get attached or it didn't make it
here.
> Can you
> > resend?
> >
> > On 6/30/20, 11:55 AM, "John Halley Gotway via RT"
<met_help at ucar.edu
> >
> > wrote:
> >
> > John,
> >
> > When working with GRIB1 or GRIB2 files, you request
data from
> them by
> > specifying a "name" and "level". Just set to the
"name" to the
> GRIB
> > abbreviation for the data... e.g. name = "APCP" for
accumulated
> > precip. For
> > GRIB data, the "level" string starts with A, P, Z, L,
or R. A is
> for
> > accumulation interval, P is for pressure level, Z is
for height
> level,
> > L is
> > for a generic level type, and R is for a specific
record number.
> >
> > So name="APCP"; level="A84"; would request a field of
84-hour
> > accumulated
> > precip.
> >
> > Just run wgrib2 on your data file to see what precip
> accumulation is
> > present there:
> >
> > wgrib2 nam.t18z.awphys.f84.tm00.20200412.grib2 | grep
APCP
> >
> > And take a look at line 609 of this README file for
some more
> info on
> > GRIB
> > levels.
> >
> > Thanks,
> > John
> >
> > On Tue, Jun 30, 2020 at 9:07 AM Eylander, John B
> ERDC-RDE-CRREL-NH CIV
> > via
> > RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: Blockedhttps://
> > rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> > >
> > > I am now trying to execute PCP_COMBINE on the WRF
GriB2
> (already
> > > post-processed) data to separate the ACPC fields
into 1-hour
> > accumulation
> > > field. Based on my reading of the MET manual and
after
> exploring
> > several
> > > online scenarios I found, I thought the execution of
> PCP_COMBINE
> > should
> > > look like so:
> > >
> > > ${MET_BIN}/pcp_combine -subtract
> > nam.t18z.awphys.f84.tm00.20200412.grib2
> > > 84 nam.t18z.awphys.f81.tm00.20200412.grib2 81
> $HOME/test.out -v 3
> > >
> > > However, I get the following error/diagnostic output
when
> trying to
> > run
> > > pcp_combine:
> > >
> > > DEBUG 2: Since the "-field" command line option was
not used,
> > parsing the
> > > command line arguments a list of files, each
followed by a
> > configuration
> > > string.
> > > DEBUG 1: Reading input file:
> nam.t18z.awphys.f84.tm00.20200412.grib2
> > > DEBUG 1: Reading data (name="APCP"; level="A84";)
from input
> file:
> > > nam.t18z.awphys.f84.tm00.20200412.grib2
> > > WARNING:
> > > WARNING: MetGrib2DataFile::data_plane() -> No
matching record
> found
> > for
> > > 'APCP/A84'
> > > WARNING:
> > > ERROR :
> > > ERROR : get_field() -> can't get data plane from
file
> > > "nam.t18z.awphys.f84.tm00.20200412.grib2"
> > > ERROR :
> > >
> > > What I can't understand is why the "level" field is
set to
> "A84".
> > From
> > > what I understand of the manual, the APCP field
should be
> > automatically
> > > able to find APCP in the GriB2 output. However,
this is not
> > happening
> > > because the level of APCP should be "surface", not
"A84".
> > >
> > > When I try to explicitly set the configuration to
set the
> level to
> > > "surface", I still get an error:
> > >
> > > $MET_BIN/pcp_combine -subtract
> > nam.t18z.awphys.f81.tm00.20200412.grib2
> > > 'name="APCP"; level="surface";'
> > nam.t18z.awphys.f84.tm00.20200412.grib2
> > > 'name="APCP"; level="surface";' $HOME/test.out -v 3
> > >
> > > DEBUG 2: Since the "-field" command line option was
not used,
> > parsing the
> > > command line arguments a list of files, each
followed by a
> > configuration
> > > string.
> > > DEBUG 1: Reading input file:
> nam.t18z.awphys.f81.tm00.20200412.grib2
> > > DEBUG 1: Reading data (name="APCP";
level="surface";) from
> input
> > file:
> > > nam.t18z.awphys.f81.tm00.20200412.grib2
> > > ERROR :
> > > ERROR : VarInfo::set_level_info_grib() - failed to
parse level
> > string
> > > 'surface'
> > > ERROR :
> > >
> > > Thanks in advance for any help you can provide.
> > >
> > > John Eylander
> > >
> > >
> > > On 6/25/20, 5:24 PM, "John Halley Gotway via RT" <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > Hello John,
> > >
> > > I see you have a few questions about the MET
software.
> > >
> > > (1) Yes, you really need to run the madis2nc
tool on the
> MADIS
> > files. I
> > > realize that this seems silly since the MADIS
data is
> already in
> > NetCDF
> > > format. The madis2nc tool is essentially a
reformatter. It
> can
> > also
> > > filter
> > > out obs based on QC flags and can compute time
summaries
> of the
> > point
> > > observations. But the main purpose is to
reformat the point
> > > observations
> > > into a common NetCDF format that the other MET
tools
> (point-stat
> > and
> > > ensemble-stat) expect.
> > >
> > > We have done a nice job in MET supporting
different
> gridded data
> > files
> > > in
> > > their native format, but we decided not to do
that with
> point
> > > observations.
> > > Instead of reading them directly, we reformat
them into a
> common
> > format
> > > using pb2nc, madis2nc, ascii2nc, lidar2nc, and
so on.
> However we
> > may
> > > redesign that in the future and support the
point obs in
> their
> > native
> > > formats. But for now madis2nc is necessary.
> > >
> > > If your madis2nc output doesn't contain the
expected obs,
> please
> > send
> > > me
> > > the command you're running, along with a sample
data file.
> > >
> > > (2) You should check your WRF output files. You
state that
> you
> > have
> > > hourly
> > > WRF output, but WRF often outputs a runtime
accumulation.
> So that
> > > output
> > > file for a 6-hour forecast contains 0 to 6 hours
of
> > accumulation. At
> > > 7-hours you'd get 0 to 7 hours of accumulation.
However,
> WRF
> > really
> > > can be
> > > configured to dump an hourly precip bucket, but
that's not
> the
> > default.
> > >
> > > If you actually have a runtime accumulation,
you'd run
> > pcp_combine to
> > > compute the difference. For example, 7-hour
accum - 6-hour
> accum
> > = 1-hr
> > > accum between 6 and 7 hours.
> > >
> > > (3) WRF and GFS are gridded forecasts. MADIS are
point
> > observations.
> > > You
> > > run the Point-Stat tool to verify against point
> observations,
> > not the
> > > Grid-Stat tool.
> > >
> > > Two other things to mention...
> > >
> > > We typically recommend that users post-process
their WRF
> output
> > using
> > > the
> > > Unified Post-Processor (
> > > Blockedhttps://
> > > dtcenter.org/community-code/unified-post-processor-
upp)Blocked
> .
> > That reads
> > > WRF NetCDF output, de-staggers the staggered
variables
> (winds),
> > derives
> > > requested variables, interpolates from WRF's
native
> vertical
> > > coordinate to
> > > pressure, and writes GRIB1 or GRIB2 output
files. If you
> really
> > are
> > > only
> > > interested in precip at the surface... and not
winds or any
> > variables
> > > on
> > > pressure levels... then technically you do not
have to run
> UPP.
> > But I
> > > would
> > > still recommend it.
> > >
> > > METplus is a suite of python wrappers intended
to make it
> easier
> > to
> > > automate calls to the MET tools. The intention
was to make
> > getting up
> > > and
> > > running with MET easier, but it's still
relatively new. I
> don't
> > know
> > > whether or not we have an existing use case
which verifies
> WRF
> > output
> > > against MADIS obs, but I'll let George chime in
about that.
> > >
> > > Thanks,
> > > John Halley Gotway
> > >
> > > On Thu, Jun 25, 2020 at 12:19 PM Eylander, John
B
> > ERDC-RDE-CRREL-NH
> > > CIV via
> > > RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: Blockedhttps://
> > >
rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> > > >
> > > > Thanks!
> > > >
> > > > On 6/25/20, 2:09 PM, "George McCabe via RT" <
> met_help at ucar.edu
> > >
> > > wrote:
> > > >
> > > > Hi John,
> > > >
> > > > I see you have questions about processing
MADIS data
> using
> > the
> > > MET
> > > > tools. I
> > > > have assigned this ticket to John Halley
Gotway, a
> software
> > > engineer
> > > > familiar with these tools. Please allow a
few
> business
> > days for a
> > > > response.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Thu, Jun 25, 2020 at 5:46 AM Eylander,
John B
> > > ERDC-RDE-CRREL-NH CIV
> > > > via
> > > > RT <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Thu Jun 25 05:46:04 2020: Request 95731
was acted
> upon.
> > > > > Transaction: Ticket created by
> > John.B.Eylander at erdc.dren.mil
> > > > > Queue: met_help
> > > > > Subject: Using MET to evaluate WRF
with MADIS
> > NETCDF data
> > > > > Owner: Nobody
> > > > > Requestors:
John.B.Eylander at erdc.dren.mil
> > > > > Status: new
> > > > > Ticket <URL: Blockedhttps://
> > > >
rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> > > > >
> > > > >
> > > > > Good morning MET help people!
> > > > >
> > > > > I’m trying to use MET to evaluate
precipitation
> skill in
> > a
> > > bunch of
> > > > model
> > > > > data we have from WRF and GFS runs.
Initially, I’m
> > focused on
> > > a high
> > > > > resolution WRF domain, at 1-hourly
intervals in
> GriB2
> > format to
> > > > compare
> > > > > with MADIS NETCDF output I have (also in
1-hourly
> files).
> > > I’ve read
> > > > > through the MET users manual to try to
figure out
> what
> > steps I
> > > have
> > > > to
> > > > > perform and how to set up the
configuration files,
> but I
> > am
> > > either
> > > > not
> > > > > understanding the manual or missing
something.
> > > > >
> > > > > 1st question: do I really need to run
the
> MADIS2NC tool
> > on the
> > > > NETCDF
> > > > > files? I’ve done so, but the output of
MADIS2NC
> removes
> > most
> > > of the
> > > > data
> > > > > only leaving information about the
station in the
> > resulting
> > > .nc file.
> > > > >
> > > > > Since I’m comparing 1-hourly WRF output
to 1-hourly
> > > MADIS/METAR (or
> > > > > mesonet) data, I don’t need to run
PCP_COMBINE (I
> will
> > when
> > > looking
> > > > at
> > > > > other 3-hourly WRF or GFS output).
> > > > >
> > > > > Leading me to my second question: I
cannot figure
> out a
> > proper
> > > > Grid_stat
> > > > > configuration file to compare the WRF
data against
> the
> > MADIS
> > > data.
> > > > I have
> > > > > searched your website, googled the
topic, and
> asked a
> > number of
> > > > others
> > > > > (with no luck). Is there a sample
configuration
> file
> > you can
> > > > provide for
> > > > > comparing WRF 1-hourly files to MADIS
NetCDF data?
> > > > >
> > > > > 3rd question: Am I doing this all
wrong?
> > > > >
> > > > > John Eylander
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > John Eylander
> > > > > Hydrologic Systems Branch
> > > > > Engineer Research and Development Center
> > > > > US Army Corp of Engineers
> > > > >
> > > > > John.B.Eylander at usace.army.mil<mailto:
> > > John.B.Eylander at usace.army.mil>
> > > > (U)
> > > > > John.B.Eylander at erdc.dren.mil<mailto:
> > > John.B.Eylander at erdc.dren.mil>
> > > > (U –
> > > > > RDE)
> > > > >
> > > > >
John.B.Eylander.civ at mail.smil.mil<mailto:
> > > > John.B.Eylander.civ at mail.smil.mil>
> > > > > (S)
> > > > > John.B.Eylander.civ at army.ic.gov<mailto:
> > > > John.B.Eylander.civ at army.ic.gov>
> > > > > (JWICS)
> > > > >
> > > > > PH: 603-646-4188
> > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working
day. Please
> do not
> > feel
> > > obliged
> > > > to
> > > > reply to this email outside of your normal
working
> hours.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
> >
>
>
>
>
>
>
--
Julie Prestopnik
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Using MET to evaluate WRF with MADIS NETCDF data
From: Julie Prestopnik
Time: Fri Jul 10 09:32:59 2020
Great, John! Thank you for letting us know. I'm so glad you got it
working. I'll go ahead and close this ticket. Please feel free to
open a
new ticket with any other questions you have.
Julie
On Fri, Jul 10, 2020 at 9:13 AM Eylander, John B ERDC-RDE-CRREL-NH CIV
via
RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731 >
>
> Julie,
>
> You can disregard my previously stated issues. I figured out my
problem.
> The errors were coming from the fact that I had the obs=fcst. The
error
> messages didn't really point me in that direction, but after looking
at it
> I figured out the errors were really coming from trying to read in
the obs
> files. Once I set independent values for the observation files, the
point
> tool ran correctly.
>
> John
>
> On 7/9/20, 6:37 PM, "Julie Prestopnik via RT" <met_help at ucar.edu>
wrote:
>
> Hi John.
>
> John is out of the office, hence the delayed response. My name
is
> Julie,
> and I work with John. I thought I would jump in and try to help
in his
> absence.
>
> I see that you are having trouble with PointStat:
>
> > WARNING: process_fcst_climo_files() -> no fields matching
APCPA1
> found in
> > file: /p/work/eylandej/MET_DATA/20200412/
> > nam.t18z.awphys.f02.tm00.20200412.MET_PCP_SUB.nc
>
>
> MET is having a hard time recognizing the field. From the
header dump
> that
> you sent (thank you!), I can see that the field name is
"APCP_01" and
> not
> "APCP". Setting the "level" field is file-format specific.
This is
> described starting on page 59 of our most recent MET User's
Guide
> <Blockedhttps://
> dtcenter.org/sites/default/files/community-code/met/docs/user-
guide/MET_Users_Guide_v9.0.3.pdfBlocked
> >.
> You have specified the field using the syntax for GRIB and GRIB2
files.
> For NetCDF files, you'll want to specify the level as "(*,*)",
which
> specifies the two dimensions for the gridded field. Please try
> changing
> the "fcst" entries for "name" and "level" as I have below:
>
> fcst = {
> > field = [
> > {
> > name = "APCP_01";
> > level = "(*,*)";
> > cat_thresh = [ >=0.1, >=0.2, >=0.3, >=0.4, >=0.5,
>=1.0,
> >=1.5,
> > >=2.0, >=3.0 ];
> > }
> >
> > ];
> >
> > }
> >
>
> If your obs values are the same, go ahead and use this as you
had it:
>
> > obs = fcst;
> >
>
> Otherwise, you'll need to change that as well.
>
> I hope this helps move you along farther. If it does not,
please feel
> free
> to upload your data to our FTP server as describe here under
"How To
> Send
> Us Data":
> Blockedhttps://
> dtcenter.org/community-code/model-evaluation-tools-met/met-help-
deskBlocked
>
> and also send the command line you are running, and I can try to
get it
> working for you.
>
> Julie
>
>
>
>
> On Mon, Jul 6, 2020 at 8:00 PM Eylander, John B ERDC-RDE-CRREL-
NH CIV
> via
> RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: Blockedhttps://
> rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> >
> > I'm still struggling to work my way through the comparison of
WRF
> with
> > MADIS data. Here's where I'm at:
> >
> > I've confirmed that the MADIS2NC results are good. The tool
and my
> use of
> > it produce output that matches what the MET documentation
states I
> should
> > be getting.
> >
> > I've run PCP_COMBINE on the WRF output to generate 1-hourly
APCP
> files.
> > The resulting NetCDF header for the PCP_COMBINE output files
is below
> > (result of ncdump -h). I'm now trying to execute the
Point_stat
> routine,
> > but I get the following error (this is just one version of the
> error, I've
> > tried to configure the point-stat tool in multiple ways but
can't
> see to
> > get the tool to read in the APCP record. I include my
configuration
> file
> > at the bottom of this. Again...trying to compare MADIS point
obs
> against
> > WRF data is the goal.
> >
> >
> >
>
**********************************************************************************************
> > DEBUG 1: Default Config File:
> >
>
/p/app/unsupported/hydrorf/met/9.0.1/share/met/config/PointStatConfig_default
> > DEBUG 1: User Config File:
> /p/home/eylandej/configs/PointStatConfig_MADIS
> > GSL_RNG_TYPE=mt19937
> > GSL_RNG_SEED=584385375
> > DEBUG 1: Forecast File: /p/work/eylandej/MET_DATA/20200412/
> > nam.t18z.awphys.f02.tm00.20200412.MET_PCP_SUB.nc
> > DEBUG 1: Observation File:
> >
/gpfs/cwfs/eylandej/MADIS/MADIS2NC_Files/20200412/20200412_2000.nc
> > DEBUG 2:
> > DEBUG 2:
> >
>
--------------------------------------------------------------------------------
> > DEBUG 2:
> > DEBUG 2: Reading data for APCPA1.
> > WARNING:
> > WARNING: process_fcst_climo_files() -> no fields matching
APCPA1
> found in
> > file: /p/work/eylandej/MET_DATA/20200412/
> > nam.t18z.awphys.f02.tm00.20200412.MET_PCP_SUB.nc
> > WARNING:
> > ERROR :
> > ERROR : process_fcst_climo_files() -> no requested forecast
data
> found!
> > Exiting...
> > ERROR :
> >
> >
>
**********************************************************************************************
> >
> > netcdf nam.t18z.awphys.f06.tm00.20200412.MET_PCP_SUB {
> > dimensions:
> > lat = 402 ;
> > lon = 613 ;
> > variables:
> > float lat(lat, lon) ;
> > lat:long_name = "latitude" ;
> > lat:units = "degrees_north" ;
> > lat:standard_name = "latitude" ;
> > float lon(lat, lon) ;
> > lon:long_name = "longitude" ;
> > lon:units = "degrees_east" ;
> > lon:standard_name = "longitude" ;
> > float APCP_01(lat, lon) ;
> > APCP_01:name = "APCP_01" ;
> > APCP_01:long_name = "Total Precipitation" ;
> > APCP_01:level = "A1" ;
> > APCP_01:units = "kg/m^2" ;
> > APCP_01:_FillValue = -9999.f ;
> > APCP_01:init_time = "20200412_180000" ;
> > APCP_01:init_time_ut = "1586714400" ;
> > APCP_01:valid_time = "20200413_000000" ;
> > APCP_01:valid_time_ut = "1586736000" ;
> > APCP_01:accum_time = "010000" ;
> > APCP_01:accum_time_sec = 3600 ;
> >
> > // global attributes:
> > :FileOrigins = "File
> /p/work/eylandej/MET_DATA/20200412/
> > nam.t18z.awphys.f06.tm00.20200412.MET_PCP_SUB.nc generated
> > 20200706_183200 UTC on host onyx09 by the MET pcp_combine
tool" ;
> > :MET_version = "V9.0.1" ;
> > :MET_tool = "pcp_combine" ;
> > :RunCommand = "Subtraction:
> >
>
/p/work/smatus/HydroRF/Archives/NAM/20200412/18/nam.t18z.awphys.f06.tm00.20200412.grib2
> > minus
> >
> >
> >
> > Configuration file:
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // May be set separately in each "field" entry
> > //
> > censor_thresh = [];
> > censor_val = [];
> > cat_thresh = [ NA ];
> > cnt_thresh = [ NA ];
> > cnt_logic = UNION;
> > wind_thresh = [ NA ];
> > wind_logic = UNION;
> > eclv_points = 0.05;
> > rank_corr_flag = FALSE;
> >
> > //
> > // Forecast and observation fields to be verified
> > //
> > fcst = {
> > field = [
> > {
> > name = "APCP";
> > level = ["A1"];
> > cat_thresh = [ >=0.1, >=0.2, >=0.3, >=0.4, >=0.5,
>=1.0,
> >=1.5,
> > >=2.0, >=3.0 ];
> > }
> >
> > ];
> >
> > }
> > obs = fcst;
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> > //
> > // Point observation filtering options
> > // May be set separately in each "obs.field" entry
> > //
> > message_type = [ "ADPSFC" ];
> > sid_inc = [];
> > sid_exc = [];
> > obs_quality = [];
> > duplicate_flag = NONE;
> > obs_summary = NONE;
> > obs_perc_value = 50;
> >
> > //
> > // Mapping of message type group name to comma-separated list
of
> values.
> > //
> > message_type_group_map = [
> > { key = "SURFACE"; val = "ADPSFC,SFCSHP,MSONET";
},
> > { key = "ANYAIR"; val = "AIRCAR,AIRCFT";
},
> > { key = "ANYSFC"; val =
"ADPSFC,SFCSHP,ADPUPA,PROFLR,MSONET"; },
> > { key = "ONLYSF"; val = "ADPSFC,SFCSHP";
},
> > { key = "LANDSF"; val = "ADPSFC,MSONET";
},
> > { key = "WATERSF"; val = "SFCSHP";
}
> > ];
> >
> >
> >
>
////////////////////////////////////////////////////////////////////////////////
> >
> >
> > John Eylander
> >
> >
> >
> >
> >
> >
> >
>
/p/work/smatus/HydroRF/Archives/NAM/20200412/18/nam.t18z.awphys.f05.tm00.20200412.grib2"
> > ;
> >
> > On 6/30/20, 12:02 PM, "John Halley Gotway via RT"
<met_help at ucar.edu
> >
> > wrote:
> >
> > Ah forgot to paste the link!
> > Blockedhttps://
> >
github.com/NCAR/MET/blob/master_v9.0/met/data/config/READMEBlocked
> >
> > Doing too many things at once.
> >
> > John
> >
> > On Tue, Jun 30, 2020 at 10:00 AM Eylander, John B
> ERDC-RDE-CRREL-NH
> > CIV via
> > RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: Blockedhttps://
> > rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> > >
> > > The README either didn't get attached or it didn't make
it
> here.
> > Can you
> > > resend?
> > >
> > > On 6/30/20, 11:55 AM, "John Halley Gotway via RT" <
> met_help at ucar.edu
> > >
> > > wrote:
> > >
> > > John,
> > >
> > > When working with GRIB1 or GRIB2 files, you request
data
> from
> > them by
> > > specifying a "name" and "level". Just set to the
"name" to
> the
> > GRIB
> > > abbreviation for the data... e.g. name = "APCP" for
> accumulated
> > > precip. For
> > > GRIB data, the "level" string starts with A, P, Z,
L, or
> R. A is
> > for
> > > accumulation interval, P is for pressure level, Z is
for
> height
> > level,
> > > L is
> > > for a generic level type, and R is for a specific
record
> number.
> > >
> > > So name="APCP"; level="A84"; would request a field
of
> 84-hour
> > > accumulated
> > > precip.
> > >
> > > Just run wgrib2 on your data file to see what precip
> > accumulation is
> > > present there:
> > >
> > > wgrib2 nam.t18z.awphys.f84.tm00.20200412.grib2 |
grep APCP
> > >
> > > And take a look at line 609 of this README file for
some
> more
> > info on
> > > GRIB
> > > levels.
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Jun 30, 2020 at 9:07 AM Eylander, John B
> > ERDC-RDE-CRREL-NH CIV
> > > via
> > > RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: Blockedhttps://
> > > rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> > > >
> > > > I am now trying to execute PCP_COMBINE on the WRF
GriB2
> > (already
> > > > post-processed) data to separate the ACPC fields
into
> 1-hour
> > > accumulation
> > > > field. Based on my reading of the MET manual and
after
> > exploring
> > > several
> > > > online scenarios I found, I thought the execution
of
> > PCP_COMBINE
> > > should
> > > > look like so:
> > > >
> > > > ${MET_BIN}/pcp_combine -subtract
> > > nam.t18z.awphys.f84.tm00.20200412.grib2
> > > > 84 nam.t18z.awphys.f81.tm00.20200412.grib2 81
> > $HOME/test.out -v 3
> > > >
> > > > However, I get the following error/diagnostic
output when
> > trying to
> > > run
> > > > pcp_combine:
> > > >
> > > > DEBUG 2: Since the "-field" command line option
was not
> used,
> > > parsing the
> > > > command line arguments a list of files, each
followed by
> a
> > > configuration
> > > > string.
> > > > DEBUG 1: Reading input file:
> > nam.t18z.awphys.f84.tm00.20200412.grib2
> > > > DEBUG 1: Reading data (name="APCP"; level="A84";)
from
> input
> > file:
> > > > nam.t18z.awphys.f84.tm00.20200412.grib2
> > > > WARNING:
> > > > WARNING: MetGrib2DataFile::data_plane() -> No
matching
> record
> > found
> > > for
> > > > 'APCP/A84'
> > > > WARNING:
> > > > ERROR :
> > > > ERROR : get_field() -> can't get data plane from
file
> > > > "nam.t18z.awphys.f84.tm00.20200412.grib2"
> > > > ERROR :
> > > >
> > > > What I can't understand is why the "level" field
is set
> to
> > "A84".
> > > From
> > > > what I understand of the manual, the APCP field
should be
> > > automatically
> > > > able to find APCP in the GriB2 output. However,
this is
> not
> > > happening
> > > > because the level of APCP should be "surface", not
"A84".
> > > >
> > > > When I try to explicitly set the configuration to
set the
> > level to
> > > > "surface", I still get an error:
> > > >
> > > > $MET_BIN/pcp_combine -subtract
> > > nam.t18z.awphys.f81.tm00.20200412.grib2
> > > > 'name="APCP"; level="surface";'
> > > nam.t18z.awphys.f84.tm00.20200412.grib2
> > > > 'name="APCP"; level="surface";' $HOME/test.out -v
3
> > > >
> > > > DEBUG 2: Since the "-field" command line option
was not
> used,
> > > parsing the
> > > > command line arguments a list of files, each
followed by
> a
> > > configuration
> > > > string.
> > > > DEBUG 1: Reading input file:
> > nam.t18z.awphys.f81.tm00.20200412.grib2
> > > > DEBUG 1: Reading data (name="APCP";
level="surface";)
> from
> > input
> > > file:
> > > > nam.t18z.awphys.f81.tm00.20200412.grib2
> > > > ERROR :
> > > > ERROR : VarInfo::set_level_info_grib() - failed
to
> parse level
> > > string
> > > > 'surface'
> > > > ERROR :
> > > >
> > > > Thanks in advance for any help you can provide.
> > > >
> > > > John Eylander
> > > >
> > > >
> > > > On 6/25/20, 5:24 PM, "John Halley Gotway via RT" <
> > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > Hello John,
> > > >
> > > > I see you have a few questions about the MET
> software.
> > > >
> > > > (1) Yes, you really need to run the madis2nc
tool on
> the
> > MADIS
> > > files. I
> > > > realize that this seems silly since the MADIS
data is
> > already in
> > > NetCDF
> > > > format. The madis2nc tool is essentially a
> reformatter. It
> > can
> > > also
> > > > filter
> > > > out obs based on QC flags and can compute time
> summaries
> > of the
> > > point
> > > > observations. But the main purpose is to
reformat
> the point
> > > > observations
> > > > into a common NetCDF format that the other MET
tools
> > (point-stat
> > > and
> > > > ensemble-stat) expect.
> > > >
> > > > We have done a nice job in MET supporting
different
> > gridded data
> > > files
> > > > in
> > > > their native format, but we decided not to do
that
> with
> > point
> > > > observations.
> > > > Instead of reading them directly, we reformat
them
> into a
> > common
> > > format
> > > > using pb2nc, madis2nc, ascii2nc, lidar2nc, and
so on.
> > However we
> > > may
> > > > redesign that in the future and support the
point
> obs in
> > their
> > > native
> > > > formats. But for now madis2nc is necessary.
> > > >
> > > > If your madis2nc output doesn't contain the
expected
> obs,
> > please
> > > send
> > > > me
> > > > the command you're running, along with a
sample data
> file.
> > > >
> > > > (2) You should check your WRF output files.
You
> state that
> > you
> > > have
> > > > hourly
> > > > WRF output, but WRF often outputs a runtime
> accumulation.
> > So that
> > > > output
> > > > file for a 6-hour forecast contains 0 to 6
hours of
> > > accumulation. At
> > > > 7-hours you'd get 0 to 7 hours of
accumulation.
> However,
> > WRF
> > > really
> > > > can be
> > > > configured to dump an hourly precip bucket,
but
> that's not
> > the
> > > default.
> > > >
> > > > If you actually have a runtime accumulation,
you'd
> run
> > > pcp_combine to
> > > > compute the difference. For example, 7-hour
accum -
> 6-hour
> > accum
> > > = 1-hr
> > > > accum between 6 and 7 hours.
> > > >
> > > > (3) WRF and GFS are gridded forecasts. MADIS
are
> point
> > > observations.
> > > > You
> > > > run the Point-Stat tool to verify against
point
> > observations,
> > > not the
> > > > Grid-Stat tool.
> > > >
> > > > Two other things to mention...
> > > >
> > > > We typically recommend that users post-process
their
> WRF
> > output
> > > using
> > > > the
> > > > Unified Post-Processor (
> > > > Blockedhttps://
> > > >
> dtcenter.org/community-code/unified-post-processor-upp)Blocked
> > .
> > > That reads
> > > > WRF NetCDF output, de-staggers the staggered
> variables
> > (winds),
> > > derives
> > > > requested variables, interpolates from WRF's
native
> > vertical
> > > > coordinate to
> > > > pressure, and writes GRIB1 or GRIB2 output
files. If
> you
> > really
> > > are
> > > > only
> > > > interested in precip at the surface... and not
winds
> or any
> > > variables
> > > > on
> > > > pressure levels... then technically you do not
have
> to run
> > UPP.
> > > But I
> > > > would
> > > > still recommend it.
> > > >
> > > > METplus is a suite of python wrappers intended
to
> make it
> > easier
> > > to
> > > > automate calls to the MET tools. The intention
was
> to make
> > > getting up
> > > > and
> > > > running with MET easier, but it's still
relatively
> new. I
> > don't
> > > know
> > > > whether or not we have an existing use case
which
> verifies
> > WRF
> > > output
> > > > against MADIS obs, but I'll let George chime
in
> about that.
> > > >
> > > > Thanks,
> > > > John Halley Gotway
> > > >
> > > > On Thu, Jun 25, 2020 at 12:19 PM Eylander,
John B
> > > ERDC-RDE-CRREL-NH
> > > > CIV via
> > > > RT <met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL: Blockedhttps://
> > > >
rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> > > > >
> > > > > Thanks!
> > > > >
> > > > > On 6/25/20, 2:09 PM, "George McCabe via RT"
<
> > met_help at ucar.edu
> > > >
> > > > wrote:
> > > > >
> > > > > Hi John,
> > > > >
> > > > > I see you have questions about
processing
> MADIS data
> > using
> > > the
> > > > MET
> > > > > tools. I
> > > > > have assigned this ticket to John Halley
> Gotway, a
> > software
> > > > engineer
> > > > > familiar with these tools. Please allow
a few
> > business
> > > days for a
> > > > > response.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > > On Thu, Jun 25, 2020 at 5:46 AM
Eylander, John
> B
> > > > ERDC-RDE-CRREL-NH CIV
> > > > > via
> > > > > RT <met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Thu Jun 25 05:46:04 2020: Request
95731 was
> acted
> > upon.
> > > > > > Transaction: Ticket created by
> > > John.B.Eylander at erdc.dren.mil
> > > > > > Queue: met_help
> > > > > > Subject: Using MET to evaluate
WRF with
> MADIS
> > > NETCDF data
> > > > > > Owner: Nobody
> > > > > > Requestors:
John.B.Eylander at erdc.dren.mil
> > > > > > Status: new
> > > > > > Ticket <URL: Blockedhttps://
> > > > >
> rt.rap.ucar.edu/rt/Ticket/Display.html?id=95731Blocked >
> > > > > >
> > > > > >
> > > > > > Good morning MET help people!
> > > > > >
> > > > > > I’m trying to use MET to evaluate
> precipitation
> > skill in
> > > a
> > > > bunch of
> > > > > model
> > > > > > data we have from WRF and GFS runs.
> Initially, I’m
> > > focused on
> > > > a high
> > > > > > resolution WRF domain, at 1-hourly
intervals
> in
> > GriB2
> > > format to
> > > > > compare
> > > > > > with MADIS NETCDF output I have (also
in
> 1-hourly
> > files).
> > > > I’ve read
> > > > > > through the MET users manual to try to
> figure out
> > what
> > > steps I
> > > > have
> > > > > to
> > > > > > perform and how to set up the
configuration
> files,
> > but I
> > > am
> > > > either
> > > > > not
> > > > > > understanding the manual or missing
> something.
> > > > > >
> > > > > > 1st question: do I really need to run
the
> > MADIS2NC tool
> > > on the
> > > > > NETCDF
> > > > > > files? I’ve done so, but the output
of
> MADIS2NC
> > removes
> > > most
> > > > of the
> > > > > data
> > > > > > only leaving information about the
station
> in the
> > > resulting
> > > > .nc file.
> > > > > >
> > > > > > Since I’m comparing 1-hourly WRF
output to
> 1-hourly
> > > > MADIS/METAR (or
> > > > > > mesonet) data, I don’t need to run
> PCP_COMBINE (I
> > will
> > > when
> > > > looking
> > > > > at
> > > > > > other 3-hourly WRF or GFS output).
> > > > > >
> > > > > > Leading me to my second question: I
cannot
> figure
> > out a
> > > proper
> > > > > Grid_stat
> > > > > > configuration file to compare the WRF
data
> against
> > the
> > > MADIS
> > > > data.
> > > > > I have
> > > > > > searched your website, googled the
topic, and
> > asked a
> > > number of
> > > > > others
> > > > > > (with no luck). Is there a sample
> configuration
> > file
> > > you can
> > > > > provide for
> > > > > > comparing WRF 1-hourly files to MADIS
NetCDF
> data?
> > > > > >
> > > > > > 3rd question: Am I doing this all
wrong?
> > > > > >
> > > > > > John Eylander
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > John Eylander
> > > > > > Hydrologic Systems Branch
> > > > > > Engineer Research and Development
Center
> > > > > > US Army Corp of Engineers
> > > > > >
> > > > > > John.B.Eylander at usace.army.mil<mailto:
> > > > John.B.Eylander at usace.army.mil>
> > > > > (U)
> > > > > > John.B.Eylander at erdc.dren.mil<mailto:
> > > > John.B.Eylander at erdc.dren.mil>
> > > > > (U –
> > > > > > RDE)
> > > > > >
> > > > > >
John.B.Eylander.civ at mail.smil.mil<mailto:
> > > > > John.B.Eylander.civ at mail.smil.mil>
> > > > > > (S)
> > > > > >
John.B.Eylander.civ at army.ic.gov<mailto:
> > > > > John.B.Eylander.civ at army.ic.gov>
> > > > > > (JWICS)
> > > > > >
> > > > > > PH: 603-646-4188
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working
day.
> Please
> > do not
> > > feel
> > > > obliged
> > > > > to
> > > > > reply to this email outside of your
normal
> working
> > hours.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
> >
>
> --
> Julie Prestopnik
> Software Engineer
> National Center for Atmospheric Research
> Research Applications Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
> My working day may not be your working day. Please do not feel
> obliged to
> reply to this email outside of your normal working hours.
>
>
>
>
>
>
--
Julie Prestopnik
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
More information about the Met_help
mailing list