[Met_help] [rt.rap.ucar.edu #93726] History for Wind direction verification
John Halley Gotway via RT
met_help at ucar.edu
Wed Apr 1 11:12:43 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Greetings MET Help
I was wondering if you had an update of the status of wind direction
verification? Looking at the slides you have online from 2018, it seems
that wind dir verification is under development. The slides also suggest
that the verification would use the U and V components instead of a wind
direction field. I just want to make sure that this info is still
accurate. We are currently archiving the URMA wind direction field, so if
we need to change our archive, I want to do that ASAP. If there is other
info on what will be supported or what version it is expected to be in that
I am missing, please let me know.
Thanks
John
--
John Wagner
Verification Task Lead
COR Task Manager
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Branch
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Wind direction verification
From: Minna Win
Time: Thu Jan 09 09:36:25 2020
Hi John,
It looks like you are interested in doing some wind direction
verification
in MET. I believe the best person to address this status is the
engineering lead, John Halley Gotway. I am going to assign this
ticket to
him. Please allow a few business days for a full response.
Regards,
Minna
---------------
Minna Win
National Center for Atmospheric Research
Developmental Testbed Center
Phone: 303-497-8423
Fax: 303-497-8401
On Thu, Jan 9, 2020 at 9:10 AM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> Thu Jan 09 09:10:39 2020: Request 93726 was acted upon.
> Transaction: Ticket created by john.l.wagner at noaa.gov
> Queue: met_help
> Subject: Wind direction verification
> Owner: Nobody
> Requestors: john.l.wagner at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726 >
>
>
> Greetings MET Help
> I was wondering if you had an update of the status of wind direction
> verification? Looking at the slides you have online from 2018, it
seems
> that wind dir verification is under development. The slides also
suggest
> that the verification would use the U and V components instead of a
wind
> direction field. I just want to make sure that this info is still
> accurate. We are currently archiving the URMA wind direction field,
so if
> we need to change our archive, I want to do that ASAP. If there is
other
> info on what will be supported or what version it is expected to be
in that
> I am missing, please let me know.
> Thanks
> John
>
> --
> John Wagner
> Verification Task Lead
> COR Task Manager
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Branch
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>
------------------------------------------------
Subject: Wind direction verification
From: John L Wagner - NOAA Federal
Time: Thu Jan 09 09:38:06 2020
Thanks Minna, much appreciated.
On Thu, Jan 9, 2020 at 11:36 AM Minna Win via RT <met_help at ucar.edu>
wrote:
> Hi John,
>
> It looks like you are interested in doing some wind direction
verification
> in MET. I believe the best person to address this status is the
> engineering lead, John Halley Gotway. I am going to assign this
ticket to
> him. Please allow a few business days for a full response.
>
> Regards,
> Minna
> ---------------
> Minna Win
> National Center for Atmospheric Research
> Developmental Testbed Center
> Phone: 303-497-8423
> Fax: 303-497-8401
>
>
>
> On Thu, Jan 9, 2020 at 9:10 AM John L Wagner - NOAA Federal via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Jan 09 09:10:39 2020: Request 93726 was acted upon.
> > Transaction: Ticket created by john.l.wagner at noaa.gov
> > Queue: met_help
> > Subject: Wind direction verification
> > Owner: Nobody
> > Requestors: john.l.wagner at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726 >
> >
> >
> > Greetings MET Help
> > I was wondering if you had an update of the status of wind
direction
> > verification? Looking at the slides you have online from 2018, it
seems
> > that wind dir verification is under development. The slides also
suggest
> > that the verification would use the U and V components instead of
a wind
> > direction field. I just want to make sure that this info is still
> > accurate. We are currently archiving the URMA wind direction
field, so
> if
> > we need to change our archive, I want to do that ASAP. If there
is other
> > info on what will be supported or what version it is expected to
be in
> that
> > I am missing, please let me know.
> > Thanks
> > John
> >
> > --
> > John Wagner
> > Verification Task Lead
> > COR Task Manager
> > NOAA/National Weather Service
> > Meteorological Development Laboratory
> > Digital Forecast Services Branch
> > SSMC2 Room 10106
> > Silver Spring, MD 20910
> > (301) 427-9471 (office)
> > (908) 902-4155 (cell/text)
> >
> >
>
>
--
John Wagner
Verification Task Lead
COR Task Manager
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Branch
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
------------------------------------------------
Subject: Wind direction verification
From: John Halley Gotway
Time: Thu Jan 09 10:20:31 2020
Hi John,
We are actively working on the MET version 9.0 release. We had
planned to
have it out in December 2019 now the end of January 2020 is a better
estimate. We have made two intermediate test releases named met-
9.0_beta1
and met-9.0_beta2 and those release notes are listed here:
https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-9-0-beta1#notes
https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-9-0-beta2#notes
The remaining development tasks for inclusion in 9.0 are the "blocker"
issues from GitHub:
https://github.com/NCAR/MET/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22MET+9.0%22+label%3A%22priority%3A+blocker%22
However, there are no additional changes included in met-9.0 that
apply to
wind direction.
Hopefully met-8.1 already contains sufficient logic for your needs.
The
Grid-Stat and Point-Stat tools generate VL1L2 and VCNT output line
types.
VL1L2 are vector-based partial sums and VCNT are vector-based
continuous
statistics. The "wind_thresh" configuration option for these tools
enable
you to filter the u/v pairs which are included in the VL1L2 and VCNT
sums
and stats.
Please take a look on page 200 of the met-8.1.2 user's guide:
https://dtcenter.org/sites/default/files/community-code/met/docs/user-
guide/MET_Users_Guide_v8.1.2.pdf
Do these stats looks sufficient? Or can you please remind me of
specific
additional features that are needed?
Thanks,
John Halley Gotway
On Thu, Jan 9, 2020 at 9:38 AM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726 >
>
> Thanks Minna, much appreciated.
>
> On Thu, Jan 9, 2020 at 11:36 AM Minna Win via RT <met_help at ucar.edu>
> wrote:
>
> > Hi John,
> >
> > It looks like you are interested in doing some wind direction
> verification
> > in MET. I believe the best person to address this status is the
> > engineering lead, John Halley Gotway. I am going to assign this
ticket
> to
> > him. Please allow a few business days for a full response.
> >
> > Regards,
> > Minna
> > ---------------
> > Minna Win
> > National Center for Atmospheric Research
> > Developmental Testbed Center
> > Phone: 303-497-8423
> > Fax: 303-497-8401
> >
> >
> >
> > On Thu, Jan 9, 2020 at 9:10 AM John L Wagner - NOAA Federal via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Jan 09 09:10:39 2020: Request 93726 was acted upon.
> > > Transaction: Ticket created by john.l.wagner at noaa.gov
> > > Queue: met_help
> > > Subject: Wind direction verification
> > > Owner: Nobody
> > > Requestors: john.l.wagner at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726
> >
> > >
> > >
> > > Greetings MET Help
> > > I was wondering if you had an update of the status of wind
direction
> > > verification? Looking at the slides you have online from 2018,
it
> seems
> > > that wind dir verification is under development. The slides
also
> suggest
> > > that the verification would use the U and V components instead
of a
> wind
> > > direction field. I just want to make sure that this info is
still
> > > accurate. We are currently archiving the URMA wind direction
field, so
> > if
> > > we need to change our archive, I want to do that ASAP. If there
is
> other
> > > info on what will be supported or what version it is expected to
be in
> > that
> > > I am missing, please let me know.
> > > Thanks
> > > John
> > >
> > > --
> > > John Wagner
> > > Verification Task Lead
> > > COR Task Manager
> > > NOAA/National Weather Service
> > > Meteorological Development Laboratory
> > > Digital Forecast Services Branch
> > > SSMC2 Room 10106
> > > Silver Spring, MD 20910
> > > (301) 427-9471 (office)
> > > (908) 902-4155 (cell/text)
> > >
> > >
> >
> >
>
> --
> John Wagner
> Verification Task Lead
> COR Task Manager
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Branch
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>
------------------------------------------------
Subject: Wind direction verification
From: John L Wagner - NOAA Federal
Time: Thu Jan 09 13:43:51 2020
Thanks John. I did miss the DIR_ERR and DIR_ABSERR on page 200.
Those
will be useful for our MAEs.
I thought I had read that wind_thresh only applied to wind speed and
not
direction. I will take a closer look at that. We verify wind
direction
conditioned on forecast and observed winds exceeding 8 knots. Can I
pass
grid_stat both the wind direction and speed grids?
We are currently computing a Heidke Skill Score for wind direction
based on
thresholds that we define. Our code currently does an adjustment to
ensure
that all of the north winds fall into the same category. Is there
something similar available in MET? Or would we have to preprocess
the
data to ensure that north isn't split between 0-45 degrees and 335-360
degrees?
Thanks! Your help is always appreciated.
John
On Thu, Jan 9, 2020 at 12:20 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Hi John,
>
> We are actively working on the MET version 9.0 release. We had
planned to
> have it out in December 2019 now the end of January 2020 is a better
> estimate. We have made two intermediate test releases named met-
9.0_beta1
> and met-9.0_beta2 and those release notes are listed here:
>
> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-9-0-beta1#notes
>
> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-9-0-beta2#notes
>
> The remaining development tasks for inclusion in 9.0 are the
"blocker"
> issues from GitHub:
>
>
https://github.com/NCAR/MET/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22MET+9.0%22+label%3A%22priority%3A+blocker%22
>
> However, there are no additional changes included in met-9.0 that
apply to
> wind direction.
>
> Hopefully met-8.1 already contains sufficient logic for your needs.
The
> Grid-Stat and Point-Stat tools generate VL1L2 and VCNT output line
types.
> VL1L2 are vector-based partial sums and VCNT are vector-based
continuous
> statistics. The "wind_thresh" configuration option for these tools
enable
> you to filter the u/v pairs which are included in the VL1L2 and VCNT
sums
> and stats.
>
> Please take a look on page 200 of the met-8.1.2 user's guide:
>
> https://dtcenter.org/sites/default/files/community-
code/met/docs/user-guide/MET_Users_Guide_v8.1.2.pdf
>
> Do these stats looks sufficient? Or can you please remind me of
specific
> additional features that are needed?
>
> Thanks,
> John Halley Gotway
>
>
>
> On Thu, Jan 9, 2020 at 9:38 AM John L Wagner - NOAA Federal via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726 >
> >
> > Thanks Minna, much appreciated.
> >
> > On Thu, Jan 9, 2020 at 11:36 AM Minna Win via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi John,
> > >
> > > It looks like you are interested in doing some wind direction
> > verification
> > > in MET. I believe the best person to address this status is the
> > > engineering lead, John Halley Gotway. I am going to assign this
ticket
> > to
> > > him. Please allow a few business days for a full response.
> > >
> > > Regards,
> > > Minna
> > > ---------------
> > > Minna Win
> > > National Center for Atmospheric Research
> > > Developmental Testbed Center
> > > Phone: 303-497-8423
> > > Fax: 303-497-8401
> > >
> > >
> > >
> > > On Thu, Jan 9, 2020 at 9:10 AM John L Wagner - NOAA Federal via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Thu Jan 09 09:10:39 2020: Request 93726 was acted upon.
> > > > Transaction: Ticket created by john.l.wagner at noaa.gov
> > > > Queue: met_help
> > > > Subject: Wind direction verification
> > > > Owner: Nobody
> > > > Requestors: john.l.wagner at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726
> > >
> > > >
> > > >
> > > > Greetings MET Help
> > > > I was wondering if you had an update of the status of wind
direction
> > > > verification? Looking at the slides you have online from
2018, it
> > seems
> > > > that wind dir verification is under development. The slides
also
> > suggest
> > > > that the verification would use the U and V components instead
of a
> > wind
> > > > direction field. I just want to make sure that this info is
still
> > > > accurate. We are currently archiving the URMA wind direction
field,
> so
> > > if
> > > > we need to change our archive, I want to do that ASAP. If
there is
> > other
> > > > info on what will be supported or what version it is expected
to be
> in
> > > that
> > > > I am missing, please let me know.
> > > > Thanks
> > > > John
> > > >
> > > > --
> > > > John Wagner
> > > > Verification Task Lead
> > > > COR Task Manager
> > > > NOAA/National Weather Service
> > > > Meteorological Development Laboratory
> > > > Digital Forecast Services Branch
> > > > SSMC2 Room 10106
> > > > Silver Spring, MD 20910
> > > > (301) 427-9471 (office)
> > > > (908) 902-4155 (cell/text)
> > > >
> > > >
> > >
> > >
> >
> > --
> > John Wagner
> > Verification Task Lead
> > COR Task Manager
> > NOAA/National Weather Service
> > Meteorological Development Laboratory
> > Digital Forecast Services Branch
> > SSMC2 Room 10106
> > Silver Spring, MD 20910
> > (301) 427-9471 (office)
> > (908) 902-4155 (cell/text)
> >
> >
>
>
--
John Wagner
Verification Task Lead
COR Task Manager
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Branch
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
------------------------------------------------
Subject: Wind direction verification
From: John Halley Gotway
Time: Fri Jan 10 09:17:40 2020
John,
Yes, the wind_thresh configuration option does apply to the VCNT
output
line type. The MET tarball runs some sample test scripts when you run
"make test" after compiling the code. I modified the configuration
file
for one of those tests to demonstrate:
*cd met-8.1.2/scripts*
*export TEST_OUT_DIR="../out"*
# Run existing command to make sure it's working OK
*../bin/grid_stat \*
*../data/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212 \*
*../data/sample_fcst/2005080712/wrfprs_ruc13_00.tm00_G212 \*
*config/GridStatConfig_all \*
*-outdir ${TEST_OUT_DIR}/grid_stat -v 2*
# Modify the configuration file (config/GridStatConfig_all) by
setting...
and then rerun the command above:
*//// Forecast and observation fields to be verified//fcst = {
wind_thresh = [ NA, NA, NA ]; field = [ { name = "UGRD"; level
= [
"Z10" ]; },*
* { name = "VGRD"; level = [ "Z10" ]; } ];}obs = fcst;obs = {
wind_thresh = [ NA, >=2, >=5 ]; }*
Instead of getting 1 VCNT output line for each masking area, you'll
now get
3: one using all U/V pairs, one using U/V pairs with speed >=2 m/s,
and one
using U/V pairs with speed >=5 m/s. I configured it so that we're
filtering based on the OBSERVATION wind speed. But you could do the
forecast instead. Or both, using the intersection (i.e. wind_logic =
INTERSECTION;). A threshold of type NA always evaluates to true, so
setting "wind_thresh = [ NA, NA, NA ];" in the fcst dictionary just
means
that we're not filtering the fcst U/V vectors.
This configuration results in the attached VCNT output. You'll see
the
TOTAL column (# of pairs used) on the first 3 lines decreases from
5558 to
4999 to 2301 as we increase the wind speed threshold from none to >=2
to
>=5.
Currently, if you try to verify wind direction directly in Point-Stat
or
Grid-Stat, you'll get this error message:
ERROR : GridStatVxOpt::process_config() -> wind direction may not be
verified using grid_stat.
That check is there because the continuous statistics in the CNT line
type
are not well-suited to circular variables (like wind direction).
However,
you mentioned applying Heidke Skill Score to wind direction, which is
a
categorical statistic. I'd honestly never thought about verifying
wind
direction in a categorical way. That certainly seems doable... as
long as
care is taken in defining the categorical thresholds well.
If you agree, I can write up a development issue for the verification
of
wind direction using categorical statistics. We'll need to make
Point-Stat
and Grid-Stat smart enough to *not* compute continuous stats for them.
Do
you have a sense for whether or not categorical vx of wind direction
is
widely done? Is this a common practice that should be added to a
community
vx package?
Thanks,
John
On Thu, Jan 9, 2020 at 1:44 PM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726 >
>
> Thanks John. I did miss the DIR_ERR and DIR_ABSERR on page 200.
Those
> will be useful for our MAEs.
> I thought I had read that wind_thresh only applied to wind speed and
not
> direction. I will take a closer look at that. We verify wind
direction
> conditioned on forecast and observed winds exceeding 8 knots. Can I
pass
> grid_stat both the wind direction and speed grids?
> We are currently computing a Heidke Skill Score for wind direction
based on
> thresholds that we define. Our code currently does an adjustment to
ensure
> that all of the north winds fall into the same category. Is there
> something similar available in MET? Or would we have to preprocess
the
> data to ensure that north isn't split between 0-45 degrees and 335-
360
> degrees?
> Thanks! Your help is always appreciated.
> John
>
> On Thu, Jan 9, 2020 at 12:20 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Hi John,
> >
> > We are actively working on the MET version 9.0 release. We had
planned
> to
> > have it out in December 2019 now the end of January 2020 is a
better
> > estimate. We have made two intermediate test releases named
> met-9.0_beta1
> > and met-9.0_beta2 and those release notes are listed here:
> >
> >
> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-9-0-beta1#notes
> >
> >
> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-9-0-beta2#notes
> >
> > The remaining development tasks for inclusion in 9.0 are the
"blocker"
> > issues from GitHub:
> >
> >
>
https://github.com/NCAR/MET/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22MET+9.0%22+label%3A%22priority%3A+blocker%22
> >
> > However, there are no additional changes included in met-9.0 that
apply
> to
> > wind direction.
> >
> > Hopefully met-8.1 already contains sufficient logic for your
needs. The
> > Grid-Stat and Point-Stat tools generate VL1L2 and VCNT output line
types.
> > VL1L2 are vector-based partial sums and VCNT are vector-based
continuous
> > statistics. The "wind_thresh" configuration option for these
tools
> enable
> > you to filter the u/v pairs which are included in the VL1L2 and
VCNT sums
> > and stats.
> >
> > Please take a look on page 200 of the met-8.1.2 user's guide:
> >
> >
> https://dtcenter.org/sites/default/files/community-
code/met/docs/user-guide/MET_Users_Guide_v8.1.2.pdf
> >
> > Do these stats looks sufficient? Or can you please remind me of
specific
> > additional features that are needed?
> >
> > Thanks,
> > John Halley Gotway
> >
> >
> >
> > On Thu, Jan 9, 2020 at 9:38 AM John L Wagner - NOAA Federal via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726 >
> > >
> > > Thanks Minna, much appreciated.
> > >
> > > On Thu, Jan 9, 2020 at 11:36 AM Minna Win via RT
<met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi John,
> > > >
> > > > It looks like you are interested in doing some wind direction
> > > verification
> > > > in MET. I believe the best person to address this status is
the
> > > > engineering lead, John Halley Gotway. I am going to assign
this
> ticket
> > > to
> > > > him. Please allow a few business days for a full response.
> > > >
> > > > Regards,
> > > > Minna
> > > > ---------------
> > > > Minna Win
> > > > National Center for Atmospheric Research
> > > > Developmental Testbed Center
> > > > Phone: 303-497-8423
> > > > Fax: 303-497-8401
> > > >
> > > >
> > > >
> > > > On Thu, Jan 9, 2020 at 9:10 AM John L Wagner - NOAA Federal
via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Thu Jan 09 09:10:39 2020: Request 93726 was acted upon.
> > > > > Transaction: Ticket created by john.l.wagner at noaa.gov
> > > > > Queue: met_help
> > > > > Subject: Wind direction verification
> > > > > Owner: Nobody
> > > > > Requestors: john.l.wagner at noaa.gov
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726
> > > >
> > > > >
> > > > >
> > > > > Greetings MET Help
> > > > > I was wondering if you had an update of the status of wind
> direction
> > > > > verification? Looking at the slides you have online from
2018, it
> > > seems
> > > > > that wind dir verification is under development. The slides
also
> > > suggest
> > > > > that the verification would use the U and V components
instead of a
> > > wind
> > > > > direction field. I just want to make sure that this info is
still
> > > > > accurate. We are currently archiving the URMA wind
direction
> field,
> > so
> > > > if
> > > > > we need to change our archive, I want to do that ASAP. If
there is
> > > other
> > > > > info on what will be supported or what version it is
expected to be
> > in
> > > > that
> > > > > I am missing, please let me know.
> > > > > Thanks
> > > > > John
> > > > >
> > > > > --
> > > > > John Wagner
> > > > > Verification Task Lead
> > > > > COR Task Manager
> > > > > NOAA/National Weather Service
> > > > > Meteorological Development Laboratory
> > > > > Digital Forecast Services Branch
> > > > > SSMC2 Room 10106
> > > > > Silver Spring, MD 20910
> > > > > (301) 427-9471 (office)
> > > > > (908) 902-4155 (cell/text)
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > John Wagner
> > > Verification Task Lead
> > > COR Task Manager
> > > NOAA/National Weather Service
> > > Meteorological Development Laboratory
> > > Digital Forecast Services Branch
> > > SSMC2 Room 10106
> > > Silver Spring, MD 20910
> > > (301) 427-9471 (office)
> > > (908) 902-4155 (cell/text)
> > >
> > >
> >
> >
>
> --
> John Wagner
> Verification Task Lead
> COR Task Manager
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Branch
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>
------------------------------------------------
Subject: Wind direction verification
From: John Halley Gotway
Time: Fri Jan 10 09:17:40 2020
VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD
OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_UNITS FCST_LEV OBS_VAR
OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH
OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL FBAR FBAR_BCL FBAR_BCU
OBAR OBAR_BCL OBAR_BCU FS_RMS FS_RMS_BCL FS_RMS_BCU OS_RMS
OS_RMS_BCL OS_RMS_BCU MSVE MSVE_BCL MSVE_BCU RMSVE RMSVE_BCL
RMSVE_BCU FSTDEV FSTDEV_BCL FSTDEV_BCU OSTDEV OSTDEV_BCL OSTDEV_BCU
FDIR FDIR_BCL FDIR_BCU ODIR ODIR_BCL ODIR_BCU FBAR_SPEED
FBAR_SPEED_BCL FBAR_SPEED_BCU OBAR_SPEED OBAR_SPEED_BCL OBAR_SPEED_BCU
VDIFF_SPEED VDIFF_SPEED_BCL VDIFF_SPEED_BCU VDIFF_DIR VDIFF_DIR_BCL
VDIFF_DIR_BCU SPEED_ERR SPEED_ERR_BCL SPEED_ERR_BCU SPEED_ABSERR
SPEED_ABSERR_BCL SPEED_ABSERR_BCU DIR_ERR DIR_ERR_BCL DIR_ERR_BCU
DIR_ABSERR DIR_ABSERR_BCL DIR_ABSERR_BCU
V8.1.2 WRF NA 120000 20050807_120000 20050807_120000 000000
20050807_120000 20050807_120000 UGRD_VGRD m/s Z10
UGRD_VGRD m/s Z10 ANALYS DTC165 NEAREST 1 NA
NA NA NA VCNT 5558 3.74987 NA NA
4.54808 NA NA 4.41425 NA NA 5.20931
NA NA 8.72616 NA NA 2.95401 NA NA
2.32896 NA NA 2.54005 NA NA 321.53259
NA NA 318.29533 NA NA 1.5864 NA
NA 1.47852 NA NA 0.13829
NA NA 358.67129 NA NA 0.10788
NA NA 0.10788 NA NA
-3.23726 NA NA 3.23726 NA
NA
V8.1.2 WRF NA 120000 20050807_120000 20050807_120000 000000
20050807_120000 20050807_120000 UGRD_VGRD m/s Z10
UGRD_VGRD m/s Z10 ANALYS DTC165 NEAREST 1 NA
>=2 NA NA VCNT 4999 3.94128 NA NA
4.9368 NA NA 4.59178 NA NA 5.47922
NA NA 9.14583 NA NA 3.02421 NA NA
2.356 NA NA 2.37694 NA NA 321.2039
NA NA 318.3645 NA NA 1.76777 NA
NA 1.65765 NA NA 0.139
NA NA 357.41527 NA NA 0.11011
NA NA 0.11011 NA NA
-2.8394 NA NA 2.8394 NA
NA
V8.1.2 WRF NA 120000 20050807_120000 20050807_120000 000000
20050807_120000 20050807_120000 UGRD_VGRD m/s Z10
UGRD_VGRD m/s Z10 ANALYS DTC165 NEAREST 1 NA
>=5 NA NA VCNT 2301 5.12665 NA NA
6.8945 NA NA 5.75431 NA NA 7.20507
NA NA 11.711 NA NA 3.42213 NA NA
2.61334 NA NA 2.09261 NA NA 324.25564
NA NA 323.04946 NA NA 2.91342 NA
NA 3.16417 NA NA 0.25877
NA NA 129.33979 NA NA -0.25075
NA NA 0.25075 NA NA
-1.20619 NA NA 1.20619 NA
NA
V8.1.2 WRF NA 120000 20050807_120000 20050807_120000 000000
20050807_120000 20050807_120000 UGRD_VGRD m/s Z10
UGRD_VGRD m/s Z10 ANALYS DTC166 NEAREST 1 NA
NA NA NA VCNT 6862 3.38203 NA NA
3.39417 NA NA 4.08693 NA NA 4.12265
NA NA 5.25182 NA NA 2.29169 NA NA
2.29453 NA NA 2.34005 NA NA 198.85917
NA NA 197.91096 NA NA 1.11762 NA
NA 1.01922 NA NA 0.099969
NA NA 208.5725 NA NA 0.098396
NA NA 0.098396 NA NA
-0.94821 NA NA 0.94821 NA
NA
V8.1.2 WRF NA 120000 20050807_120000 20050807_120000 000000
20050807_120000 20050807_120000 UGRD_VGRD m/s Z10
UGRD_VGRD m/s Z10 ANALYS DTC166 NEAREST 1 NA
>=2 NA NA VCNT 5064 3.95965 NA NA
4.25088 NA NA 4.59894 NA NA 4.75464
NA NA 5.97137 NA NA 2.44364 NA NA
2.33911 NA NA 2.12994 NA NA 203.43175
NA NA 200.77084 NA NA 1.45029 NA
NA 1.36308 NA NA 0.10894
NA NA 238.94247 NA NA 0.087212
NA NA 0.087212 NA NA
-2.66091 NA NA 2.66091 NA
NA
V8.1.2 WRF NA 120000 20050807_120000 20050807_120000 000000
20050807_120000 20050807_120000 UGRD_VGRD m/s Z10
UGRD_VGRD m/s Z10 ANALYS DTC166 NEAREST 1 NA
>=5 NA NA VCNT 1707 6.22249 NA NA
6.63735 NA NA 6.57925 NA NA 6.89914
NA NA 7.17047 NA NA 2.67777 NA NA
2.13707 NA NA 1.88249 NA NA 219.19075
NA NA 218.45588 NA NA 2.98734 NA
NA 2.98765 NA NA 0.038318
NA NA 309.289 NA NA -0.00031144
NA NA 0.00031144 NA NA
-0.73487 NA NA 0.73487 NA
NA
V8.1.2 WRF NA 120000 20050807_120000 20050807_120000 000000
20050807_120000 20050807_120000 UGRD_VGRD m/s Z10
UGRD_VGRD m/s Z10 ANALYS CONUS NEAREST 1 NA
NA NA NA VCNT 5479 2.42081 NA NA
2.94389 NA NA 2.79816 NA NA 3.59153
NA NA 6.64081 NA NA 2.57698 NA NA
1.40336 NA NA 2.05732 NA NA 191.7345
NA NA 195.62712 NA NA 0.51859 NA
NA 0.74411 NA NA 0.22944
NA NA 24.45345 NA NA -0.22553
NA NA 0.22553 NA NA
3.89263 NA NA 3.89263 NA
NA
V8.1.2 WRF NA 120000 20050807_120000 20050807_120000 000000
20050807_120000 20050807_120000 UGRD_VGRD m/s Z10
UGRD_VGRD m/s Z10 ANALYS CONUS NEAREST 1 NA
>=2 NA NA VCNT 3738 2.79318 NA NA
3.86247 NA NA 3.14667 NA NA 4.28418
NA NA 8.42055 NA NA 2.90182 NA NA
1.44904 NA NA 1.85353 NA NA 200.69646
NA NA 199.90877 NA NA 0.73851 NA
NA 1.08253 NA NA 0.34424
NA NA 18.21877 NA NA -0.34402
NA NA 0.34402 NA NA
-0.78768 NA NA 0.78768 NA
NA
V8.1.2 WRF NA 120000 20050807_120000 20050807_120000 000000
20050807_120000 20050807_120000 UGRD_VGRD m/s Z10
UGRD_VGRD m/s Z10 ANALYS CONUS NEAREST 1 NA
>=5 NA NA VCNT 973 3.81144 NA NA
6.41176 NA NA 4.22176 NA NA 6.60771
NA NA 16.73279 NA NA 4.09057 NA NA
1.81553 NA NA 1.59725 NA NA 232.80359
NA NA 225.59105 NA NA 1.53664 NA
NA 2.4398 NA NA 0.93543
NA NA 33.68885 NA NA -0.90316
NA NA 0.90316 NA NA
-7.21254 NA NA 7.21254 NA
NA
V8.1.2 WRF NA 120000 20050807_120000 20050807_120000 000000
20050807_120000 20050807_120000 UGRD_VGRD m/s Z10
UGRD_VGRD m/s Z10 ANALYS LMV NEAREST 1 NA
NA NA NA VCNT 524 1.77358 NA NA
1.55639 NA NA 1.89213 NA NA 1.7337
NA NA 2.65076 NA NA 1.62812 NA NA
0.65923 NA NA 0.76378 NA NA 86.13029
NA NA 96.1869 NA NA 0.79086 NA
NA 0.75213 NA NA 0.14063
NA NA 17.07817 NA NA 0.038723
NA NA 0.038723 NA NA
10.05661 NA NA 10.05661 NA
NA
V8.1.2 WRF NA 120000 20050807_120000 20050807_120000 000000
20050807_120000 20050807_120000 UGRD_VGRD m/s Z10
UGRD_VGRD m/s Z10 ANALYS LMV NEAREST 1 NA
>=2 NA NA VCNT 204 1.83001 NA NA
2.3393 NA NA 1.94407 NA NA 2.38151
NA NA 2.98039 NA NA 1.72638 NA NA
0.65611 NA NA 0.44636 NA NA 90.98028
NA NA 97.35791 NA NA 1.02207 NA
NA 1.32491 NA NA 0.32935
NA NA 297.52227 NA NA -0.30284
NA NA 0.30284 NA NA
6.37763 NA NA 6.37763 NA
NA
------------------------------------------------
Subject: Wind direction verification
From: John Halley Gotway
Time: Fri Jan 10 09:32:57 2020
John,
I did think of one additional complication that'd make the categorical
vx
of wind direction more complicated.
Presumably, just like with the VCNT line type, you'd want to filter
the
pairs used based on the wind speed. I'd assume that the categorical
stats
for stronger winds would be much better than those for light and
variable
winds. But you'd also need to define the thresholds for the
categories,
such as northerly winds being between 45 and 135 degrees, for example.
So we might configure it like this:
*fcst = { wind_thresh = [ NA, >=2, >=5 ]; field = [ { name =
"WDIR";*
* level = [ "Z10" ];*
* cat_thresh = [ >=45&&<=135 ]; } ];}*
We are applying 2 threshold steps here... (1) wind_thresh to filter
the U/V
pairs and (2) cat_thresh to define the categories.
But the output stat header only has column for FCST_THRESH. For
categorical output (CTC, CTS, MCTC, and MCTS lines types), we write
the
cat_thresh to that column. For vector stats (VL1L2, VAL1L2, and VCNT
line
types), we write the wind_thresh to that column.
In this case, we'd have 2 thresholds to include in the metadata and
only 1
column to contain them. Perhaps we could just concatenate them with a
comma, but that'll make the logic for reading them back into STAT-
Analysis
a bit more tricky.
Just some more details to consider.
Thanks,
John
On Fri, Jan 10, 2020 at 9:17 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> John,
>
> Yes, the wind_thresh configuration option does apply to the VCNT
output
> line type. The MET tarball runs some sample test scripts when you
run
> "make test" after compiling the code. I modified the configuration
file
> for one of those tests to demonstrate:
>
> *cd met-8.1.2/scripts*
> *export TEST_OUT_DIR="../out"*
> # Run existing command to make sure it's working OK
> *../bin/grid_stat \*
> *../data/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212 \*
> *../data/sample_fcst/2005080712/wrfprs_ruc13_00.tm00_G212 \*
> *config/GridStatConfig_all \*
> *-outdir ${TEST_OUT_DIR}/grid_stat -v 2*
>
> # Modify the configuration file (config/GridStatConfig_all) by
setting...
> and then rerun the command above:
>
>
>
>
>
>
>
> *//// Forecast and observation fields to be verified//fcst = {
> wind_thresh = [ NA, NA, NA ]; field = [ { name = "UGRD";
level = [
> "Z10" ]; },*
>
>
>
>
> * { name = "VGRD"; level = [ "Z10" ]; } ];}obs = fcst;obs =
{
> wind_thresh = [ NA, >=2, >=5 ]; }*
>
> Instead of getting 1 VCNT output line for each masking area, you'll
now
> get 3: one using all U/V pairs, one using U/V pairs with speed >=2
m/s, and
> one using U/V pairs with speed >=5 m/s. I configured it so that
we're
> filtering based on the OBSERVATION wind speed. But you could do the
> forecast instead. Or both, using the intersection (i.e. wind_logic
=
> INTERSECTION;). A threshold of type NA always evaluates to true, so
> setting "wind_thresh = [ NA, NA, NA ];" in the fcst dictionary just
means
> that we're not filtering the fcst U/V vectors.
>
> This configuration results in the attached VCNT output. You'll see
the
> TOTAL column (# of pairs used) on the first 3 lines decreases from
5558 to
> 4999 to 2301 as we increase the wind speed threshold from none to
>=2 to
> >=5.
>
> Currently, if you try to verify wind direction directly in Point-
Stat or
> Grid-Stat, you'll get this error message:
> ERROR : GridStatVxOpt::process_config() -> wind direction may not
be
> verified using grid_stat.
>
> That check is there because the continuous statistics in the CNT
line type
> are not well-suited to circular variables (like wind direction).
However,
> you mentioned applying Heidke Skill Score to wind direction, which
is a
> categorical statistic. I'd honestly never thought about verifying
wind
> direction in a categorical way. That certainly seems doable... as
long as
> care is taken in defining the categorical thresholds well.
>
> If you agree, I can write up a development issue for the
verification of
> wind direction using categorical statistics. We'll need to make
Point-Stat
> and Grid-Stat smart enough to *not* compute continuous stats for
them. Do
> you have a sense for whether or not categorical vx of wind direction
is
> widely done? Is this a common practice that should be added to a
community
> vx package?
>
> Thanks,
> John
>
> On Thu, Jan 9, 2020 at 1:44 PM John L Wagner - NOAA Federal via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726 >
>>
>> Thanks John. I did miss the DIR_ERR and DIR_ABSERR on page 200.
Those
>> will be useful for our MAEs.
>> I thought I had read that wind_thresh only applied to wind speed
and not
>> direction. I will take a closer look at that. We verify wind
direction
>> conditioned on forecast and observed winds exceeding 8 knots. Can
I pass
>> grid_stat both the wind direction and speed grids?
>> We are currently computing a Heidke Skill Score for wind direction
based
>> on
>> thresholds that we define. Our code currently does an adjustment
to
>> ensure
>> that all of the north winds fall into the same category. Is there
>> something similar available in MET? Or would we have to preprocess
the
>> data to ensure that north isn't split between 0-45 degrees and 335-
360
>> degrees?
>> Thanks! Your help is always appreciated.
>> John
>>
>> On Thu, Jan 9, 2020 at 12:20 PM John Halley Gotway via RT <
>> met_help at ucar.edu>
>> wrote:
>>
>> > Hi John,
>> >
>> > We are actively working on the MET version 9.0 release. We had
planned
>> to
>> > have it out in December 2019 now the end of January 2020 is a
better
>> > estimate. We have made two intermediate test releases named
>> met-9.0_beta1
>> > and met-9.0_beta2 and those release notes are listed here:
>> >
>> >
>> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-9-0-beta1#notes
>> >
>> >
>> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-9-0-beta2#notes
>> >
>> > The remaining development tasks for inclusion in 9.0 are the
"blocker"
>> > issues from GitHub:
>> >
>> >
>>
https://github.com/NCAR/MET/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22MET+9.0%22+label%3A%22priority%3A+blocker%22
>> >
>> > However, there are no additional changes included in met-9.0 that
apply
>> to
>> > wind direction.
>> >
>> > Hopefully met-8.1 already contains sufficient logic for your
needs. The
>> > Grid-Stat and Point-Stat tools generate VL1L2 and VCNT output
line
>> types.
>> > VL1L2 are vector-based partial sums and VCNT are vector-based
continuous
>> > statistics. The "wind_thresh" configuration option for these
tools
>> enable
>> > you to filter the u/v pairs which are included in the VL1L2 and
VCNT
>> sums
>> > and stats.
>> >
>> > Please take a look on page 200 of the met-8.1.2 user's guide:
>> >
>> >
>> https://dtcenter.org/sites/default/files/community-
code/met/docs/user-guide/MET_Users_Guide_v8.1.2.pdf
>> >
>> > Do these stats looks sufficient? Or can you please remind me of
>> specific
>> > additional features that are needed?
>> >
>> > Thanks,
>> > John Halley Gotway
>> >
>> >
>> >
>> > On Thu, Jan 9, 2020 at 9:38 AM John L Wagner - NOAA Federal via
RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726 >
>> > >
>> > > Thanks Minna, much appreciated.
>> > >
>> > > On Thu, Jan 9, 2020 at 11:36 AM Minna Win via RT
<met_help at ucar.edu>
>> > > wrote:
>> > >
>> > > > Hi John,
>> > > >
>> > > > It looks like you are interested in doing some wind direction
>> > > verification
>> > > > in MET. I believe the best person to address this status is
the
>> > > > engineering lead, John Halley Gotway. I am going to assign
this
>> ticket
>> > > to
>> > > > him. Please allow a few business days for a full response.
>> > > >
>> > > > Regards,
>> > > > Minna
>> > > > ---------------
>> > > > Minna Win
>> > > > National Center for Atmospheric Research
>> > > > Developmental Testbed Center
>> > > > Phone: 303-497-8423
>> > > > Fax: 303-497-8401
>> > > >
>> > > >
>> > > >
>> > > > On Thu, Jan 9, 2020 at 9:10 AM John L Wagner - NOAA Federal
via RT <
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > > >
>> > > > > Thu Jan 09 09:10:39 2020: Request 93726 was acted upon.
>> > > > > Transaction: Ticket created by john.l.wagner at noaa.gov
>> > > > > Queue: met_help
>> > > > > Subject: Wind direction verification
>> > > > > Owner: Nobody
>> > > > > Requestors: john.l.wagner at noaa.gov
>> > > > > Status: new
>> > > > > Ticket <URL:
>> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726
>> > > >
>> > > > >
>> > > > >
>> > > > > Greetings MET Help
>> > > > > I was wondering if you had an update of the status of wind
>> direction
>> > > > > verification? Looking at the slides you have online from
2018, it
>> > > seems
>> > > > > that wind dir verification is under development. The
slides also
>> > > suggest
>> > > > > that the verification would use the U and V components
instead of
>> a
>> > > wind
>> > > > > direction field. I just want to make sure that this info
is still
>> > > > > accurate. We are currently archiving the URMA wind
direction
>> field,
>> > so
>> > > > if
>> > > > > we need to change our archive, I want to do that ASAP. If
there
>> is
>> > > other
>> > > > > info on what will be supported or what version it is
expected to
>> be
>> > in
>> > > > that
>> > > > > I am missing, please let me know.
>> > > > > Thanks
>> > > > > John
>> > > > >
>> > > > > --
>> > > > > John Wagner
>> > > > > Verification Task Lead
>> > > > > COR Task Manager
>> > > > > NOAA/National Weather Service
>> > > > > Meteorological Development Laboratory
>> > > > > Digital Forecast Services Branch
>> > > > > SSMC2 Room 10106
>> > > > > Silver Spring, MD 20910
>> > > > > (301) 427-9471 (office)
>> > > > > (908) 902-4155 (cell/text)
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> > > --
>> > > John Wagner
>> > > Verification Task Lead
>> > > COR Task Manager
>> > > NOAA/National Weather Service
>> > > Meteorological Development Laboratory
>> > > Digital Forecast Services Branch
>> > > SSMC2 Room 10106
>> > > Silver Spring, MD 20910
>> > > (301) 427-9471 (office)
>> > > (908) 902-4155 (cell/text)
>> > >
>> > >
>> >
>> >
>>
>> --
>> John Wagner
>> Verification Task Lead
>> COR Task Manager
>> NOAA/National Weather Service
>> Meteorological Development Laboratory
>> Digital Forecast Services Branch
>> SSMC2 Room 10106
>> Silver Spring, MD 20910
>> (301) 427-9471 (office)
>> (908) 902-4155 (cell/text)
>>
>>
------------------------------------------------
Subject: Wind direction verification
From: John L Wagner - NOAA Federal
Time: Thu Jan 16 11:26:58 2020
Hi John
Sorry for the delay in my response. Brooke has been running some
tests
with wind direction for grid_stat and I have been digging into how we
compute our scores in our MOS-2000 software.
To answer your questions from your last two emails, we would need a
way to
define the categories for wind direction in order to compute the HSS.
The
categories that we're using currently are 25, 70, 115, 160, 205, 250,
295,
and 999 (or 345). These numbers at first made no sense to me until we
looked at the data and realized that our station-based forecasts and
obs
were all rounded to ten degrees. This made 4 or 5 possibilities for
each
wind direction category:
North: -15 to 25
Obs: -10, 0, 10, 20
Northeast: 25 to 70
Obs: 30, 40, 50, 60, 70
East: 70 to 115
Obs: 80, 90, 100, 110
Southeast: 115 to 160
Obs: 120, 130, 140, 150, 160
South: 155 to 205
Obs: 170, 180, 190, 200
Southwest: 205 to 250
Obs: 210, 220, 230, 240, 250
West: 250 to 295
Obs: 260, 270, 280, 290
Northwest: 295 to 999
Obs: 300, 310, 320, 330, 340
We do subtract 360 degrees from any forecast or ob greater than 345
degrees
to ensure that all of the north winds are in the same category. I'm
not
sure yet if these categories make sense yet for gridded wind
direction,
where the numbers likely will not be rounded for us.
I'm not sure how many other centers compute the HSS for wind
direction. I
know that MDL has been calculating it for at least the last 12-15
years.
It does seem like a useful score to me.
Brooke was able to trick grid_stat into computing a HSS by changing
the
variable name from WDir to WD in the netcdf files. This trick didn't
work
for point_stat though which is where we need to compute this (for
reference, we currently compute MAE, HSS, relative frequencies, and
convergence scores for station-based
<https://veritas.nws.noaa.gov/ndfd-
stats/verification/index.php?mo_id=201910&ty_id=pt&rr_id=ptreray&re_id=co&wx_id=wd&pr_id=&md_id=nd&cy_id=00&pd_id=pl>
forecasts and MAE for gridded
<https://veritas.nws.noaa.gov/ndfd-
stats/wfosummary/gridded/actionform.php?station1=US&station2=SAME&station3=SAME&station4=SAME&element=cl&rng=None&beginMonth=201910&data_type1=NDFD&data_type2=GMOS&data_type3=WPC&data_type4=BLEND&cycle=0&proj=INC&endMonth=201910&wfo=US&submit=Plot+Graph&comp_type=URMA>
forecasts).
In order to proceed, it looks like we will need a way to convert our
wind
speeds and directions into the U and V components. Is there a MET
utility
that can handle this? Or do you know of any other utilities that
exist
that will before we create one? I know how to do this in MOS-2000,
but
that won't help us with GRIB2 or netCDF files. CAMPS will eventually
help
us, but the CAMPS code that would do this is not ready yet. WGRIB2
looks
like it will convert from U and V to speed and direction, but not the
opposite.
For what its worth, for the majority of what we verify, the U and V
components will not be available to us. Only the speed and direction.
Thanks
John
On Fri, Jan 10, 2020 at 11:32 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> John,
>
> I did think of one additional complication that'd make the
categorical vx
> of wind direction more complicated.
>
> Presumably, just like with the VCNT line type, you'd want to filter
the
> pairs used based on the wind speed. I'd assume that the categorical
stats
> for stronger winds would be much better than those for light and
variable
> winds. But you'd also need to define the thresholds for the
categories,
> such as northerly winds being between 45 and 135 degrees, for
example.
>
> So we might configure it like this:
>
>
>
>
> *fcst = { wind_thresh = [ NA, >=2, >=5 ]; field = [ { name
=
> "WDIR";*
> * level = [ "Z10" ];*
>
>
> * cat_thresh = [ >=45&&<=135 ]; } ];}*
>
> We are applying 2 threshold steps here... (1) wind_thresh to filter
the U/V
> pairs and (2) cat_thresh to define the categories.
>
> But the output stat header only has column for FCST_THRESH. For
> categorical output (CTC, CTS, MCTC, and MCTS lines types), we write
the
> cat_thresh to that column. For vector stats (VL1L2, VAL1L2, and
VCNT line
> types), we write the wind_thresh to that column.
>
> In this case, we'd have 2 thresholds to include in the metadata and
only 1
> column to contain them. Perhaps we could just concatenate them with
a
> comma, but that'll make the logic for reading them back into STAT-
Analysis
> a bit more tricky.
>
> Just some more details to consider.
>
> Thanks,
> John
>
>
>
>
>
> On Fri, Jan 10, 2020 at 9:17 AM John Halley Gotway <johnhg at ucar.edu>
> wrote:
>
> > John,
> >
> > Yes, the wind_thresh configuration option does apply to the VCNT
output
> > line type. The MET tarball runs some sample test scripts when you
run
> > "make test" after compiling the code. I modified the
configuration file
> > for one of those tests to demonstrate:
> >
> > *cd met-8.1.2/scripts*
> > *export TEST_OUT_DIR="../out"*
> > # Run existing command to make sure it's working OK
> > *../bin/grid_stat \*
> > *../data/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212 \*
> > *../data/sample_fcst/2005080712/wrfprs_ruc13_00.tm00_G212 \*
> > *config/GridStatConfig_all \*
> > *-outdir ${TEST_OUT_DIR}/grid_stat -v 2*
> >
> > # Modify the configuration file (config/GridStatConfig_all) by
> setting...
> > and then rerun the command above:
> >
> >
> >
> >
> >
> >
> >
> > *//// Forecast and observation fields to be verified//fcst = {
> > wind_thresh = [ NA, NA, NA ]; field = [ { name = "UGRD";
level =
> [
> > "Z10" ]; },*
> >
> >
> >
> >
> > * { name = "VGRD"; level = [ "Z10" ]; } ];}obs = fcst;obs
= {
> > wind_thresh = [ NA, >=2, >=5 ]; }*
> >
> > Instead of getting 1 VCNT output line for each masking area,
you'll now
> > get 3: one using all U/V pairs, one using U/V pairs with speed >=2
m/s,
> and
> > one using U/V pairs with speed >=5 m/s. I configured it so that
we're
> > filtering based on the OBSERVATION wind speed. But you could do
the
> > forecast instead. Or both, using the intersection (i.e.
wind_logic =
> > INTERSECTION;). A threshold of type NA always evaluates to true,
so
> > setting "wind_thresh = [ NA, NA, NA ];" in the fcst dictionary
just means
> > that we're not filtering the fcst U/V vectors.
> >
> > This configuration results in the attached VCNT output. You'll
see the
> > TOTAL column (# of pairs used) on the first 3 lines decreases from
5558
> to
> > 4999 to 2301 as we increase the wind speed threshold from none to
>=2 to
> > >=5.
> >
> > Currently, if you try to verify wind direction directly in Point-
Stat or
> > Grid-Stat, you'll get this error message:
> > ERROR : GridStatVxOpt::process_config() -> wind direction may not
be
> > verified using grid_stat.
> >
> > That check is there because the continuous statistics in the CNT
line
> type
> > are not well-suited to circular variables (like wind direction).
> However,
> > you mentioned applying Heidke Skill Score to wind direction, which
is a
> > categorical statistic. I'd honestly never thought about verifying
wind
> > direction in a categorical way. That certainly seems doable... as
long
> as
> > care is taken in defining the categorical thresholds well.
> >
> > If you agree, I can write up a development issue for the
verification of
> > wind direction using categorical statistics. We'll need to make
> Point-Stat
> > and Grid-Stat smart enough to *not* compute continuous stats for
them.
> Do
> > you have a sense for whether or not categorical vx of wind
direction is
> > widely done? Is this a common practice that should be added to a
> community
> > vx package?
> >
> > Thanks,
> > John
> >
> > On Thu, Jan 9, 2020 at 1:44 PM John L Wagner - NOAA Federal via RT
<
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726 >
> >>
> >> Thanks John. I did miss the DIR_ERR and DIR_ABSERR on page 200.
Those
> >> will be useful for our MAEs.
> >> I thought I had read that wind_thresh only applied to wind speed
and not
> >> direction. I will take a closer look at that. We verify wind
direction
> >> conditioned on forecast and observed winds exceeding 8 knots.
Can I
> pass
> >> grid_stat both the wind direction and speed grids?
> >> We are currently computing a Heidke Skill Score for wind
direction based
> >> on
> >> thresholds that we define. Our code currently does an adjustment
to
> >> ensure
> >> that all of the north winds fall into the same category. Is
there
> >> something similar available in MET? Or would we have to
preprocess the
> >> data to ensure that north isn't split between 0-45 degrees and
335-360
> >> degrees?
> >> Thanks! Your help is always appreciated.
> >> John
> >>
> >> On Thu, Jan 9, 2020 at 12:20 PM John Halley Gotway via RT <
> >> met_help at ucar.edu>
> >> wrote:
> >>
> >> > Hi John,
> >> >
> >> > We are actively working on the MET version 9.0 release. We had
> planned
> >> to
> >> > have it out in December 2019 now the end of January 2020 is a
better
> >> > estimate. We have made two intermediate test releases named
> >> met-9.0_beta1
> >> > and met-9.0_beta2 and those release notes are listed here:
> >> >
> >> >
> >>
> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-9-0-beta1#notes
> >> >
> >> >
> >>
> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-9-0-beta2#notes
> >> >
> >> > The remaining development tasks for inclusion in 9.0 are the
"blocker"
> >> > issues from GitHub:
> >> >
> >> >
> >>
>
https://github.com/NCAR/MET/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22MET+9.0%22+label%3A%22priority%3A+blocker%22
> >> >
> >> > However, there are no additional changes included in met-9.0
that
> apply
> >> to
> >> > wind direction.
> >> >
> >> > Hopefully met-8.1 already contains sufficient logic for your
needs.
> The
> >> > Grid-Stat and Point-Stat tools generate VL1L2 and VCNT output
line
> >> types.
> >> > VL1L2 are vector-based partial sums and VCNT are vector-based
> continuous
> >> > statistics. The "wind_thresh" configuration option for these
tools
> >> enable
> >> > you to filter the u/v pairs which are included in the VL1L2 and
VCNT
> >> sums
> >> > and stats.
> >> >
> >> > Please take a look on page 200 of the met-8.1.2 user's guide:
> >> >
> >> >
> >>
> https://dtcenter.org/sites/default/files/community-
code/met/docs/user-guide/MET_Users_Guide_v8.1.2.pdf
> >> >
> >> > Do these stats looks sufficient? Or can you please remind me
of
> >> specific
> >> > additional features that are needed?
> >> >
> >> > Thanks,
> >> > John Halley Gotway
> >> >
> >> >
> >> >
> >> > On Thu, Jan 9, 2020 at 9:38 AM John L Wagner - NOAA Federal via
RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726
>
> >> > >
> >> > > Thanks Minna, much appreciated.
> >> > >
> >> > > On Thu, Jan 9, 2020 at 11:36 AM Minna Win via RT
<met_help at ucar.edu
> >
> >> > > wrote:
> >> > >
> >> > > > Hi John,
> >> > > >
> >> > > > It looks like you are interested in doing some wind
direction
> >> > > verification
> >> > > > in MET. I believe the best person to address this status
is the
> >> > > > engineering lead, John Halley Gotway. I am going to assign
this
> >> ticket
> >> > > to
> >> > > > him. Please allow a few business days for a full response.
> >> > > >
> >> > > > Regards,
> >> > > > Minna
> >> > > > ---------------
> >> > > > Minna Win
> >> > > > National Center for Atmospheric Research
> >> > > > Developmental Testbed Center
> >> > > > Phone: 303-497-8423
> >> > > > Fax: 303-497-8401
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Thu, Jan 9, 2020 at 9:10 AM John L Wagner - NOAA Federal
via
> RT <
> >> > > > met_help at ucar.edu> wrote:
> >> > > >
> >> > > > >
> >> > > > > Thu Jan 09 09:10:39 2020: Request 93726 was acted upon.
> >> > > > > Transaction: Ticket created by john.l.wagner at noaa.gov
> >> > > > > Queue: met_help
> >> > > > > Subject: Wind direction verification
> >> > > > > Owner: Nobody
> >> > > > > Requestors: john.l.wagner at noaa.gov
> >> > > > > Status: new
> >> > > > > Ticket <URL:
> >> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726
> >> > > >
> >> > > > >
> >> > > > >
> >> > > > > Greetings MET Help
> >> > > > > I was wondering if you had an update of the status of
wind
> >> direction
> >> > > > > verification? Looking at the slides you have online from
2018,
> it
> >> > > seems
> >> > > > > that wind dir verification is under development. The
slides
> also
> >> > > suggest
> >> > > > > that the verification would use the U and V components
instead
> of
> >> a
> >> > > wind
> >> > > > > direction field. I just want to make sure that this info
is
> still
> >> > > > > accurate. We are currently archiving the URMA wind
direction
> >> field,
> >> > so
> >> > > > if
> >> > > > > we need to change our archive, I want to do that ASAP.
If there
> >> is
> >> > > other
> >> > > > > info on what will be supported or what version it is
expected to
> >> be
> >> > in
> >> > > > that
> >> > > > > I am missing, please let me know.
> >> > > > > Thanks
> >> > > > > John
> >> > > > >
> >> > > > > --
> >> > > > > John Wagner
> >> > > > > Verification Task Lead
> >> > > > > COR Task Manager
> >> > > > > NOAA/National Weather Service
> >> > > > > Meteorological Development Laboratory
> >> > > > > Digital Forecast Services Branch
> >> > > > > SSMC2 Room 10106
> >> > > > > Silver Spring, MD 20910
> >> > > > > (301) 427-9471 (office)
> >> > > > > (908) 902-4155 (cell/text)
> >> > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > >
> >> > > --
> >> > > John Wagner
> >> > > Verification Task Lead
> >> > > COR Task Manager
> >> > > NOAA/National Weather Service
> >> > > Meteorological Development Laboratory
> >> > > Digital Forecast Services Branch
> >> > > SSMC2 Room 10106
> >> > > Silver Spring, MD 20910
> >> > > (301) 427-9471 (office)
> >> > > (908) 902-4155 (cell/text)
> >> > >
> >> > >
> >> >
> >> >
> >>
> >> --
> >> John Wagner
> >> Verification Task Lead
> >> COR Task Manager
> >> NOAA/National Weather Service
> >> Meteorological Development Laboratory
> >> Digital Forecast Services Branch
> >> SSMC2 Room 10106
> >> Silver Spring, MD 20910
> >> (301) 427-9471 (office)
> >> (908) 902-4155 (cell/text)
> >>
> >>
>
>
--
John Wagner
Verification Task Lead
COR Task Manager
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Branch
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
------------------------------------------------
Subject: Wind direction verification
From: John Halley Gotway
Time: Wed Apr 01 11:01:28 2020
Hi John,
I'm just cleaning up some old tickets in MET-Help and ran across this
one. I wanted to let you know that I did use the info you sent to
create a new development issue for
MET:https://github.com/NCAR/MET/issues/1236
Thanks,
John Halley Gotway
------------------------------------------------
More information about the Met_help
mailing list