[Met_help] [rt.rap.ucar.edu #93726] History for Wind direction verification

John Halley Gotway via RT met_help at ucar.edu
Wed Apr 1 11:12:43 MDT 2020


----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Greetings MET Help
I was wondering if you had an update of the status of wind direction
verification?  Looking at the slides you have online from 2018, it seems
that wind dir verification is under development.  The slides also suggest
that the verification would use the U and V components instead of a wind
direction field.  I just want to make sure that this info is still
accurate.  We are currently archiving the URMA wind direction field, so if
we need to change our archive, I want to do that ASAP.  If there is other
info on what will be supported or what version it is expected to be in that
I am missing, please let me know.
Thanks
John

-- 
John Wagner
Verification Task Lead
COR Task Manager
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Branch
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Wind direction verification
From: Minna Win
Time: Thu Jan 09 09:36:25 2020

Hi John,

It looks like you are interested in doing some wind direction
verification
in MET.  I believe the best person to address this status is the
engineering lead, John Halley Gotway.  I am going to assign this
ticket to
him.  Please allow a few business days for a full response.

Regards,
Minna
---------------
Minna Win
National Center for Atmospheric Research
Developmental Testbed Center
Phone: 303-497-8423
Fax:   303-497-8401



On Thu, Jan 9, 2020 at 9:10 AM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:

>
> Thu Jan 09 09:10:39 2020: Request 93726 was acted upon.
> Transaction: Ticket created by john.l.wagner at noaa.gov
>        Queue: met_help
>      Subject: Wind direction verification
>        Owner: Nobody
>   Requestors: john.l.wagner at noaa.gov
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726 >
>
>
> Greetings MET Help
> I was wondering if you had an update of the status of wind direction
> verification?  Looking at the slides you have online from 2018, it
seems
> that wind dir verification is under development.  The slides also
suggest
> that the verification would use the U and V components instead of a
wind
> direction field.  I just want to make sure that this info is still
> accurate.  We are currently archiving the URMA wind direction field,
so if
> we need to change our archive, I want to do that ASAP.  If there is
other
> info on what will be supported or what version it is expected to be
in that
> I am missing, please let me know.
> Thanks
> John
>
> --
> John Wagner
> Verification Task Lead
> COR Task Manager
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Branch
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>

------------------------------------------------
Subject: Wind direction verification
From: John L Wagner - NOAA Federal
Time: Thu Jan 09 09:38:06 2020

Thanks Minna, much appreciated.

On Thu, Jan 9, 2020 at 11:36 AM Minna Win via RT <met_help at ucar.edu>
wrote:

> Hi John,
>
> It looks like you are interested in doing some wind direction
verification
> in MET.  I believe the best person to address this status is the
> engineering lead, John Halley Gotway.  I am going to assign this
ticket to
> him.  Please allow a few business days for a full response.
>
> Regards,
> Minna
> ---------------
> Minna Win
> National Center for Atmospheric Research
> Developmental Testbed Center
> Phone: 303-497-8423
> Fax:   303-497-8401
>
>
>
> On Thu, Jan 9, 2020 at 9:10 AM John L Wagner - NOAA Federal via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Thu Jan 09 09:10:39 2020: Request 93726 was acted upon.
> > Transaction: Ticket created by john.l.wagner at noaa.gov
> >        Queue: met_help
> >      Subject: Wind direction verification
> >        Owner: Nobody
> >   Requestors: john.l.wagner at noaa.gov
> >       Status: new
> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726 >
> >
> >
> > Greetings MET Help
> > I was wondering if you had an update of the status of wind
direction
> > verification?  Looking at the slides you have online from 2018, it
seems
> > that wind dir verification is under development.  The slides also
suggest
> > that the verification would use the U and V components instead of
a wind
> > direction field.  I just want to make sure that this info is still
> > accurate.  We are currently archiving the URMA wind direction
field, so
> if
> > we need to change our archive, I want to do that ASAP.  If there
is other
> > info on what will be supported or what version it is expected to
be in
> that
> > I am missing, please let me know.
> > Thanks
> > John
> >
> > --
> > John Wagner
> > Verification Task Lead
> > COR Task Manager
> > NOAA/National Weather Service
> > Meteorological Development Laboratory
> > Digital Forecast Services Branch
> > SSMC2 Room 10106
> > Silver Spring, MD 20910
> > (301) 427-9471 (office)
> > (908) 902-4155 (cell/text)
> >
> >
>
>

--
John Wagner
Verification Task Lead
COR Task Manager
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Branch
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)

------------------------------------------------
Subject: Wind direction verification
From: John Halley Gotway
Time: Thu Jan 09 10:20:31 2020

Hi John,

We are actively working on the MET version 9.0 release.  We had
planned to
have it out in December 2019 now the end of January 2020 is a better
estimate.  We have made two intermediate test releases named met-
9.0_beta1
and met-9.0_beta2 and those release notes are listed here:
https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-9-0-beta1#notes
https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-9-0-beta2#notes

The remaining development tasks for inclusion in 9.0 are the "blocker"
issues from GitHub:
https://github.com/NCAR/MET/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22MET+9.0%22+label%3A%22priority%3A+blocker%22

However, there are no additional changes included in met-9.0 that
apply to
wind direction.

Hopefully met-8.1 already contains sufficient logic for your needs.
The
Grid-Stat and Point-Stat tools generate VL1L2 and VCNT output line
types.
VL1L2 are vector-based partial sums and VCNT are vector-based
continuous
statistics.  The "wind_thresh" configuration option for these tools
enable
you to filter the u/v pairs which are included in the VL1L2 and VCNT
sums
and stats.

Please take a look on page 200 of the met-8.1.2 user's guide:
https://dtcenter.org/sites/default/files/community-code/met/docs/user-
guide/MET_Users_Guide_v8.1.2.pdf

Do these stats looks sufficient?  Or can you please remind me of
specific
additional features that are needed?

Thanks,
John Halley Gotway



On Thu, Jan 9, 2020 at 9:38 AM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726 >
>
> Thanks Minna, much appreciated.
>
> On Thu, Jan 9, 2020 at 11:36 AM Minna Win via RT <met_help at ucar.edu>
> wrote:
>
> > Hi John,
> >
> > It looks like you are interested in doing some wind direction
> verification
> > in MET.  I believe the best person to address this status is the
> > engineering lead, John Halley Gotway.  I am going to assign this
ticket
> to
> > him.  Please allow a few business days for a full response.
> >
> > Regards,
> > Minna
> > ---------------
> > Minna Win
> > National Center for Atmospheric Research
> > Developmental Testbed Center
> > Phone: 303-497-8423
> > Fax:   303-497-8401
> >
> >
> >
> > On Thu, Jan 9, 2020 at 9:10 AM John L Wagner - NOAA Federal via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Thu Jan 09 09:10:39 2020: Request 93726 was acted upon.
> > > Transaction: Ticket created by john.l.wagner at noaa.gov
> > >        Queue: met_help
> > >      Subject: Wind direction verification
> > >        Owner: Nobody
> > >   Requestors: john.l.wagner at noaa.gov
> > >       Status: new
> > >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726
> >
> > >
> > >
> > > Greetings MET Help
> > > I was wondering if you had an update of the status of wind
direction
> > > verification?  Looking at the slides you have online from 2018,
it
> seems
> > > that wind dir verification is under development.  The slides
also
> suggest
> > > that the verification would use the U and V components instead
of a
> wind
> > > direction field.  I just want to make sure that this info is
still
> > > accurate.  We are currently archiving the URMA wind direction
field, so
> > if
> > > we need to change our archive, I want to do that ASAP.  If there
is
> other
> > > info on what will be supported or what version it is expected to
be in
> > that
> > > I am missing, please let me know.
> > > Thanks
> > > John
> > >
> > > --
> > > John Wagner
> > > Verification Task Lead
> > > COR Task Manager
> > > NOAA/National Weather Service
> > > Meteorological Development Laboratory
> > > Digital Forecast Services Branch
> > > SSMC2 Room 10106
> > > Silver Spring, MD 20910
> > > (301) 427-9471 (office)
> > > (908) 902-4155 (cell/text)
> > >
> > >
> >
> >
>
> --
> John Wagner
> Verification Task Lead
> COR Task Manager
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Branch
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>

------------------------------------------------
Subject: Wind direction verification
From: John L Wagner - NOAA Federal
Time: Thu Jan 09 13:43:51 2020

Thanks John.  I did miss the DIR_ERR and DIR_ABSERR on page 200.
Those
will be useful for our MAEs.
I thought I had read that wind_thresh only applied to wind speed and
not
direction.  I will take a closer look at that.  We verify wind
direction
conditioned on forecast and observed winds exceeding 8 knots.  Can I
pass
grid_stat both the wind direction and speed grids?
We are currently computing a Heidke Skill Score for wind direction
based on
thresholds that we define.  Our code currently does an adjustment to
ensure
that all of the north winds fall into the same category.  Is there
something similar available in MET?  Or would we have to preprocess
the
data to ensure that north isn't split between 0-45 degrees and 335-360
degrees?
Thanks!  Your help is always appreciated.
John

On Thu, Jan 9, 2020 at 12:20 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:

> Hi John,
>
> We are actively working on the MET version 9.0 release.  We had
planned to
> have it out in December 2019 now the end of January 2020 is a better
> estimate.  We have made two intermediate test releases named met-
9.0_beta1
> and met-9.0_beta2 and those release notes are listed here:
>
> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-9-0-beta1#notes
>
> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-9-0-beta2#notes
>
> The remaining development tasks for inclusion in 9.0 are the
"blocker"
> issues from GitHub:
>
>
https://github.com/NCAR/MET/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22MET+9.0%22+label%3A%22priority%3A+blocker%22
>
> However, there are no additional changes included in met-9.0 that
apply to
> wind direction.
>
> Hopefully met-8.1 already contains sufficient logic for your needs.
The
> Grid-Stat and Point-Stat tools generate VL1L2 and VCNT output line
types.
> VL1L2 are vector-based partial sums and VCNT are vector-based
continuous
> statistics.  The "wind_thresh" configuration option for these tools
enable
> you to filter the u/v pairs which are included in the VL1L2 and VCNT
sums
> and stats.
>
> Please take a look on page 200 of the met-8.1.2 user's guide:
>
> https://dtcenter.org/sites/default/files/community-
code/met/docs/user-guide/MET_Users_Guide_v8.1.2.pdf
>
> Do these stats looks sufficient?  Or can you please remind me of
specific
> additional features that are needed?
>
> Thanks,
> John Halley Gotway
>
>
>
> On Thu, Jan 9, 2020 at 9:38 AM John L Wagner - NOAA Federal via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726 >
> >
> > Thanks Minna, much appreciated.
> >
> > On Thu, Jan 9, 2020 at 11:36 AM Minna Win via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi John,
> > >
> > > It looks like you are interested in doing some wind direction
> > verification
> > > in MET.  I believe the best person to address this status is the
> > > engineering lead, John Halley Gotway.  I am going to assign this
ticket
> > to
> > > him.  Please allow a few business days for a full response.
> > >
> > > Regards,
> > > Minna
> > > ---------------
> > > Minna Win
> > > National Center for Atmospheric Research
> > > Developmental Testbed Center
> > > Phone: 303-497-8423
> > > Fax:   303-497-8401
> > >
> > >
> > >
> > > On Thu, Jan 9, 2020 at 9:10 AM John L Wagner - NOAA Federal via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Thu Jan 09 09:10:39 2020: Request 93726 was acted upon.
> > > > Transaction: Ticket created by john.l.wagner at noaa.gov
> > > >        Queue: met_help
> > > >      Subject: Wind direction verification
> > > >        Owner: Nobody
> > > >   Requestors: john.l.wagner at noaa.gov
> > > >       Status: new
> > > >  Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726
> > >
> > > >
> > > >
> > > > Greetings MET Help
> > > > I was wondering if you had an update of the status of wind
direction
> > > > verification?  Looking at the slides you have online from
2018, it
> > seems
> > > > that wind dir verification is under development.  The slides
also
> > suggest
> > > > that the verification would use the U and V components instead
of a
> > wind
> > > > direction field.  I just want to make sure that this info is
still
> > > > accurate.  We are currently archiving the URMA wind direction
field,
> so
> > > if
> > > > we need to change our archive, I want to do that ASAP.  If
there is
> > other
> > > > info on what will be supported or what version it is expected
to be
> in
> > > that
> > > > I am missing, please let me know.
> > > > Thanks
> > > > John
> > > >
> > > > --
> > > > John Wagner
> > > > Verification Task Lead
> > > > COR Task Manager
> > > > NOAA/National Weather Service
> > > > Meteorological Development Laboratory
> > > > Digital Forecast Services Branch
> > > > SSMC2 Room 10106
> > > > Silver Spring, MD 20910
> > > > (301) 427-9471 (office)
> > > > (908) 902-4155 (cell/text)
> > > >
> > > >
> > >
> > >
> >
> > --
> > John Wagner
> > Verification Task Lead
> > COR Task Manager
> > NOAA/National Weather Service
> > Meteorological Development Laboratory
> > Digital Forecast Services Branch
> > SSMC2 Room 10106
> > Silver Spring, MD 20910
> > (301) 427-9471 (office)
> > (908) 902-4155 (cell/text)
> >
> >
>
>

--
John Wagner
Verification Task Lead
COR Task Manager
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Branch
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)

------------------------------------------------
Subject: Wind direction verification
From: John Halley Gotway
Time: Fri Jan 10 09:17:40 2020

John,

Yes, the wind_thresh configuration option does apply to the VCNT
output
line type.  The MET tarball runs some sample test scripts when you run
"make test" after compiling the code.  I modified the configuration
file
for one of those tests to demonstrate:

*cd met-8.1.2/scripts*
*export TEST_OUT_DIR="../out"*
# Run existing command to make sure it's working OK
*../bin/grid_stat \*
*../data/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212 \*
*../data/sample_fcst/2005080712/wrfprs_ruc13_00.tm00_G212 \*
*config/GridStatConfig_all \*
*-outdir ${TEST_OUT_DIR}/grid_stat -v 2*

#  Modify the configuration file (config/GridStatConfig_all) by
setting...
and then rerun the command above:







*//// Forecast and observation fields to be verified//fcst = {
 wind_thresh = [ NA, NA, NA ];   field = [      { name = "UGRD"; level
= [
"Z10" ]; },*




*      { name = "VGRD"; level = [ "Z10" ];  }   ];}obs = fcst;obs = {
wind_thresh = [ NA, >=2, >=5 ]; }*

Instead of getting 1 VCNT output line for each masking area, you'll
now get
3: one using all U/V pairs, one using U/V pairs with speed >=2 m/s,
and one
using U/V pairs with speed >=5 m/s.  I configured it so that we're
filtering based on the OBSERVATION wind speed.  But you could do the
forecast instead.  Or both, using the intersection (i.e. wind_logic =
INTERSECTION;).  A threshold of type NA always evaluates to true, so
setting "wind_thresh = [ NA, NA, NA ];" in the fcst dictionary just
means
that we're not filtering the fcst U/V vectors.

This configuration results in the attached VCNT output.  You'll see
the
TOTAL column (# of pairs used) on the first 3 lines decreases from
5558 to
4999 to 2301 as we increase the wind speed threshold from none to >=2
to
>=5.

Currently, if you try to verify wind direction directly in Point-Stat
or
Grid-Stat, you'll get this error message:
ERROR  : GridStatVxOpt::process_config() -> wind direction may not be
verified using grid_stat.

That check is there because the continuous statistics in the CNT line
type
are not well-suited to circular variables (like wind direction).
However,
you mentioned applying Heidke Skill Score to wind direction, which is
a
categorical statistic.  I'd honestly never thought about verifying
wind
direction in a categorical way.  That certainly seems doable... as
long as
care is taken in defining the categorical thresholds well.

If you agree, I can write up a development issue for the verification
of
wind direction using categorical statistics.  We'll need to make
Point-Stat
and Grid-Stat smart enough to *not* compute continuous stats for them.
Do
you have a sense for whether or not categorical vx of wind direction
is
widely done?  Is this a common practice that should be added to a
community
vx package?

Thanks,
John

On Thu, Jan 9, 2020 at 1:44 PM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726 >
>
> Thanks John.  I did miss the DIR_ERR and DIR_ABSERR on page 200.
Those
> will be useful for our MAEs.
> I thought I had read that wind_thresh only applied to wind speed and
not
> direction.  I will take a closer look at that.  We verify wind
direction
> conditioned on forecast and observed winds exceeding 8 knots.  Can I
pass
> grid_stat both the wind direction and speed grids?
> We are currently computing a Heidke Skill Score for wind direction
based on
> thresholds that we define.  Our code currently does an adjustment to
ensure
> that all of the north winds fall into the same category.  Is there
> something similar available in MET?  Or would we have to preprocess
the
> data to ensure that north isn't split between 0-45 degrees and 335-
360
> degrees?
> Thanks!  Your help is always appreciated.
> John
>
> On Thu, Jan 9, 2020 at 12:20 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Hi John,
> >
> > We are actively working on the MET version 9.0 release.  We had
planned
> to
> > have it out in December 2019 now the end of January 2020 is a
better
> > estimate.  We have made two intermediate test releases named
> met-9.0_beta1
> > and met-9.0_beta2 and those release notes are listed here:
> >
> >
> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-9-0-beta1#notes
> >
> >
> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-9-0-beta2#notes
> >
> > The remaining development tasks for inclusion in 9.0 are the
"blocker"
> > issues from GitHub:
> >
> >
>
https://github.com/NCAR/MET/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22MET+9.0%22+label%3A%22priority%3A+blocker%22
> >
> > However, there are no additional changes included in met-9.0 that
apply
> to
> > wind direction.
> >
> > Hopefully met-8.1 already contains sufficient logic for your
needs.  The
> > Grid-Stat and Point-Stat tools generate VL1L2 and VCNT output line
types.
> > VL1L2 are vector-based partial sums and VCNT are vector-based
continuous
> > statistics.  The "wind_thresh" configuration option for these
tools
> enable
> > you to filter the u/v pairs which are included in the VL1L2 and
VCNT sums
> > and stats.
> >
> > Please take a look on page 200 of the met-8.1.2 user's guide:
> >
> >
> https://dtcenter.org/sites/default/files/community-
code/met/docs/user-guide/MET_Users_Guide_v8.1.2.pdf
> >
> > Do these stats looks sufficient?  Or can you please remind me of
specific
> > additional features that are needed?
> >
> > Thanks,
> > John Halley Gotway
> >
> >
> >
> > On Thu, Jan 9, 2020 at 9:38 AM John L Wagner - NOAA Federal via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726 >
> > >
> > > Thanks Minna, much appreciated.
> > >
> > > On Thu, Jan 9, 2020 at 11:36 AM Minna Win via RT
<met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi John,
> > > >
> > > > It looks like you are interested in doing some wind direction
> > > verification
> > > > in MET.  I believe the best person to address this status is
the
> > > > engineering lead, John Halley Gotway.  I am going to assign
this
> ticket
> > > to
> > > > him.  Please allow a few business days for a full response.
> > > >
> > > > Regards,
> > > > Minna
> > > > ---------------
> > > > Minna Win
> > > > National Center for Atmospheric Research
> > > > Developmental Testbed Center
> > > > Phone: 303-497-8423
> > > > Fax:   303-497-8401
> > > >
> > > >
> > > >
> > > > On Thu, Jan 9, 2020 at 9:10 AM John L Wagner - NOAA Federal
via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Thu Jan 09 09:10:39 2020: Request 93726 was acted upon.
> > > > > Transaction: Ticket created by john.l.wagner at noaa.gov
> > > > >        Queue: met_help
> > > > >      Subject: Wind direction verification
> > > > >        Owner: Nobody
> > > > >   Requestors: john.l.wagner at noaa.gov
> > > > >       Status: new
> > > > >  Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726
> > > >
> > > > >
> > > > >
> > > > > Greetings MET Help
> > > > > I was wondering if you had an update of the status of wind
> direction
> > > > > verification?  Looking at the slides you have online from
2018, it
> > > seems
> > > > > that wind dir verification is under development.  The slides
also
> > > suggest
> > > > > that the verification would use the U and V components
instead of a
> > > wind
> > > > > direction field.  I just want to make sure that this info is
still
> > > > > accurate.  We are currently archiving the URMA wind
direction
> field,
> > so
> > > > if
> > > > > we need to change our archive, I want to do that ASAP.  If
there is
> > > other
> > > > > info on what will be supported or what version it is
expected to be
> > in
> > > > that
> > > > > I am missing, please let me know.
> > > > > Thanks
> > > > > John
> > > > >
> > > > > --
> > > > > John Wagner
> > > > > Verification Task Lead
> > > > > COR Task Manager
> > > > > NOAA/National Weather Service
> > > > > Meteorological Development Laboratory
> > > > > Digital Forecast Services Branch
> > > > > SSMC2 Room 10106
> > > > > Silver Spring, MD 20910
> > > > > (301) 427-9471 (office)
> > > > > (908) 902-4155 (cell/text)
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > --
> > > John Wagner
> > > Verification Task Lead
> > > COR Task Manager
> > > NOAA/National Weather Service
> > > Meteorological Development Laboratory
> > > Digital Forecast Services Branch
> > > SSMC2 Room 10106
> > > Silver Spring, MD 20910
> > > (301) 427-9471 (office)
> > > (908) 902-4155 (cell/text)
> > >
> > >
> >
> >
>
> --
> John Wagner
> Verification Task Lead
> COR Task Manager
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Branch
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>

------------------------------------------------
Subject: Wind direction verification
From: John Halley Gotway
Time: Fri Jan 10 09:17:40 2020

VERSION MODEL DESC FCST_LEAD FCST_VALID_BEG  FCST_VALID_END  OBS_LEAD
OBS_VALID_BEG   OBS_VALID_END   FCST_VAR  FCST_UNITS FCST_LEV OBS_VAR
OBS_UNITS OBS_LEV OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH
OBS_THRESH COV_THRESH ALPHA LINE_TYPE TOTAL FBAR    FBAR_BCL FBAR_BCU
OBAR    OBAR_BCL OBAR_BCU FS_RMS  FS_RMS_BCL FS_RMS_BCU OS_RMS
OS_RMS_BCL OS_RMS_BCU MSVE     MSVE_BCL MSVE_BCU RMSVE   RMSVE_BCL
RMSVE_BCU FSTDEV  FSTDEV_BCL FSTDEV_BCU OSTDEV  OSTDEV_BCL OSTDEV_BCU
FDIR      FDIR_BCL FDIR_BCU ODIR      ODIR_BCL ODIR_BCU FBAR_SPEED
FBAR_SPEED_BCL FBAR_SPEED_BCU OBAR_SPEED OBAR_SPEED_BCL OBAR_SPEED_BCU
VDIFF_SPEED VDIFF_SPEED_BCL VDIFF_SPEED_BCU VDIFF_DIR VDIFF_DIR_BCL
VDIFF_DIR_BCU SPEED_ERR   SPEED_ERR_BCL SPEED_ERR_BCU SPEED_ABSERR
SPEED_ABSERR_BCL SPEED_ABSERR_BCU DIR_ERR  DIR_ERR_BCL DIR_ERR_BCU
DIR_ABSERR DIR_ABSERR_BCL DIR_ABSERR_BCU
V8.1.2  WRF   NA   120000    20050807_120000 20050807_120000 000000
20050807_120000 20050807_120000 UGRD_VGRD m/s        Z10
UGRD_VGRD m/s       Z10     ANALYS DTC165  NEAREST     1           NA
NA         NA         NA    VCNT       5558 3.74987       NA       NA
4.54808       NA       NA 4.41425         NA         NA 5.20931
NA         NA  8.72616       NA       NA 2.95401        NA        NA
2.32896         NA         NA 2.54005         NA         NA 321.53259
NA       NA 318.29533       NA       NA    1.5864              NA
NA    1.47852             NA             NA    0.13829
NA              NA 358.67129            NA            NA  0.10788
NA            NA   0.10788                  NA               NA
-3.23726          NA          NA    3.23726             NA
NA
V8.1.2  WRF   NA   120000    20050807_120000 20050807_120000 000000
20050807_120000 20050807_120000 UGRD_VGRD m/s        Z10
UGRD_VGRD m/s       Z10     ANALYS DTC165  NEAREST     1           NA
>=2        NA         NA    VCNT       4999 3.94128       NA       NA
4.9368        NA       NA 4.59178         NA         NA 5.47922
NA         NA  9.14583       NA       NA 3.02421        NA        NA
2.356           NA         NA 2.37694         NA         NA 321.2039
NA       NA 318.3645        NA       NA    1.76777             NA
NA    1.65765             NA             NA    0.139
NA              NA 357.41527            NA            NA  0.11011
NA            NA   0.11011                  NA               NA
-2.8394           NA          NA    2.8394              NA
NA
V8.1.2  WRF   NA   120000    20050807_120000 20050807_120000 000000
20050807_120000 20050807_120000 UGRD_VGRD m/s        Z10
UGRD_VGRD m/s       Z10     ANALYS DTC165  NEAREST     1           NA
>=5        NA         NA    VCNT       2301 5.12665       NA       NA
6.8945        NA       NA 5.75431         NA         NA 7.20507
NA         NA 11.711         NA       NA 3.42213        NA        NA
2.61334         NA         NA 2.09261         NA         NA 324.25564
NA       NA 323.04946       NA       NA    2.91342             NA
NA    3.16417             NA             NA    0.25877
NA              NA 129.33979            NA            NA -0.25075
NA            NA   0.25075                  NA               NA
-1.20619          NA          NA    1.20619             NA
NA
V8.1.2  WRF   NA   120000    20050807_120000 20050807_120000 000000
20050807_120000 20050807_120000 UGRD_VGRD m/s        Z10
UGRD_VGRD m/s       Z10     ANALYS DTC166  NEAREST     1           NA
NA         NA         NA    VCNT       6862 3.38203       NA       NA
3.39417       NA       NA 4.08693         NA         NA 4.12265
NA         NA  5.25182       NA       NA 2.29169        NA        NA
2.29453         NA         NA 2.34005         NA         NA 198.85917
NA       NA 197.91096       NA       NA    1.11762             NA
NA    1.01922             NA             NA    0.099969
NA              NA 208.5725             NA            NA  0.098396
NA            NA   0.098396                 NA               NA
-0.94821          NA          NA    0.94821             NA
NA
V8.1.2  WRF   NA   120000    20050807_120000 20050807_120000 000000
20050807_120000 20050807_120000 UGRD_VGRD m/s        Z10
UGRD_VGRD m/s       Z10     ANALYS DTC166  NEAREST     1           NA
>=2        NA         NA    VCNT       5064 3.95965       NA       NA
4.25088       NA       NA 4.59894         NA         NA 4.75464
NA         NA  5.97137       NA       NA 2.44364        NA        NA
2.33911         NA         NA 2.12994         NA         NA 203.43175
NA       NA 200.77084       NA       NA    1.45029             NA
NA    1.36308             NA             NA    0.10894
NA              NA 238.94247            NA            NA  0.087212
NA            NA   0.087212                 NA               NA
-2.66091          NA          NA    2.66091             NA
NA
V8.1.2  WRF   NA   120000    20050807_120000 20050807_120000 000000
20050807_120000 20050807_120000 UGRD_VGRD m/s        Z10
UGRD_VGRD m/s       Z10     ANALYS DTC166  NEAREST     1           NA
>=5        NA         NA    VCNT       1707 6.22249       NA       NA
6.63735       NA       NA 6.57925         NA         NA 6.89914
NA         NA  7.17047       NA       NA 2.67777        NA        NA
2.13707         NA         NA 1.88249         NA         NA 219.19075
NA       NA 218.45588       NA       NA    2.98734             NA
NA    2.98765             NA             NA    0.038318
NA              NA 309.289              NA            NA -0.00031144
NA            NA   0.00031144               NA               NA
-0.73487          NA          NA    0.73487             NA
NA
V8.1.2  WRF   NA   120000    20050807_120000 20050807_120000 000000
20050807_120000 20050807_120000 UGRD_VGRD m/s        Z10
UGRD_VGRD m/s       Z10     ANALYS CONUS   NEAREST     1           NA
NA         NA         NA    VCNT       5479 2.42081       NA       NA
2.94389       NA       NA 2.79816         NA         NA 3.59153
NA         NA  6.64081       NA       NA 2.57698        NA        NA
1.40336         NA         NA 2.05732         NA         NA 191.7345
NA       NA 195.62712       NA       NA    0.51859             NA
NA    0.74411             NA             NA    0.22944
NA              NA  24.45345            NA            NA -0.22553
NA            NA   0.22553                  NA               NA
3.89263          NA          NA    3.89263             NA
NA
V8.1.2  WRF   NA   120000    20050807_120000 20050807_120000 000000
20050807_120000 20050807_120000 UGRD_VGRD m/s        Z10
UGRD_VGRD m/s       Z10     ANALYS CONUS   NEAREST     1           NA
>=2        NA         NA    VCNT       3738 2.79318       NA       NA
3.86247       NA       NA 3.14667         NA         NA 4.28418
NA         NA  8.42055       NA       NA 2.90182        NA        NA
1.44904         NA         NA 1.85353         NA         NA 200.69646
NA       NA 199.90877       NA       NA    0.73851             NA
NA    1.08253             NA             NA    0.34424
NA              NA  18.21877            NA            NA -0.34402
NA            NA   0.34402                  NA               NA
-0.78768          NA          NA    0.78768             NA
NA
V8.1.2  WRF   NA   120000    20050807_120000 20050807_120000 000000
20050807_120000 20050807_120000 UGRD_VGRD m/s        Z10
UGRD_VGRD m/s       Z10     ANALYS CONUS   NEAREST     1           NA
>=5        NA         NA    VCNT        973 3.81144       NA       NA
6.41176       NA       NA 4.22176         NA         NA 6.60771
NA         NA 16.73279       NA       NA 4.09057        NA        NA
1.81553         NA         NA 1.59725         NA         NA 232.80359
NA       NA 225.59105       NA       NA    1.53664             NA
NA    2.4398              NA             NA    0.93543
NA              NA  33.68885            NA            NA -0.90316
NA            NA   0.90316                  NA               NA
-7.21254          NA          NA    7.21254             NA
NA
V8.1.2  WRF   NA   120000    20050807_120000 20050807_120000 000000
20050807_120000 20050807_120000 UGRD_VGRD m/s        Z10
UGRD_VGRD m/s       Z10     ANALYS LMV     NEAREST     1           NA
NA         NA         NA    VCNT        524 1.77358       NA       NA
1.55639       NA       NA 1.89213         NA         NA 1.7337
NA         NA  2.65076       NA       NA 1.62812        NA        NA
0.65923         NA         NA 0.76378         NA         NA  86.13029
NA       NA  96.1869        NA       NA    0.79086             NA
NA    0.75213             NA             NA    0.14063
NA              NA  17.07817            NA            NA  0.038723
NA            NA   0.038723                 NA               NA
10.05661          NA          NA   10.05661             NA
NA
V8.1.2  WRF   NA   120000    20050807_120000 20050807_120000 000000
20050807_120000 20050807_120000 UGRD_VGRD m/s        Z10
UGRD_VGRD m/s       Z10     ANALYS LMV     NEAREST     1           NA
>=2        NA         NA    VCNT        204 1.83001       NA       NA
2.3393        NA       NA 1.94407         NA         NA 2.38151
NA         NA  2.98039       NA       NA 1.72638        NA        NA
0.65611         NA         NA 0.44636         NA         NA  90.98028
NA       NA  97.35791       NA       NA    1.02207             NA
NA    1.32491             NA             NA    0.32935
NA              NA 297.52227            NA            NA -0.30284
NA            NA   0.30284                  NA               NA
6.37763          NA          NA    6.37763             NA
NA

------------------------------------------------
Subject: Wind direction verification
From: John Halley Gotway
Time: Fri Jan 10 09:32:57 2020

John,

I did think of one additional complication that'd make the categorical
vx
of wind direction more complicated.

Presumably, just like with the VCNT line type, you'd want to filter
the
pairs used based on the wind speed.  I'd assume that the categorical
stats
for stronger winds would be much better than those for light and
variable
winds.  But you'd also need to define the thresholds for the
categories,
such as northerly winds being between 45 and 135 degrees, for example.

So we might configure it like this:




*fcst = {   wind_thresh = [ NA, >=2, >=5 ];   field = [      { name =
"WDIR";*
*        level = [ "Z10" ];*


*        cat_thresh = [ >=45&&<=135 ]; }   ];}*

We are applying 2 threshold steps here... (1) wind_thresh to filter
the U/V
pairs and (2) cat_thresh to define the categories.

But the output stat header only has column for FCST_THRESH.  For
categorical output (CTC, CTS, MCTC, and MCTS lines types), we write
the
cat_thresh to that column.  For vector stats (VL1L2, VAL1L2, and VCNT
line
types), we write the wind_thresh to that column.

In this case, we'd have 2 thresholds to include in the metadata and
only 1
column to contain them.  Perhaps we could just concatenate them with a
comma, but that'll make the logic for reading them back into STAT-
Analysis
a bit more tricky.

Just some more details to consider.

Thanks,
John





On Fri, Jan 10, 2020 at 9:17 AM John Halley Gotway <johnhg at ucar.edu>
wrote:

> John,
>
> Yes, the wind_thresh configuration option does apply to the VCNT
output
> line type.  The MET tarball runs some sample test scripts when you
run
> "make test" after compiling the code.  I modified the configuration
file
> for one of those tests to demonstrate:
>
> *cd met-8.1.2/scripts*
> *export TEST_OUT_DIR="../out"*
> # Run existing command to make sure it's working OK
> *../bin/grid_stat \*
> *../data/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212 \*
> *../data/sample_fcst/2005080712/wrfprs_ruc13_00.tm00_G212 \*
> *config/GridStatConfig_all \*
> *-outdir ${TEST_OUT_DIR}/grid_stat -v 2*
>
> #  Modify the configuration file (config/GridStatConfig_all) by
setting...
> and then rerun the command above:
>
>
>
>
>
>
>
> *//// Forecast and observation fields to be verified//fcst = {
>  wind_thresh = [ NA, NA, NA ];   field = [      { name = "UGRD";
level = [
> "Z10" ]; },*
>
>
>
>
> *      { name = "VGRD"; level = [ "Z10" ];  }   ];}obs = fcst;obs =
{
> wind_thresh = [ NA, >=2, >=5 ]; }*
>
> Instead of getting 1 VCNT output line for each masking area, you'll
now
> get 3: one using all U/V pairs, one using U/V pairs with speed >=2
m/s, and
> one using U/V pairs with speed >=5 m/s.  I configured it so that
we're
> filtering based on the OBSERVATION wind speed.  But you could do the
> forecast instead.  Or both, using the intersection (i.e. wind_logic
=
> INTERSECTION;).  A threshold of type NA always evaluates to true, so
> setting "wind_thresh = [ NA, NA, NA ];" in the fcst dictionary just
means
> that we're not filtering the fcst U/V vectors.
>
> This configuration results in the attached VCNT output.  You'll see
the
> TOTAL column (# of pairs used) on the first 3 lines decreases from
5558 to
> 4999 to 2301 as we increase the wind speed threshold from none to
>=2 to
> >=5.
>
> Currently, if you try to verify wind direction directly in Point-
Stat or
> Grid-Stat, you'll get this error message:
> ERROR  : GridStatVxOpt::process_config() -> wind direction may not
be
> verified using grid_stat.
>
> That check is there because the continuous statistics in the CNT
line type
> are not well-suited to circular variables (like wind direction).
However,
> you mentioned applying Heidke Skill Score to wind direction, which
is a
> categorical statistic.  I'd honestly never thought about verifying
wind
> direction in a categorical way.  That certainly seems doable... as
long as
> care is taken in defining the categorical thresholds well.
>
> If you agree, I can write up a development issue for the
verification of
> wind direction using categorical statistics.  We'll need to make
Point-Stat
> and Grid-Stat smart enough to *not* compute continuous stats for
them.  Do
> you have a sense for whether or not categorical vx of wind direction
is
> widely done?  Is this a common practice that should be added to a
community
> vx package?
>
> Thanks,
> John
>
> On Thu, Jan 9, 2020 at 1:44 PM John L Wagner - NOAA Federal via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726 >
>>
>> Thanks John.  I did miss the DIR_ERR and DIR_ABSERR on page 200.
Those
>> will be useful for our MAEs.
>> I thought I had read that wind_thresh only applied to wind speed
and not
>> direction.  I will take a closer look at that.  We verify wind
direction
>> conditioned on forecast and observed winds exceeding 8 knots.  Can
I pass
>> grid_stat both the wind direction and speed grids?
>> We are currently computing a Heidke Skill Score for wind direction
based
>> on
>> thresholds that we define.  Our code currently does an adjustment
to
>> ensure
>> that all of the north winds fall into the same category.  Is there
>> something similar available in MET?  Or would we have to preprocess
the
>> data to ensure that north isn't split between 0-45 degrees and 335-
360
>> degrees?
>> Thanks!  Your help is always appreciated.
>> John
>>
>> On Thu, Jan 9, 2020 at 12:20 PM John Halley Gotway via RT <
>> met_help at ucar.edu>
>> wrote:
>>
>> > Hi John,
>> >
>> > We are actively working on the MET version 9.0 release.  We had
planned
>> to
>> > have it out in December 2019 now the end of January 2020 is a
better
>> > estimate.  We have made two intermediate test releases named
>> met-9.0_beta1
>> > and met-9.0_beta2 and those release notes are listed here:
>> >
>> >
>> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-9-0-beta1#notes
>> >
>> >
>> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-9-0-beta2#notes
>> >
>> > The remaining development tasks for inclusion in 9.0 are the
"blocker"
>> > issues from GitHub:
>> >
>> >
>>
https://github.com/NCAR/MET/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22MET+9.0%22+label%3A%22priority%3A+blocker%22
>> >
>> > However, there are no additional changes included in met-9.0 that
apply
>> to
>> > wind direction.
>> >
>> > Hopefully met-8.1 already contains sufficient logic for your
needs.  The
>> > Grid-Stat and Point-Stat tools generate VL1L2 and VCNT output
line
>> types.
>> > VL1L2 are vector-based partial sums and VCNT are vector-based
continuous
>> > statistics.  The "wind_thresh" configuration option for these
tools
>> enable
>> > you to filter the u/v pairs which are included in the VL1L2 and
VCNT
>> sums
>> > and stats.
>> >
>> > Please take a look on page 200 of the met-8.1.2 user's guide:
>> >
>> >
>> https://dtcenter.org/sites/default/files/community-
code/met/docs/user-guide/MET_Users_Guide_v8.1.2.pdf
>> >
>> > Do these stats looks sufficient?  Or can you please remind me of
>> specific
>> > additional features that are needed?
>> >
>> > Thanks,
>> > John Halley Gotway
>> >
>> >
>> >
>> > On Thu, Jan 9, 2020 at 9:38 AM John L Wagner - NOAA Federal via
RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726 >
>> > >
>> > > Thanks Minna, much appreciated.
>> > >
>> > > On Thu, Jan 9, 2020 at 11:36 AM Minna Win via RT
<met_help at ucar.edu>
>> > > wrote:
>> > >
>> > > > Hi John,
>> > > >
>> > > > It looks like you are interested in doing some wind direction
>> > > verification
>> > > > in MET.  I believe the best person to address this status is
the
>> > > > engineering lead, John Halley Gotway.  I am going to assign
this
>> ticket
>> > > to
>> > > > him.  Please allow a few business days for a full response.
>> > > >
>> > > > Regards,
>> > > > Minna
>> > > > ---------------
>> > > > Minna Win
>> > > > National Center for Atmospheric Research
>> > > > Developmental Testbed Center
>> > > > Phone: 303-497-8423
>> > > > Fax:   303-497-8401
>> > > >
>> > > >
>> > > >
>> > > > On Thu, Jan 9, 2020 at 9:10 AM John L Wagner - NOAA Federal
via RT <
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > > >
>> > > > > Thu Jan 09 09:10:39 2020: Request 93726 was acted upon.
>> > > > > Transaction: Ticket created by john.l.wagner at noaa.gov
>> > > > >        Queue: met_help
>> > > > >      Subject: Wind direction verification
>> > > > >        Owner: Nobody
>> > > > >   Requestors: john.l.wagner at noaa.gov
>> > > > >       Status: new
>> > > > >  Ticket <URL:
>> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726
>> > > >
>> > > > >
>> > > > >
>> > > > > Greetings MET Help
>> > > > > I was wondering if you had an update of the status of wind
>> direction
>> > > > > verification?  Looking at the slides you have online from
2018, it
>> > > seems
>> > > > > that wind dir verification is under development.  The
slides also
>> > > suggest
>> > > > > that the verification would use the U and V components
instead of
>> a
>> > > wind
>> > > > > direction field.  I just want to make sure that this info
is still
>> > > > > accurate.  We are currently archiving the URMA wind
direction
>> field,
>> > so
>> > > > if
>> > > > > we need to change our archive, I want to do that ASAP.  If
there
>> is
>> > > other
>> > > > > info on what will be supported or what version it is
expected to
>> be
>> > in
>> > > > that
>> > > > > I am missing, please let me know.
>> > > > > Thanks
>> > > > > John
>> > > > >
>> > > > > --
>> > > > > John Wagner
>> > > > > Verification Task Lead
>> > > > > COR Task Manager
>> > > > > NOAA/National Weather Service
>> > > > > Meteorological Development Laboratory
>> > > > > Digital Forecast Services Branch
>> > > > > SSMC2 Room 10106
>> > > > > Silver Spring, MD 20910
>> > > > > (301) 427-9471 (office)
>> > > > > (908) 902-4155 (cell/text)
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> > > --
>> > > John Wagner
>> > > Verification Task Lead
>> > > COR Task Manager
>> > > NOAA/National Weather Service
>> > > Meteorological Development Laboratory
>> > > Digital Forecast Services Branch
>> > > SSMC2 Room 10106
>> > > Silver Spring, MD 20910
>> > > (301) 427-9471 (office)
>> > > (908) 902-4155 (cell/text)
>> > >
>> > >
>> >
>> >
>>
>> --
>> John Wagner
>> Verification Task Lead
>> COR Task Manager
>> NOAA/National Weather Service
>> Meteorological Development Laboratory
>> Digital Forecast Services Branch
>> SSMC2 Room 10106
>> Silver Spring, MD 20910
>> (301) 427-9471 (office)
>> (908) 902-4155 (cell/text)
>>
>>

------------------------------------------------
Subject: Wind direction verification
From: John L Wagner - NOAA Federal
Time: Thu Jan 16 11:26:58 2020

Hi John
Sorry for the delay in my response.  Brooke has been running some
tests
with wind direction for grid_stat and I have been digging into how we
compute our scores in our MOS-2000 software.
To answer your questions from your last two emails, we would need a
way to
define the categories for wind direction in order to compute the HSS.
The
categories that we're using currently are  25, 70, 115, 160, 205, 250,
295,
and 999 (or 345).  These numbers at first made no sense to me until we
looked at the data and realized that our station-based forecasts and
obs
were all rounded to ten degrees.  This made 4 or 5 possibilities for
each
wind direction category:

North: -15 to 25
Obs: -10, 0, 10, 20

Northeast: 25 to 70
Obs: 30, 40, 50, 60, 70

East: 70 to 115
Obs: 80, 90, 100, 110

Southeast: 115 to 160
Obs: 120, 130, 140, 150, 160

South: 155 to 205
Obs: 170, 180, 190, 200

Southwest: 205 to 250
Obs: 210, 220, 230, 240, 250

West: 250 to 295
Obs: 260, 270, 280, 290

Northwest: 295 to 999
Obs: 300, 310, 320, 330, 340

We do subtract 360 degrees from any forecast or ob greater than 345
degrees
to ensure that all of the north winds are in the same category.  I'm
not
sure yet if these categories make sense yet for gridded wind
direction,
where the numbers likely will not be rounded for us.
I'm not sure how many other centers compute the HSS for wind
direction.  I
know that MDL has been calculating it for at least the last 12-15
years.
It does seem like a useful score to me.
Brooke was able to trick grid_stat into computing a HSS by changing
the
variable name from WDir to WD in the netcdf files.  This trick didn't
work
for point_stat though which is where we need to compute this (for
reference, we currently compute MAE, HSS, relative frequencies, and
convergence scores for station-based
<https://veritas.nws.noaa.gov/ndfd-
stats/verification/index.php?mo_id=201910&ty_id=pt&rr_id=ptreray&re_id=co&wx_id=wd&pr_id=&md_id=nd&cy_id=00&pd_id=pl>
forecasts and MAE for gridded
<https://veritas.nws.noaa.gov/ndfd-
stats/wfosummary/gridded/actionform.php?station1=US&station2=SAME&station3=SAME&station4=SAME&element=cl&rng=None&beginMonth=201910&data_type1=NDFD&data_type2=GMOS&data_type3=WPC&data_type4=BLEND&cycle=0&proj=INC&endMonth=201910&wfo=US&submit=Plot+Graph&comp_type=URMA>
forecasts).

In order to proceed, it looks like we will need a way to convert our
wind
speeds and directions into the U and V components.  Is there a MET
utility
that can handle this?  Or do you know of any other utilities that
exist
that will before we create one?  I know how to do this in MOS-2000,
but
that won't help us with GRIB2 or netCDF files.  CAMPS will eventually
help
us, but the CAMPS code that would do this is not ready yet.  WGRIB2
looks
like it will convert from U and V to speed and direction, but not the
opposite.
For what its worth, for the majority of what we verify, the U and V
components will not be available to us.  Only the speed and direction.

Thanks
John

On Fri, Jan 10, 2020 at 11:32 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:

> John,
>
> I did think of one additional complication that'd make the
categorical vx
> of wind direction more complicated.
>
> Presumably, just like with the VCNT line type, you'd want to filter
the
> pairs used based on the wind speed.  I'd assume that the categorical
stats
> for stronger winds would be much better than those for light and
variable
> winds.  But you'd also need to define the thresholds for the
categories,
> such as northerly winds being between 45 and 135 degrees, for
example.
>
> So we might configure it like this:
>
>
>
>
> *fcst = {   wind_thresh = [ NA, >=2, >=5 ];   field = [      { name
=
> "WDIR";*
> *        level = [ "Z10" ];*
>
>
> *        cat_thresh = [ >=45&&<=135 ]; }   ];}*
>
> We are applying 2 threshold steps here... (1) wind_thresh to filter
the U/V
> pairs and (2) cat_thresh to define the categories.
>
> But the output stat header only has column for FCST_THRESH.  For
> categorical output (CTC, CTS, MCTC, and MCTS lines types), we write
the
> cat_thresh to that column.  For vector stats (VL1L2, VAL1L2, and
VCNT line
> types), we write the wind_thresh to that column.
>
> In this case, we'd have 2 thresholds to include in the metadata and
only 1
> column to contain them.  Perhaps we could just concatenate them with
a
> comma, but that'll make the logic for reading them back into STAT-
Analysis
> a bit more tricky.
>
> Just some more details to consider.
>
> Thanks,
> John
>
>
>
>
>
> On Fri, Jan 10, 2020 at 9:17 AM John Halley Gotway <johnhg at ucar.edu>
> wrote:
>
> > John,
> >
> > Yes, the wind_thresh configuration option does apply to the VCNT
output
> > line type.  The MET tarball runs some sample test scripts when you
run
> > "make test" after compiling the code.  I modified the
configuration file
> > for one of those tests to demonstrate:
> >
> > *cd met-8.1.2/scripts*
> > *export TEST_OUT_DIR="../out"*
> > # Run existing command to make sure it's working OK
> > *../bin/grid_stat \*
> > *../data/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212 \*
> > *../data/sample_fcst/2005080712/wrfprs_ruc13_00.tm00_G212 \*
> > *config/GridStatConfig_all \*
> > *-outdir ${TEST_OUT_DIR}/grid_stat -v 2*
> >
> > #  Modify the configuration file (config/GridStatConfig_all) by
> setting...
> > and then rerun the command above:
> >
> >
> >
> >
> >
> >
> >
> > *//// Forecast and observation fields to be verified//fcst = {
> >  wind_thresh = [ NA, NA, NA ];   field = [      { name = "UGRD";
level =
> [
> > "Z10" ]; },*
> >
> >
> >
> >
> > *      { name = "VGRD"; level = [ "Z10" ];  }   ];}obs = fcst;obs
= {
> > wind_thresh = [ NA, >=2, >=5 ]; }*
> >
> > Instead of getting 1 VCNT output line for each masking area,
you'll now
> > get 3: one using all U/V pairs, one using U/V pairs with speed >=2
m/s,
> and
> > one using U/V pairs with speed >=5 m/s.  I configured it so that
we're
> > filtering based on the OBSERVATION wind speed.  But you could do
the
> > forecast instead.  Or both, using the intersection (i.e.
wind_logic =
> > INTERSECTION;).  A threshold of type NA always evaluates to true,
so
> > setting "wind_thresh = [ NA, NA, NA ];" in the fcst dictionary
just means
> > that we're not filtering the fcst U/V vectors.
> >
> > This configuration results in the attached VCNT output.  You'll
see the
> > TOTAL column (# of pairs used) on the first 3 lines decreases from
5558
> to
> > 4999 to 2301 as we increase the wind speed threshold from none to
>=2 to
> > >=5.
> >
> > Currently, if you try to verify wind direction directly in Point-
Stat or
> > Grid-Stat, you'll get this error message:
> > ERROR  : GridStatVxOpt::process_config() -> wind direction may not
be
> > verified using grid_stat.
> >
> > That check is there because the continuous statistics in the CNT
line
> type
> > are not well-suited to circular variables (like wind direction).
> However,
> > you mentioned applying Heidke Skill Score to wind direction, which
is a
> > categorical statistic.  I'd honestly never thought about verifying
wind
> > direction in a categorical way.  That certainly seems doable... as
long
> as
> > care is taken in defining the categorical thresholds well.
> >
> > If you agree, I can write up a development issue for the
verification of
> > wind direction using categorical statistics.  We'll need to make
> Point-Stat
> > and Grid-Stat smart enough to *not* compute continuous stats for
them.
> Do
> > you have a sense for whether or not categorical vx of wind
direction is
> > widely done?  Is this a common practice that should be added to a
> community
> > vx package?
> >
> > Thanks,
> > John
> >
> > On Thu, Jan 9, 2020 at 1:44 PM John L Wagner - NOAA Federal via RT
<
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726 >
> >>
> >> Thanks John.  I did miss the DIR_ERR and DIR_ABSERR on page 200.
Those
> >> will be useful for our MAEs.
> >> I thought I had read that wind_thresh only applied to wind speed
and not
> >> direction.  I will take a closer look at that.  We verify wind
direction
> >> conditioned on forecast and observed winds exceeding 8 knots.
Can I
> pass
> >> grid_stat both the wind direction and speed grids?
> >> We are currently computing a Heidke Skill Score for wind
direction based
> >> on
> >> thresholds that we define.  Our code currently does an adjustment
to
> >> ensure
> >> that all of the north winds fall into the same category.  Is
there
> >> something similar available in MET?  Or would we have to
preprocess the
> >> data to ensure that north isn't split between 0-45 degrees and
335-360
> >> degrees?
> >> Thanks!  Your help is always appreciated.
> >> John
> >>
> >> On Thu, Jan 9, 2020 at 12:20 PM John Halley Gotway via RT <
> >> met_help at ucar.edu>
> >> wrote:
> >>
> >> > Hi John,
> >> >
> >> > We are actively working on the MET version 9.0 release.  We had
> planned
> >> to
> >> > have it out in December 2019 now the end of January 2020 is a
better
> >> > estimate.  We have made two intermediate test releases named
> >> met-9.0_beta1
> >> > and met-9.0_beta2 and those release notes are listed here:
> >> >
> >> >
> >>
> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-9-0-beta1#notes
> >> >
> >> >
> >>
> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
version-9-0-beta2#notes
> >> >
> >> > The remaining development tasks for inclusion in 9.0 are the
"blocker"
> >> > issues from GitHub:
> >> >
> >> >
> >>
>
https://github.com/NCAR/MET/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22MET+9.0%22+label%3A%22priority%3A+blocker%22
> >> >
> >> > However, there are no additional changes included in met-9.0
that
> apply
> >> to
> >> > wind direction.
> >> >
> >> > Hopefully met-8.1 already contains sufficient logic for your
needs.
> The
> >> > Grid-Stat and Point-Stat tools generate VL1L2 and VCNT output
line
> >> types.
> >> > VL1L2 are vector-based partial sums and VCNT are vector-based
> continuous
> >> > statistics.  The "wind_thresh" configuration option for these
tools
> >> enable
> >> > you to filter the u/v pairs which are included in the VL1L2 and
VCNT
> >> sums
> >> > and stats.
> >> >
> >> > Please take a look on page 200 of the met-8.1.2 user's guide:
> >> >
> >> >
> >>
> https://dtcenter.org/sites/default/files/community-
code/met/docs/user-guide/MET_Users_Guide_v8.1.2.pdf
> >> >
> >> > Do these stats looks sufficient?  Or can you please remind me
of
> >> specific
> >> > additional features that are needed?
> >> >
> >> > Thanks,
> >> > John Halley Gotway
> >> >
> >> >
> >> >
> >> > On Thu, Jan 9, 2020 at 9:38 AM John L Wagner - NOAA Federal via
RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726
>
> >> > >
> >> > > Thanks Minna, much appreciated.
> >> > >
> >> > > On Thu, Jan 9, 2020 at 11:36 AM Minna Win via RT
<met_help at ucar.edu
> >
> >> > > wrote:
> >> > >
> >> > > > Hi John,
> >> > > >
> >> > > > It looks like you are interested in doing some wind
direction
> >> > > verification
> >> > > > in MET.  I believe the best person to address this status
is the
> >> > > > engineering lead, John Halley Gotway.  I am going to assign
this
> >> ticket
> >> > > to
> >> > > > him.  Please allow a few business days for a full response.
> >> > > >
> >> > > > Regards,
> >> > > > Minna
> >> > > > ---------------
> >> > > > Minna Win
> >> > > > National Center for Atmospheric Research
> >> > > > Developmental Testbed Center
> >> > > > Phone: 303-497-8423
> >> > > > Fax:   303-497-8401
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Thu, Jan 9, 2020 at 9:10 AM John L Wagner - NOAA Federal
via
> RT <
> >> > > > met_help at ucar.edu> wrote:
> >> > > >
> >> > > > >
> >> > > > > Thu Jan 09 09:10:39 2020: Request 93726 was acted upon.
> >> > > > > Transaction: Ticket created by john.l.wagner at noaa.gov
> >> > > > >        Queue: met_help
> >> > > > >      Subject: Wind direction verification
> >> > > > >        Owner: Nobody
> >> > > > >   Requestors: john.l.wagner at noaa.gov
> >> > > > >       Status: new
> >> > > > >  Ticket <URL:
> >> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=93726
> >> > > >
> >> > > > >
> >> > > > >
> >> > > > > Greetings MET Help
> >> > > > > I was wondering if you had an update of the status of
wind
> >> direction
> >> > > > > verification?  Looking at the slides you have online from
2018,
> it
> >> > > seems
> >> > > > > that wind dir verification is under development.  The
slides
> also
> >> > > suggest
> >> > > > > that the verification would use the U and V components
instead
> of
> >> a
> >> > > wind
> >> > > > > direction field.  I just want to make sure that this info
is
> still
> >> > > > > accurate.  We are currently archiving the URMA wind
direction
> >> field,
> >> > so
> >> > > > if
> >> > > > > we need to change our archive, I want to do that ASAP.
If there
> >> is
> >> > > other
> >> > > > > info on what will be supported or what version it is
expected to
> >> be
> >> > in
> >> > > > that
> >> > > > > I am missing, please let me know.
> >> > > > > Thanks
> >> > > > > John
> >> > > > >
> >> > > > > --
> >> > > > > John Wagner
> >> > > > > Verification Task Lead
> >> > > > > COR Task Manager
> >> > > > > NOAA/National Weather Service
> >> > > > > Meteorological Development Laboratory
> >> > > > > Digital Forecast Services Branch
> >> > > > > SSMC2 Room 10106
> >> > > > > Silver Spring, MD 20910
> >> > > > > (301) 427-9471 (office)
> >> > > > > (908) 902-4155 (cell/text)
> >> > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > >
> >> > > --
> >> > > John Wagner
> >> > > Verification Task Lead
> >> > > COR Task Manager
> >> > > NOAA/National Weather Service
> >> > > Meteorological Development Laboratory
> >> > > Digital Forecast Services Branch
> >> > > SSMC2 Room 10106
> >> > > Silver Spring, MD 20910
> >> > > (301) 427-9471 (office)
> >> > > (908) 902-4155 (cell/text)
> >> > >
> >> > >
> >> >
> >> >
> >>
> >> --
> >> John Wagner
> >> Verification Task Lead
> >> COR Task Manager
> >> NOAA/National Weather Service
> >> Meteorological Development Laboratory
> >> Digital Forecast Services Branch
> >> SSMC2 Room 10106
> >> Silver Spring, MD 20910
> >> (301) 427-9471 (office)
> >> (908) 902-4155 (cell/text)
> >>
> >>
>
>

--
John Wagner
Verification Task Lead
COR Task Manager
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Branch
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)

------------------------------------------------
Subject: Wind direction verification
From: John Halley Gotway
Time: Wed Apr 01 11:01:28 2020

Hi John,

I'm just cleaning up some old tickets in MET-Help and ran across this
one.  I wanted to let you know that I did use the info you sent to
create a new development issue for
MET:https://github.com/NCAR/MET/issues/1236

Thanks,
John Halley Gotway

------------------------------------------------


More information about the Met_help mailing list