[Met_help] [rt.rap.ucar.edu #67570] History for Questions on Neighborhood Method (UNCLASSIFIED)

Tue Jun 24 13:29:09 MDT 2014

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Classification: UNCLASSIFIED
Caveats: NONE

I am trying to understand the application of the MET Grid-Stat neighborhood
method for the calculation of neighborhood continuous statistics for scalar
variables such as 2m AGL temperature or 2m AGL relative humidity. I captured
the following paragraph from the MET V4.1 User's Guide. I then tried to
express in the paragraph below that, my interpretation of the User's Guide
explanation. I also have the following questions which arose as I tried to
interpret the User's Guide:

Is the scoring considered a "hit" based on the continuous statistics error
calculations (i.e. forecast-observation pair errors) or based on a whether
the category threshold condition was met?

Do the values of individual forecast grid points get assigned a new value
based on the values of all the forecast grid points within the neighborhood
before the gridpoint to gridpoint scoring is performed so as to generate
continuous statistics? What about the situation where the observation grid
points are assigned new values as well based on the application of the same
neighborhood?

"MET also incorporates several neighborhood methods to give credit to
forecasts that are close to the observations, but not necessarily exactly
matched up in
space. Also referred to as "fuzzy" verification methods, these methods do
not just
compare a single forecast at each grid point to a single observation at each
grid point; they
compare the forecasts and observations in a neighborhood surrounding the
point of
interest. With the neighborhood method, the user chooses a distance within
which the
forecast event can fall from the observed event and still be considered a
hit. In MET this
is implemented by defining a square search window around each grid point.
Within the search window, the number of observed events is compared to the
number of
Forecast events. In this way, credit is given to forecasts that are close to
the
Observations without requiring a strict match between forecasted events and
observed
events at any particular grid point. The neighborhood methods allow the user
to see how
forecast skill varies with neighborhood size and can help determine the
smallest
neighborhood size that can be used to give sufficiently accurate forecasts."

My interpretation: The "neighborhood" can be applied to both the forecast
field and the observed field as well as just one or the other. Within the
neighborhood, a number of observed values will be compared to the same
number of forecast values and the scoring is based on those comparisons
rather than a single score derived from the difference calculated at a
particular grid point. Instead of requiring that a "hit" be defined by the
forecast-observation difference of 0 at each grid point, the use of a
neighborhood for the observed field allows you to define a "hit" if the
forecast value falls within a range of the observed values included in the
neighborhood. Further, if the same neighborhood also applies to the forecast
field, then, that too, provides a range of forecast values which can be
considered "hits' when compared to the range of observed values.

Thanks.

R/
John

Mr John W. Raby, Meteorologist
U.S. Army Research Laboratory
White Sands Missile Range, NM 88002
(575) 678-2004 DSN 258-2004
FAX (575) 678-1230 DSN 258-1230
Email: john.w.raby2.civ at mail.mil

Classification: UNCLASSIFIED
Caveats: NONE

----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Questions on Neighborhood Method (UNCLASSIFIED)
From: John Halley Gotway
Time: Wed Jun 11 13:04:48 2014

John,

I read through your email and see that you have several questions
about how
the neighborhood methods are implemented in the Grid-Stat tool.
Rather
than addressing each individual point in your email, let me first try
to
lay out the steps that are applied in Grid-Stat.  After reading this,
please let me know if you have further questions.

(1) First, neighborhood methods are currently only available in
grid_stat
since they require both the forecast and observation to be gridded
(and on
the same grid).  There are neighborhood methods defined for point
observations, but those are not yet available in MET.

(2) The Fractions Skill Score (FSS) and Fractions Brier Score (FBS)
are the
two most popular neighborhood methods statistics, and they are
contained in
the NBRCNT (Neighborhood Continuous Statistics) line type.

(3) FSS and FBS are computed using two main parameters: a threshold
(cat_thresh in the config file) and a neighborhood size ("nbr.width"
in the
config file is the "width" of a square box, e.g. width = 3 means a 3x3
box
containing 9 points).  The neighborhood width must be odd so that the
box
can be centered on each grid point.  A NBRCNT output line will be
generated
for each combination of threshold and neighborhood width.

(4) The first step is to apply one of the thresholds from the
"cat_thresh"
list to both the forecast and observation fields.  These steps are
applied
equally to both fields - not one or the other.  This replaces the raw
fields with fields of 0's and 1's... 1 where the threshold criteria is
met
and 0 otherwise.

(5) Next, for each grid point in the forecast field, draw an nxn box
around
it (where n is the neighborhood width).  Count up the number of 1's
inside
that box, and compute a ratio of the number of 1's divided by the size
of
the box (nxn) to get a number between 0 and 1.  We call this number a
"fractional coverage" value for that grid point.  For example, if the
event
occurred at 4 of the 9 grid points, the fractional coverage value
would be
4/9 = 0.44.  Doing this for every grid point in the forecast field
generates a forecast "fractional coverage" field.

As a small side note, for grid points near the edge of the grid or
near
missing data values, not all of the nxn points in the neighborhood
will
contain valid data.  The "nbr.vld_thresh" config file setting
specifies
what percentage of grid points must contain valid data for a
fractional
coverage value to be computed.  A value of 1.0 (the default) means
that all
nxn points must valid data for fractional coverage to be computed at
the
point.

(6) Apply the same logic listed in (5) to the observation field to
compute
an observation "fractional coverage" field.

(7) At each grid point, we now have a forecast fractional coverage
value
between 0 and 1 and an observation fractional coverage value between 0
and
1.

(8) The Fractions Skill Score and Fractions Brier Score are computed
using
these fractional coverage values directly.  They are written out in
the
NBRCNT line type.

(9) MET also contains NBRCTC and NBRCTS line types, which are not
commonly
used.  These are computed by applying a threshold ("nbrhd.cov_thresh"
in
the config file) to those fractional coverage fields.  Applying a
threshold
converts those fractional coverage fields back into fields of 0's and
1's.
A 2x2 contingency table is computed over those thresholded fields.
The
contingency table counts are stored in the NBRCTC line type, and the
corresponding statistics are stored in the NBRCTS line type.  But as I
said, these are not commonly used.

As I mentioned, the FSS and FBS are the most commonly used
neighborhood
methods statistics.  For a given categorical threshold, you'll find
that as
you increase the neighborhood size, the FSS will increase as well.
FSS is
often used to determine the scale at which a forecast is skillful.
For
example, a 4km model run will contain a lot of great detail at the
finest
scales, but likely won't be good at getting that detail exactly right.
For
a particular threshold of interest, you might compute FSS for several
neighborhood sizes, 3x3, 5x5, 7x7, and so on.  Suppose you see that
the
model becomes skillful at the 5x5 neighborhood size.  So even though
the
model has great detail at 4km, it's much more skill full at 20km
resolution.  If you were making forecasts to the public, you might
choose
to upscale your model output to 20km prior to distributing it to the
public
- otherwise, you'd be implying more confidence in the details of the
model
than you actually have.

Sorry for the long email!  Hopefully, that helps explain it.  Just let
us
know if you have more questions.

Thanks,
John

On Tue, Jun 10, 2014 at 11:26 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> Tue Jun 10 11:26:28 2014: Request 67570 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>        Queue: met_help
>      Subject: Questions on Neighborhood Method (UNCLASSIFIED)
>        Owner: Nobody
>   Requestors: john.w.raby2.civ at mail.mil
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> I am trying to understand the application of the MET Grid-Stat
neighborhood
> method for the calculation of neighborhood continuous statistics for
scalar
> variables such as 2m AGL temperature or 2m AGL relative humidity. I
> captured
> the following paragraph from the MET V4.1 User's Guide. I then tried
to
> express in the paragraph below that, my interpretation of the User's
Guide
> explanation. I also have the following questions which arose as I
tried to
> interpret the User's Guide:
>
> Is the scoring considered a "hit" based on the continuous statistics
error
> calculations (i.e. forecast-observation pair errors) or based on a
whether
> the category threshold condition was met?
>
> Do the values of individual forecast grid points get assigned a new
value
> based on the values of all the forecast grid points within the
neighborhood
> before the gridpoint to gridpoint scoring is performed so as to
generate
> continuous statistics? What about the situation where the
observation grid
> points are assigned new values as well based on the application of
the same
> neighborhood?
>
> "MET also incorporates several neighborhood methods to give credit
to
> forecasts that are close to the observations, but not necessarily
exactly
> matched up in
> space. Also referred to as "fuzzy" verification methods, these
methods do
> not just
> compare a single forecast at each grid point to a single observation
at
> each
> grid point; they
> compare the forecasts and observations in a neighborhood surrounding
the
> point of
> interest. With the neighborhood method, the user chooses a distance
within
> which the
> forecast event can fall from the observed event and still be
considered a
> hit. In MET this
> is implemented by defining a square search window around each grid
point.
> Within the search window, the number of observed events is compared
to the
> number of
> Forecast events. In this way, credit is given to forecasts that are
close
> to
> the
> Observations without requiring a strict match between forecasted
events and
> observed
> events at any particular grid point. The neighborhood methods allow
the
> user
> to see how
> forecast skill varies with neighborhood size and can help determine
the
> smallest
> neighborhood size that can be used to give sufficiently accurate
> forecasts."
>
> My interpretation: The "neighborhood" can be applied to both the
forecast
> field and the observed field as well as just one or the other.
Within the
> neighborhood, a number of observed values will be compared to the
same
> number of forecast values and the scoring is based on those
comparisons
> rather than a single score derived from the difference calculated at
a
> particular grid point. Instead of requiring that a "hit" be defined
by the
> forecast-observation difference of 0 at each grid point, the use of
a
> neighborhood for the observed field allows you to define a "hit" if
the
> forecast value falls within a range of the observed values included
in the
> neighborhood. Further, if the same neighborhood also applies to the
> forecast
> field, then, that too, provides a range of forecast values which can
be
> considered "hits' when compared to the range of observed values.
>
> Thanks.
>
> R/
> John
>
> Mr John W. Raby, Meteorologist
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> (575) 678-2004 DSN 258-2004
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
>
>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #67570] Questions on Neighborhood Method (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed Jun 11 13:37:53 2014

John -

Having your detailed explanation of how the method works is very
helpful. It allows you to slowly understand it by putting each piece
together in the right order.

So, my understanding now leads me to answer my questions as follows:

Q:Is the scoring considered a "hit" based on the continuous statistics
error
> calculations (i.e. forecast-observation pair errors) or based on a
whether
> the category threshold condition was met?
A: Scoring is based on categorical forecasts and categorical
observations. (0-miss and 1-hit where a hit is defined as an
occurrence when the forecast (observation) meets the threshold
criteria.

Q: Do the values of individual forecast grid points get assigned a new
value
> based on the values of all the forecast grid points within the
neighborhood
> before the gridpoint to gridpoint scoring is performed so as to
generate
> continuous statistics?
A: No re-assigining of forecast grid point values.

Q: What about the situation where the observation grid
> points are assigned new values as well based on the application of
the same
> neighborhood?
A:  No re-assigining of observation grid point values.

Thanks for taking the time to explain the process.

R/
John

________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Wednesday, June 11, 2014 1:04 PM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #67570] Questions on Neighborhood Method
(UNCLASSIFIED)

John,