[Met_help] [rt.rap.ucar.edu #67570] History for Questions on Neighborhood Method (UNCLASSIFIED)

John Halley Gotway via RT met_help at ucar.edu
Tue Jun 24 13:29:09 MDT 2014


----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Classification: UNCLASSIFIED
Caveats: NONE

I am trying to understand the application of the MET Grid-Stat neighborhood
method for the calculation of neighborhood continuous statistics for scalar
variables such as 2m AGL temperature or 2m AGL relative humidity. I captured
the following paragraph from the MET V4.1 User's Guide. I then tried to
express in the paragraph below that, my interpretation of the User's Guide
explanation. I also have the following questions which arose as I tried to
interpret the User's Guide:

Is the scoring considered a "hit" based on the continuous statistics error
calculations (i.e. forecast-observation pair errors) or based on a whether
the category threshold condition was met?

Do the values of individual forecast grid points get assigned a new value
based on the values of all the forecast grid points within the neighborhood
before the gridpoint to gridpoint scoring is performed so as to generate
continuous statistics? What about the situation where the observation grid
points are assigned new values as well based on the application of the same
neighborhood?

"MET also incorporates several neighborhood methods to give credit to
forecasts that are close to the observations, but not necessarily exactly
matched up in
space. Also referred to as "fuzzy" verification methods, these methods do
not just
compare a single forecast at each grid point to a single observation at each
grid point; they
compare the forecasts and observations in a neighborhood surrounding the
point of
interest. With the neighborhood method, the user chooses a distance within
which the
forecast event can fall from the observed event and still be considered a
hit. In MET this
is implemented by defining a square search window around each grid point.
Within the search window, the number of observed events is compared to the
number of
Forecast events. In this way, credit is given to forecasts that are close to
the
Observations without requiring a strict match between forecasted events and
observed
events at any particular grid point. The neighborhood methods allow the user
to see how
forecast skill varies with neighborhood size and can help determine the
smallest
neighborhood size that can be used to give sufficiently accurate forecasts."

My interpretation: The "neighborhood" can be applied to both the forecast
field and the observed field as well as just one or the other. Within the
neighborhood, a number of observed values will be compared to the same
number of forecast values and the scoring is based on those comparisons
rather than a single score derived from the difference calculated at a
particular grid point. Instead of requiring that a "hit" be defined by the
forecast-observation difference of 0 at each grid point, the use of a
neighborhood for the observed field allows you to define a "hit" if the
forecast value falls within a range of the observed values included in the
neighborhood. Further, if the same neighborhood also applies to the forecast
field, then, that too, provides a range of forecast values which can be
considered "hits' when compared to the range of observed values.

Thanks.

R/
John

Mr John W. Raby, Meteorologist
U.S. Army Research Laboratory
White Sands Missile Range, NM 88002
(575) 678-2004 DSN 258-2004
FAX (575) 678-1230 DSN 258-1230
Email: john.w.raby2.civ at mail.mil





Classification: UNCLASSIFIED
Caveats: NONE




----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Questions on Neighborhood Method (UNCLASSIFIED)
From: John Halley Gotway
Time: Wed Jun 11 13:04:48 2014

John,

I read through your email and see that you have several questions
about how
the neighborhood methods are implemented in the Grid-Stat tool.
Rather
than addressing each individual point in your email, let me first try
to
lay out the steps that are applied in Grid-Stat.  After reading this,
please let me know if you have further questions.

(1) First, neighborhood methods are currently only available in
grid_stat
since they require both the forecast and observation to be gridded
(and on
the same grid).  There are neighborhood methods defined for point
observations, but those are not yet available in MET.

(2) The Fractions Skill Score (FSS) and Fractions Brier Score (FBS)
are the
two most popular neighborhood methods statistics, and they are
contained in
the NBRCNT (Neighborhood Continuous Statistics) line type.

(3) FSS and FBS are computed using two main parameters: a threshold
(cat_thresh in the config file) and a neighborhood size ("nbr.width"
in the
config file is the "width" of a square box, e.g. width = 3 means a 3x3
box
containing 9 points).  The neighborhood width must be odd so that the
box
can be centered on each grid point.  A NBRCNT output line will be
generated
for each combination of threshold and neighborhood width.

(4) The first step is to apply one of the thresholds from the
"cat_thresh"
list to both the forecast and observation fields.  These steps are
applied
equally to both fields - not one or the other.  This replaces the raw
fields with fields of 0's and 1's... 1 where the threshold criteria is
met
and 0 otherwise.

(5) Next, for each grid point in the forecast field, draw an nxn box
around
it (where n is the neighborhood width).  Count up the number of 1's
inside
that box, and compute a ratio of the number of 1's divided by the size
of
the box (nxn) to get a number between 0 and 1.  We call this number a
"fractional coverage" value for that grid point.  For example, if the
event
occurred at 4 of the 9 grid points, the fractional coverage value
would be
4/9 = 0.44.  Doing this for every grid point in the forecast field
generates a forecast "fractional coverage" field.

As a small side note, for grid points near the edge of the grid or
near
missing data values, not all of the nxn points in the neighborhood
will
contain valid data.  The "nbr.vld_thresh" config file setting
specifies
what percentage of grid points must contain valid data for a
fractional
coverage value to be computed.  A value of 1.0 (the default) means
that all
nxn points must valid data for fractional coverage to be computed at
the
point.

(6) Apply the same logic listed in (5) to the observation field to
compute
an observation "fractional coverage" field.

(7) At each grid point, we now have a forecast fractional coverage
value
between 0 and 1 and an observation fractional coverage value between 0
and
1.

(8) The Fractions Skill Score and Fractions Brier Score are computed
using
these fractional coverage values directly.  They are written out in
the
NBRCNT line type.

(9) MET also contains NBRCTC and NBRCTS line types, which are not
commonly
used.  These are computed by applying a threshold ("nbrhd.cov_thresh"
in
the config file) to those fractional coverage fields.  Applying a
threshold
converts those fractional coverage fields back into fields of 0's and
1's.
A 2x2 contingency table is computed over those thresholded fields.
The
contingency table counts are stored in the NBRCTC line type, and the
corresponding statistics are stored in the NBRCTS line type.  But as I
said, these are not commonly used.

As I mentioned, the FSS and FBS are the most commonly used
neighborhood
methods statistics.  For a given categorical threshold, you'll find
that as
you increase the neighborhood size, the FSS will increase as well.
FSS is
often used to determine the scale at which a forecast is skillful.
For
example, a 4km model run will contain a lot of great detail at the
finest
scales, but likely won't be good at getting that detail exactly right.
For
a particular threshold of interest, you might compute FSS for several
neighborhood sizes, 3x3, 5x5, 7x7, and so on.  Suppose you see that
the
model becomes skillful at the 5x5 neighborhood size.  So even though
the
model has great detail at 4km, it's much more skill full at 20km
resolution.  If you were making forecasts to the public, you might
choose
to upscale your model output to 20km prior to distributing it to the
public
- otherwise, you'd be implying more confidence in the details of the
model
than you actually have.

Sorry for the long email!  Hopefully, that helps explain it.  Just let
us
know if you have more questions.

Thanks,
John



On Tue, Jun 10, 2014 at 11:26 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> Tue Jun 10 11:26:28 2014: Request 67570 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>        Queue: met_help
>      Subject: Questions on Neighborhood Method (UNCLASSIFIED)
>        Owner: Nobody
>   Requestors: john.w.raby2.civ at mail.mil
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> I am trying to understand the application of the MET Grid-Stat
neighborhood
> method for the calculation of neighborhood continuous statistics for
scalar
> variables such as 2m AGL temperature or 2m AGL relative humidity. I
> captured
> the following paragraph from the MET V4.1 User's Guide. I then tried
to
> express in the paragraph below that, my interpretation of the User's
Guide
> explanation. I also have the following questions which arose as I
tried to
> interpret the User's Guide:
>
> Is the scoring considered a "hit" based on the continuous statistics
error
> calculations (i.e. forecast-observation pair errors) or based on a
whether
> the category threshold condition was met?
>
> Do the values of individual forecast grid points get assigned a new
value
> based on the values of all the forecast grid points within the
neighborhood
> before the gridpoint to gridpoint scoring is performed so as to
generate
> continuous statistics? What about the situation where the
observation grid
> points are assigned new values as well based on the application of
the same
> neighborhood?
>
> "MET also incorporates several neighborhood methods to give credit
to
> forecasts that are close to the observations, but not necessarily
exactly
> matched up in
> space. Also referred to as "fuzzy" verification methods, these
methods do
> not just
> compare a single forecast at each grid point to a single observation
at
> each
> grid point; they
> compare the forecasts and observations in a neighborhood surrounding
the
> point of
> interest. With the neighborhood method, the user chooses a distance
within
> which the
> forecast event can fall from the observed event and still be
considered a
> hit. In MET this
> is implemented by defining a square search window around each grid
point.
> Within the search window, the number of observed events is compared
to the
> number of
> Forecast events. In this way, credit is given to forecasts that are
close
> to
> the
> Observations without requiring a strict match between forecasted
events and
> observed
> events at any particular grid point. The neighborhood methods allow
the
> user
> to see how
> forecast skill varies with neighborhood size and can help determine
the
> smallest
> neighborhood size that can be used to give sufficiently accurate
> forecasts."
>
> My interpretation: The "neighborhood" can be applied to both the
forecast
> field and the observed field as well as just one or the other.
Within the
> neighborhood, a number of observed values will be compared to the
same
> number of forecast values and the scoring is based on those
comparisons
> rather than a single score derived from the difference calculated at
a
> particular grid point. Instead of requiring that a "hit" be defined
by the
> forecast-observation difference of 0 at each grid point, the use of
a
> neighborhood for the observed field allows you to define a "hit" if
the
> forecast value falls within a range of the observed values included
in the
> neighborhood. Further, if the same neighborhood also applies to the
> forecast
> field, then, that too, provides a range of forecast values which can
be
> considered "hits' when compared to the range of observed values.
>
> Thanks.
>
> R/
> John
>
> Mr John W. Raby, Meteorologist
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> (575) 678-2004 DSN 258-2004
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
>
>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #67570] Questions on Neighborhood Method (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed Jun 11 13:37:53 2014

John -

Having your detailed explanation of how the method works is very
helpful. It allows you to slowly understand it by putting each piece
together in the right order.

So, my understanding now leads me to answer my questions as follows:

Q:Is the scoring considered a "hit" based on the continuous statistics
error
> calculations (i.e. forecast-observation pair errors) or based on a
whether
> the category threshold condition was met?
A: Scoring is based on categorical forecasts and categorical
observations. (0-miss and 1-hit where a hit is defined as an
occurrence when the forecast (observation) meets the threshold
criteria.

Q: Do the values of individual forecast grid points get assigned a new
value
> based on the values of all the forecast grid points within the
neighborhood
> before the gridpoint to gridpoint scoring is performed so as to
generate
> continuous statistics?
A: No re-assigining of forecast grid point values.

Q: What about the situation where the observation grid
> points are assigned new values as well based on the application of
the same
> neighborhood?
A:  No re-assigining of observation grid point values.

Thanks for taking the time to explain the process.

R/
John

________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Wednesday, June 11, 2014 1:04 PM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #67570] Questions on Neighborhood Method
(UNCLASSIFIED)

John,

I read through your email and see that you have several questions
about how
the neighborhood methods are implemented in the Grid-Stat tool.
Rather
than addressing each individual point in your email, let me first try
to
lay out the steps that are applied in Grid-Stat.  After reading this,
please let me know if you have further questions.

(1) First, neighborhood methods are currently only available in
grid_stat
since they require both the forecast and observation to be gridded
(and on
the same grid).  There are neighborhood methods defined for point
observations, but those are not yet available in MET.

(2) The Fractions Skill Score (FSS) and Fractions Brier Score (FBS)
are the
two most popular neighborhood methods statistics, and they are
contained in
the NBRCNT (Neighborhood Continuous Statistics) line type.

(3) FSS and FBS are computed using two main parameters: a threshold
(cat_thresh in the config file) and a neighborhood size ("nbr.width"
in the
config file is the "width" of a square box, e.g. width = 3 means a 3x3
box
containing 9 points).  The neighborhood width must be odd so that the
box
can be centered on each grid point.  A NBRCNT output line will be
generated
for each combination of threshold and neighborhood width.

(4) The first step is to apply one of the thresholds from the
"cat_thresh"
list to both the forecast and observation fields.  These steps are
applied
equally to both fields - not one or the other.  This replaces the raw
fields with fields of 0's and 1's... 1 where the threshold criteria is
met
and 0 otherwise.

(5) Next, for each grid point in the forecast field, draw an nxn box
around
it (where n is the neighborhood width).  Count up the number of 1's
inside
that box, and compute a ratio of the number of 1's divided by the size
of
the box (nxn) to get a number between 0 and 1.  We call this number a
"fractional coverage" value for that grid point.  For example, if the
event
occurred at 4 of the 9 grid points, the fractional coverage value
would be
4/9 = 0.44.  Doing this for every grid point in the forecast field
generates a forecast "fractional coverage" field.

As a small side note, for grid points near the edge of the grid or
near
missing data values, not all of the nxn points in the neighborhood
will
contain valid data.  The "nbr.vld_thresh" config file setting
specifies
what percentage of grid points must contain valid data for a
fractional
coverage value to be computed.  A value of 1.0 (the default) means
that all
nxn points must valid data for fractional coverage to be computed at
the
point.

(6) Apply the same logic listed in (5) to the observation field to
compute
an observation "fractional coverage" field.

(7) At each grid point, we now have a forecast fractional coverage
value
between 0 and 1 and an observation fractional coverage value between 0
and
1.

(8) The Fractions Skill Score and Fractions Brier Score are computed
using
these fractional coverage values directly.  They are written out in
the
NBRCNT line type.

(9) MET also contains NBRCTC and NBRCTS line types, which are not
commonly
used.  These are computed by applying a threshold ("nbrhd.cov_thresh"
in
the config file) to those fractional coverage fields.  Applying a
threshold
converts those fractional coverage fields back into fields of 0's and
1's.
A 2x2 contingency table is computed over those thresholded fields.
The
contingency table counts are stored in the NBRCTC line type, and the
corresponding statistics are stored in the NBRCTS line type.  But as I
said, these are not commonly used.

As I mentioned, the FSS and FBS are the most commonly used
neighborhood
methods statistics.  For a given categorical threshold, you'll find
that as
you increase the neighborhood size, the FSS will increase as well.
FSS is
often used to determine the scale at which a forecast is skillful.
For
example, a 4km model run will contain a lot of great detail at the
finest
scales, but likely won't be good at getting that detail exactly right.
For
a particular threshold of interest, you might compute FSS for several
neighborhood sizes, 3x3, 5x5, 7x7, and so on.  Suppose you see that
the
model becomes skillful at the 5x5 neighborhood size.  So even though
the
model has great detail at 4km, it's much more skill full at 20km
resolution.  If you were making forecasts to the public, you might
choose
to upscale your model output to 20km prior to distributing it to the
public
- otherwise, you'd be implying more confidence in the details of the
model
than you actually have.

Sorry for the long email!  Hopefully, that helps explain it.  Just let
us
know if you have more questions.

Thanks,
John



On Tue, Jun 10, 2014 at 11:26 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> Tue Jun 10 11:26:28 2014: Request 67570 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
>        Queue: met_help
>      Subject: Questions on Neighborhood Method (UNCLASSIFIED)
>        Owner: Nobody
>   Requestors: john.w.raby2.civ at mail.mil
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> I am trying to understand the application of the MET Grid-Stat
neighborhood
> method for the calculation of neighborhood continuous statistics for
scalar
> variables such as 2m AGL temperature or 2m AGL relative humidity. I
> captured
> the following paragraph from the MET V4.1 User's Guide. I then tried
to
> express in the paragraph below that, my interpretation of the User's
Guide
> explanation. I also have the following questions which arose as I
tried to
> interpret the User's Guide:
>
> Is the scoring considered a "hit" based on the continuous statistics
error
> calculations (i.e. forecast-observation pair errors) or based on a
whether
> the category threshold condition was met?
>
> Do the values of individual forecast grid points get assigned a new
value
> based on the values of all the forecast grid points within the
neighborhood
> before the gridpoint to gridpoint scoring is performed so as to
generate
> continuous statistics? What about the situation where the
observation grid
> points are assigned new values as well based on the application of
the same
> neighborhood?
>
> "MET also incorporates several neighborhood methods to give credit
to
> forecasts that are close to the observations, but not necessarily
exactly
> matched up in
> space. Also referred to as "fuzzy" verification methods, these
methods do
> not just
> compare a single forecast at each grid point to a single observation
at
> each
> grid point; they
> compare the forecasts and observations in a neighborhood surrounding
the
> point of
> interest. With the neighborhood method, the user chooses a distance
within
> which the
> forecast event can fall from the observed event and still be
considered a
> hit. In MET this
> is implemented by defining a square search window around each grid
point.
> Within the search window, the number of observed events is compared
to the
> number of
> Forecast events. In this way, credit is given to forecasts that are
close
> to
> the
> Observations without requiring a strict match between forecasted
events and
> observed
> events at any particular grid point. The neighborhood methods allow
the
> user
> to see how
> forecast skill varies with neighborhood size and can help determine
the
> smallest
> neighborhood size that can be used to give sufficiently accurate
> forecasts."
>
> My interpretation: The "neighborhood" can be applied to both the
forecast
> field and the observed field as well as just one or the other.
Within the
> neighborhood, a number of observed values will be compared to the
same
> number of forecast values and the scoring is based on those
comparisons
> rather than a single score derived from the difference calculated at
a
> particular grid point. Instead of requiring that a "hit" be defined
by the
> forecast-observation difference of 0 at each grid point, the use of
a
> neighborhood for the observed field allows you to define a "hit" if
the
> forecast value falls within a range of the observed values included
in the
> neighborhood. Further, if the same neighborhood also applies to the
> forecast
> field, then, that too, provides a range of forecast values which can
be
> considered "hits' when compared to the range of observed values.
>
> Thanks.
>
> R/
> John
>
> Mr John W. Raby, Meteorologist
> U.S. Army Research Laboratory
> White Sands Missile Range, NM 88002
> (575) 678-2004 DSN 258-2004
> FAX (575) 678-1230 DSN 258-1230
> Email: john.w.raby2.civ at mail.mil
>
>
>
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>



------------------------------------------------
Subject: Questions on Neighborhood Method (UNCLASSIFIED)
From: John Halley Gotway
Time: Wed Jun 11 16:00:52 2014

John,

For your first question on "hits", your answer looks correct.

For your other two questions about whether the grid point gets
assigned a
new value, I would say the answer is yes.  When applying the
neighborhood
methods, the value at each forecast (and observation) grid point gets
replaced by the fractional coverage within the neighborhood around
that
point.  The FSS and FBS statistics are computed over those fractional
coverage values, not the raw forecast (and observation) values.  So
the
stats in the NBRCNT, NBRCTC, and NBRCTS line types are computed over
the
fractional coverage fields (derived from the raw forecast and
observation
fields).

The stats in all the other line types (like CNT, CTC, and CTS) are
computed
over the raw forecast and observation fields directly.

I think you may be confusing two different sections of the config
file:
"nbrhd" vs "interp".

The nbrhd section controls the logic when computing NBRCNT, NBRCTC,
and
NBRCTS output line types.

The interp section is separate and does not interact with the nbrhd
section.  It enables you to smooth either the forecast or observation
field
or both.  By default, its set up to do no smoothing.  Suppose for
example,
you setup the interp section like this:
interp = {
   field      = BOTH;
   vld_thresh = 1.0;

   type = [
      {
         method = UW_MEAN;
         width  = 5;
      }
   ];
};

The would apply a 5x5 smoothing box to each grid point.  In this case,
the
raw forecast value would be replaced by the average forecast value
using
the 25 closest points.  Then the output in the CNT, CTC, and CTS line
types
would be computed over those smoothed values.

So smoothing using the "interp" settings and neighborhood methods
using the
"nbrhd" settings are both available, but are different.

Hope that helps.

Thanks,
John



On Wed, Jun 11, 2014 at 1:37 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
>
> John -
>
> Having your detailed explanation of how the method works is very
helpful.
> It allows you to slowly understand it by putting each piece together
in the
> right order.
>
> So, my understanding now leads me to answer my questions as follows:
>
> Q:Is the scoring considered a "hit" based on the continuous
statistics
> error
> > calculations (i.e. forecast-observation pair errors) or based on a
> whether
> > the category threshold condition was met?
> A: Scoring is based on categorical forecasts and categorical
observations.
> (0-miss and 1-hit where a hit is defined as an occurrence when the
forecast
> (observation) meets the threshold criteria.
>
> Q: Do the values of individual forecast grid points get assigned a
new
> value
> > based on the values of all the forecast grid points within the
> neighborhood
> > before the gridpoint to gridpoint scoring is performed so as to
generate
> > continuous statistics?
> A: No re-assigining of forecast grid point values.
>
> Q: What about the situation where the observation grid
> > points are assigned new values as well based on the application of
the
> same
> > neighborhood?
> A:  No re-assigining of observation grid point values.
>
> Thanks for taking the time to explain the process.
>
> R/
> John
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Wednesday, June 11, 2014 1:04 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #67570] Questions on Neighborhood
Method
> (UNCLASSIFIED)
>
> John,
>
> I read through your email and see that you have several questions
about how
> the neighborhood methods are implemented in the Grid-Stat tool.
Rather
> than addressing each individual point in your email, let me first
try to
> lay out the steps that are applied in Grid-Stat.  After reading
this,
> please let me know if you have further questions.
>
> (1) First, neighborhood methods are currently only available in
grid_stat
> since they require both the forecast and observation to be gridded
(and on
> the same grid).  There are neighborhood methods defined for point
> observations, but those are not yet available in MET.
>
> (2) The Fractions Skill Score (FSS) and Fractions Brier Score (FBS)
are the
> two most popular neighborhood methods statistics, and they are
contained in
> the NBRCNT (Neighborhood Continuous Statistics) line type.
>
> (3) FSS and FBS are computed using two main parameters: a threshold
> (cat_thresh in the config file) and a neighborhood size ("nbr.width"
in the
> config file is the "width" of a square box, e.g. width = 3 means a
3x3 box
> containing 9 points).  The neighborhood width must be odd so that
the box
> can be centered on each grid point.  A NBRCNT output line will be
generated
> for each combination of threshold and neighborhood width.
>
> (4) The first step is to apply one of the thresholds from the
"cat_thresh"
> list to both the forecast and observation fields.  These steps are
applied
> equally to both fields - not one or the other.  This replaces the
raw
> fields with fields of 0's and 1's... 1 where the threshold criteria
is met
> and 0 otherwise.
>
> (5) Next, for each grid point in the forecast field, draw an nxn box
around
> it (where n is the neighborhood width).  Count up the number of 1's
inside
> that box, and compute a ratio of the number of 1's divided by the
size of
> the box (nxn) to get a number between 0 and 1.  We call this number
a
> "fractional coverage" value for that grid point.  For example, if
the event
> occurred at 4 of the 9 grid points, the fractional coverage value
would be
> 4/9 = 0.44.  Doing this for every grid point in the forecast field
> generates a forecast "fractional coverage" field.
>
> As a small side note, for grid points near the edge of the grid or
near
> missing data values, not all of the nxn points in the neighborhood
will
> contain valid data.  The "nbr.vld_thresh" config file setting
specifies
> what percentage of grid points must contain valid data for a
fractional
> coverage value to be computed.  A value of 1.0 (the default) means
that all
> nxn points must valid data for fractional coverage to be computed at
the
> point.
>
> (6) Apply the same logic listed in (5) to the observation field to
compute
> an observation "fractional coverage" field.
>
> (7) At each grid point, we now have a forecast fractional coverage
value
> between 0 and 1 and an observation fractional coverage value between
0 and
> 1.
>
> (8) The Fractions Skill Score and Fractions Brier Score are computed
using
> these fractional coverage values directly.  They are written out in
the
> NBRCNT line type.
>
> (9) MET also contains NBRCTC and NBRCTS line types, which are not
commonly
> used.  These are computed by applying a threshold
("nbrhd.cov_thresh" in
> the config file) to those fractional coverage fields.  Applying a
threshold
> converts those fractional coverage fields back into fields of 0's
and 1's.
> A 2x2 contingency table is computed over those thresholded fields.
The
> contingency table counts are stored in the NBRCTC line type, and the
> corresponding statistics are stored in the NBRCTS line type.  But as
I
> said, these are not commonly used.
>
> As I mentioned, the FSS and FBS are the most commonly used
neighborhood
> methods statistics.  For a given categorical threshold, you'll find
that as
> you increase the neighborhood size, the FSS will increase as well.
FSS is
> often used to determine the scale at which a forecast is skillful.
For
> example, a 4km model run will contain a lot of great detail at the
finest
> scales, but likely won't be good at getting that detail exactly
right.  For
> a particular threshold of interest, you might compute FSS for
several
> neighborhood sizes, 3x3, 5x5, 7x7, and so on.  Suppose you see that
the
> model becomes skillful at the 5x5 neighborhood size.  So even though
the
> model has great detail at 4km, it's much more skill full at 20km
> resolution.  If you were making forecasts to the public, you might
choose
> to upscale your model output to 20km prior to distributing it to the
public
> - otherwise, you'd be implying more confidence in the details of the
model
> than you actually have.
>
> Sorry for the long email!  Hopefully, that helps explain it.  Just
let us
> know if you have more questions.
>
> Thanks,
> John
>
>
>
> On Tue, Jun 10, 2014 at 11:26 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Jun 10 11:26:28 2014: Request 67570 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> >        Queue: met_help
> >      Subject: Questions on Neighborhood Method (UNCLASSIFIED)
> >        Owner: Nobody
> >   Requestors: john.w.raby2.civ at mail.mil
> >       Status: new
> >  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > I am trying to understand the application of the MET Grid-Stat
> neighborhood
> > method for the calculation of neighborhood continuous statistics
for
> scalar
> > variables such as 2m AGL temperature or 2m AGL relative humidity.
I
> > captured
> > the following paragraph from the MET V4.1 User's Guide. I then
tried to
> > express in the paragraph below that, my interpretation of the
User's
> Guide
> > explanation. I also have the following questions which arose as I
tried
> to
> > interpret the User's Guide:
> >
> > Is the scoring considered a "hit" based on the continuous
statistics
> error
> > calculations (i.e. forecast-observation pair errors) or based on a
> whether
> > the category threshold condition was met?
> >
> > Do the values of individual forecast grid points get assigned a
new value
> > based on the values of all the forecast grid points within the
> neighborhood
> > before the gridpoint to gridpoint scoring is performed so as to
generate
> > continuous statistics? What about the situation where the
observation
> grid
> > points are assigned new values as well based on the application of
the
> same
> > neighborhood?
> >
> > "MET also incorporates several neighborhood methods to give credit
to
> > forecasts that are close to the observations, but not necessarily
exactly
> > matched up in
> > space. Also referred to as "fuzzy" verification methods, these
methods do
> > not just
> > compare a single forecast at each grid point to a single
observation at
> > each
> > grid point; they
> > compare the forecasts and observations in a neighborhood
surrounding the
> > point of
> > interest. With the neighborhood method, the user chooses a
distance
> within
> > which the
> > forecast event can fall from the observed event and still be
considered a
> > hit. In MET this
> > is implemented by defining a square search window around each grid
point.
> > Within the search window, the number of observed events is
compared to
> the
> > number of
> > Forecast events. In this way, credit is given to forecasts that
are close
> > to
> > the
> > Observations without requiring a strict match between forecasted
events
> and
> > observed
> > events at any particular grid point. The neighborhood methods
allow the
> > user
> > to see how
> > forecast skill varies with neighborhood size and can help
determine the
> > smallest
> > neighborhood size that can be used to give sufficiently accurate
> > forecasts."
> >
> > My interpretation: The "neighborhood" can be applied to both the
forecast
> > field and the observed field as well as just one or the other.
Within the
> > neighborhood, a number of observed values will be compared to the
same
> > number of forecast values and the scoring is based on those
comparisons
> > rather than a single score derived from the difference calculated
at a
> > particular grid point. Instead of requiring that a "hit" be
defined by
> the
> > forecast-observation difference of 0 at each grid point, the use
of a
> > neighborhood for the observed field allows you to define a "hit"
if the
> > forecast value falls within a range of the observed values
included in
> the
> > neighborhood. Further, if the same neighborhood also applies to
the
> > forecast
> > field, then, that too, provides a range of forecast values which
can be
> > considered "hits' when compared to the range of observed values.
> >
> > Thanks.
> >
> > R/
> > John
> >
> > Mr John W. Raby, Meteorologist
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > (575) 678-2004 DSN 258-2004
> > FAX (575) 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> >
> >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
>
>

------------------------------------------------
Subject: Questions on Neighborhood Method (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Jun 12 07:59:40 2014

Classification: UNCLASSIFIED
Caveats: NONE

John -

Thanks for correcting my understanding of the way the nbrhd and the
interp
sections are working independently. I need to think about this to
thoroughly
grasp what it means. Now, I can start thinking about the two separate
types of
scoring provided by the settings in these two sections.

For the nbrhd section, the results (based on fractional coverage which
is
computed from hits or misses from categorical forecasts) are
summarized/captured in the FSS and FBS scores, but for the interp,
there is no
overall "score" to summarize the results other than the statistics in
the CNT,
CTC, and CTS line types which include the ME, MAE and RMSE computed
over the
raw forecast and observation fields directly.

For both sections, depending on your settings, it's possible that the
individual grid point values are assigned new values.

In your example for the interp section, you mentioned that the "raw
forecast
value would be replaced by the average forecast value using the 25
closest
points". Since your setting  was "BOTH" wouldn't the same happen to
the raw
observed values?

R/
John

-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, June 11, 2014 4:01 PM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #67570] Questions on Neighborhood Method
(UNCLASSIFIED)

John,

For your first question on "hits", your answer looks correct.

For your other two questions about whether the grid point gets
assigned a new
value, I would say the answer is yes.  When applying the neighborhood
methods,
the value at each forecast (and observation) grid point gets replaced
by the
fractional coverage within the neighborhood around that point.  The
FSS and
FBS statistics are computed over those fractional coverage values, not
the raw
forecast (and observation) values.  So the stats in the NBRCNT,
NBRCTC, and
NBRCTS line types are computed over the fractional coverage fields
(derived
from the raw forecast and observation fields).

The stats in all the other line types (like CNT, CTC, and CTS) are
computed
over the raw forecast and observation fields directly.

I think you may be confusing two different sections of the config
file:
"nbrhd" vs "interp".

The nbrhd section controls the logic when computing NBRCNT, NBRCTC,
and NBRCTS
output line types.

The interp section is separate and does not interact with the nbrhd
section.
It enables you to smooth either the forecast or observation field or
both.  By
default, its set up to do no smoothing.  Suppose for example, you
setup the
interp section like this:
interp = {
   field      = BOTH;
   vld_thresh = 1.0;

   type = [
      {
         method = UW_MEAN;
         width  = 5;
      }
   ];
};

The would apply a 5x5 smoothing box to each grid point.  In this case,
the raw
forecast value would be replaced by the average forecast value using
the 25
closest points.  Then the output in the CNT, CTC, and CTS line types
would be
computed over those smoothed values.

So smoothing using the "interp" settings and neighborhood methods
using the
"nbrhd" settings are both available, but are different.

Hope that helps.

Thanks,
John



On Wed, Jun 11, 2014 at 1:37 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
>
> John -
>
> Having your detailed explanation of how the method works is very
helpful.
> It allows you to slowly understand it by putting each piece together
> in the right order.
>
> So, my understanding now leads me to answer my questions as follows:
>
> Q:Is the scoring considered a "hit" based on the continuous
statistics
> error
> > calculations (i.e. forecast-observation pair errors) or based on a
> whether
> > the category threshold condition was met?
> A: Scoring is based on categorical forecasts and categorical
observations.
> (0-miss and 1-hit where a hit is defined as an occurrence when the
> forecast
> (observation) meets the threshold criteria.
>
> Q: Do the values of individual forecast grid points get assigned a
new
> value
> > based on the values of all the forecast grid points within the
> neighborhood
> > before the gridpoint to gridpoint scoring is performed so as to
> > generate continuous statistics?
> A: No re-assigining of forecast grid point values.
>
> Q: What about the situation where the observation grid
> > points are assigned new values as well based on the application of
> > the
> same
> > neighborhood?
> A:  No re-assigining of observation grid point values.
>
> Thanks for taking the time to explain the process.
>
> R/
> John
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Wednesday, June 11, 2014 1:04 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #67570] Questions on Neighborhood
Method
> (UNCLASSIFIED)
>
> John,
>
> I read through your email and see that you have several questions
> about how the neighborhood methods are implemented in the Grid-Stat
> tool.  Rather than addressing each individual point in your email,
let
> me first try to lay out the steps that are applied in Grid-Stat.
> After reading this, please let me know if you have further
questions.
>
> (1) First, neighborhood methods are currently only available in
> grid_stat since they require both the forecast and observation to be
> gridded (and on the same grid).  There are neighborhood methods
> defined for point observations, but those are not yet available in
MET.
>
> (2) The Fractions Skill Score (FSS) and Fractions Brier Score (FBS)
> are the two most popular neighborhood methods statistics, and they
are
> contained in the NBRCNT (Neighborhood Continuous Statistics) line
type.
>
> (3) FSS and FBS are computed using two main parameters: a threshold
> (cat_thresh in the config file) and a neighborhood size ("nbr.width"
> in the config file is the "width" of a square box, e.g. width = 3
> means a 3x3 box containing 9 points).  The neighborhood width must
be
> odd so that the box can be centered on each grid point.  A NBRCNT
> output line will be generated for each combination of threshold and
> neighborhood width.
>
> (4) The first step is to apply one of the thresholds from the
"cat_thresh"
> list to both the forecast and observation fields.  These steps are
> applied equally to both fields - not one or the other.  This
replaces
> the raw fields with fields of 0's and 1's... 1 where the threshold
> criteria is met and 0 otherwise.
>
> (5) Next, for each grid point in the forecast field, draw an nxn box
> around it (where n is the neighborhood width).  Count up the number
of
> 1's inside that box, and compute a ratio of the number of 1's
divided
> by the size of the box (nxn) to get a number between 0 and 1.  We
call
> this number a "fractional coverage" value for that grid point.  For
> example, if the event occurred at 4 of the 9 grid points, the
> fractional coverage value would be
> 4/9 = 0.44.  Doing this for every grid point in the forecast field
> generates a forecast "fractional coverage" field.
>
> As a small side note, for grid points near the edge of the grid or
> near missing data values, not all of the nxn points in the
> neighborhood will contain valid data.  The "nbr.vld_thresh" config
> file setting specifies what percentage of grid points must contain
> valid data for a fractional coverage value to be computed.  A value
of
> 1.0 (the default) means that all nxn points must valid data for
> fractional coverage to be computed at the point.
>
> (6) Apply the same logic listed in (5) to the observation field to
> compute an observation "fractional coverage" field.
>
> (7) At each grid point, we now have a forecast fractional coverage
> value between 0 and 1 and an observation fractional coverage value
> between 0 and 1.
>
> (8) The Fractions Skill Score and Fractions Brier Score are computed
> using these fractional coverage values directly.  They are written
out
> in the NBRCNT line type.
>
> (9) MET also contains NBRCTC and NBRCTS line types, which are not
> commonly used.  These are computed by applying a threshold
> ("nbrhd.cov_thresh" in the config file) to those fractional coverage
> fields.  Applying a threshold converts those fractional coverage
fields back
> into fields of 0's and 1's.
> A 2x2 contingency table is computed over those thresholded fields.
> The contingency table counts are stored in the NBRCTC line type, and
> the corresponding statistics are stored in the NBRCTS line type.
But
> as I said, these are not commonly used.
>
> As I mentioned, the FSS and FBS are the most commonly used
> neighborhood methods statistics.  For a given categorical threshold,
> you'll find that as you increase the neighborhood size, the FSS will
> increase as well.  FSS is often used to determine the scale at which
a
> forecast is skillful.  For example, a 4km model run will contain a
lot
> of great detail at the finest scales, but likely won't be good at
> getting that detail exactly right.  For a particular threshold of
> interest, you might compute FSS for several neighborhood sizes, 3x3,
> 5x5, 7x7, and so on.  Suppose you see that the model becomes
skillful
> at the 5x5 neighborhood size.  So even though the model has great
> detail at 4km, it's much more skill full at 20km resolution.  If you
> were making forecasts to the public, you might choose to upscale
your
> model output to 20km prior to distributing it to the public
> - otherwise, you'd be implying more confidence in the details of the
> model than you actually have.
>
> Sorry for the long email!  Hopefully, that helps explain it.  Just
let
> us know if you have more questions.
>
> Thanks,
> John
>
>
>
> On Tue, Jun 10, 2014 at 11:26 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Jun 10 11:26:28 2014: Request 67570 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> >        Queue: met_help
> >      Subject: Questions on Neighborhood Method (UNCLASSIFIED)
> >        Owner: Nobody
> >   Requestors: john.w.raby2.civ at mail.mil
> >       Status: new
> >  Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > I am trying to understand the application of the MET Grid-Stat
> neighborhood
> > method for the calculation of neighborhood continuous statistics
for
> scalar
> > variables such as 2m AGL temperature or 2m AGL relative humidity.
I
> > captured the following paragraph from the MET V4.1 User's Guide. I
> > then tried to express in the paragraph below that, my
interpretation
> > of the User's
> Guide
> > explanation. I also have the following questions which arose as I
> > tried
> to
> > interpret the User's Guide:
> >
> > Is the scoring considered a "hit" based on the continuous
statistics
> error
> > calculations (i.e. forecast-observation pair errors) or based on a
> whether
> > the category threshold condition was met?
> >
> > Do the values of individual forecast grid points get assigned a
new
> > value based on the values of all the forecast grid points within
the
> neighborhood
> > before the gridpoint to gridpoint scoring is performed so as to
> > generate continuous statistics? What about the situation where the
> > observation
> grid
> > points are assigned new values as well based on the application of
> > the
> same
> > neighborhood?
> >
> > "MET also incorporates several neighborhood methods to give credit
> > to forecasts that are close to the observations, but not
necessarily
> > exactly matched up in space. Also referred to as "fuzzy"
> > verification methods, these methods do not just compare a single
> > forecast at each grid point to a single observation at each grid
> > point; they compare the forecasts and observations in a
neighborhood
> > surrounding the point of interest. With the neighborhood method,
the
> > user chooses a distance
> within
> > which the
> > forecast event can fall from the observed event and still be
> > considered a hit. In MET this is implemented by defining a square
> > search window around each grid point.
> > Within the search window, the number of observed events is
compared
> > to
> the
> > number of
> > Forecast events. In this way, credit is given to forecasts that
are
> > close to the Observations without requiring a strict match between
> > forecasted events
> and
> > observed
> > events at any particular grid point. The neighborhood methods
allow
> > the user to see how forecast skill varies with neighborhood size
and
> > can help determine the smallest neighborhood size that can be used
> > to give sufficiently accurate forecasts."
> >
> > My interpretation: The "neighborhood" can be applied to both the
> > forecast field and the observed field as well as just one or the
> > other. Within the neighborhood, a number of observed values will
be
> > compared to the same number of forecast values and the scoring is
> > based on those comparisons rather than a single score derived from
> > the difference calculated at a particular grid point. Instead of
> > requiring that a "hit" be defined by
> the
> > forecast-observation difference of 0 at each grid point, the use
of
> > a neighborhood for the observed field allows you to define a "hit"
> > if the forecast value falls within a range of the observed values
> > included in
> the
> > neighborhood. Further, if the same neighborhood also applies to
the
> > forecast field, then, that too, provides a range of forecast
values
> > which can be considered "hits' when compared to the range of
> > observed values.
> >
> > Thanks.
> >
> > R/
> > John
> >
> > Mr John W. Raby, Meteorologist
> > U.S. Army Research Laboratory
> > White Sands Missile Range, NM 88002
> > (575) 678-2004 DSN 258-2004
> > FAX (575) 678-1230 DSN 258-1230
> > Email: john.w.raby2.civ at mail.mil
> >
> >
> >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
>
>


Classification: UNCLASSIFIED
Caveats: NONE



------------------------------------------------
Subject: Questions on Neighborhood Method (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Jun 12 10:18:45 2014

John,

Yes, since "field = BOTH;" that smoothing filter would be applied to
both
the forecast and observation fields, and then they'd be compared grid-
point
by grid-point.

I our verification work here in the DTC, we have not really used the
"interp" options much.  Instead, we use the default setting which says
to
do no smoothing on either field.  However, we do use the "nbrhd"
options to
compute FSS and see how it varies based on threshold and neighborhood
size.

Thanks,
John


On Thu, Jun 12, 2014 at 7:59 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for correcting my understanding of the way the nbrhd and the
interp
> sections are working independently. I need to think about this to
> thoroughly
> grasp what it means. Now, I can start thinking about the two
separate
> types of
> scoring provided by the settings in these two sections.
>
> For the nbrhd section, the results (based on fractional coverage
which is
> computed from hits or misses from categorical forecasts) are
> summarized/captured in the FSS and FBS scores, but for the interp,
there
> is no
> overall "score" to summarize the results other than the statistics
in the
> CNT,
> CTC, and CTS line types which include the ME, MAE and RMSE computed
over
> the
> raw forecast and observation fields directly.
>
> For both sections, depending on your settings, it's possible that
the
> individual grid point values are assigned new values.
>
> In your example for the interp section, you mentioned that the "raw
> forecast
> value would be replaced by the average forecast value using the 25
closest
> points". Since your setting  was "BOTH" wouldn't the same happen to
the raw
> observed values?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, June 11, 2014 4:01 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #67570] Questions on Neighborhood
Method
> (UNCLASSIFIED)
>
> John,
>
> For your first question on "hits", your answer looks correct.
>
> For your other two questions about whether the grid point gets
assigned a
> new
> value, I would say the answer is yes.  When applying the
neighborhood
> methods,
> the value at each forecast (and observation) grid point gets
replaced by
> the
> fractional coverage within the neighborhood around that point.  The
FSS and
> FBS statistics are computed over those fractional coverage values,
not the
> raw
> forecast (and observation) values.  So the stats in the NBRCNT,
NBRCTC, and
> NBRCTS line types are computed over the fractional coverage fields
(derived
> from the raw forecast and observation fields).
>
> The stats in all the other line types (like CNT, CTC, and CTS) are
computed
> over the raw forecast and observation fields directly.
>
> I think you may be confusing two different sections of the config
file:
> "nbrhd" vs "interp".
>
> The nbrhd section controls the logic when computing NBRCNT, NBRCTC,
and
> NBRCTS
> output line types.
>
> The interp section is separate and does not interact with the nbrhd
> section.
> It enables you to smooth either the forecast or observation field or
both.
>  By
> default, its set up to do no smoothing.  Suppose for example, you
setup the
> interp section like this:
> interp = {
>    field      = BOTH;
>    vld_thresh = 1.0;
>
>    type = [
>       {
>          method = UW_MEAN;
>          width  = 5;
>       }
>    ];
> };
>
> The would apply a 5x5 smoothing box to each grid point.  In this
case, the
> raw
> forecast value would be replaced by the average forecast value using
the 25
> closest points.  Then the output in the CNT, CTC, and CTS line types
would
> be
> computed over those smoothed values.
>
> So smoothing using the "interp" settings and neighborhood methods
using the
> "nbrhd" settings are both available, but are different.
>
> Hope that helps.
>
> Thanks,
> John
>
>
>
> On Wed, Jun 11, 2014 at 1:37 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
> >
> > John -
> >
> > Having your detailed explanation of how the method works is very
helpful.
> > It allows you to slowly understand it by putting each piece
together
> > in the right order.
> >
> > So, my understanding now leads me to answer my questions as
follows:
> >
> > Q:Is the scoring considered a "hit" based on the continuous
statistics
> > error
> > > calculations (i.e. forecast-observation pair errors) or based on
a
> > whether
> > > the category threshold condition was met?
> > A: Scoring is based on categorical forecasts and categorical
> observations.
> > (0-miss and 1-hit where a hit is defined as an occurrence when the
> > forecast
> > (observation) meets the threshold criteria.
> >
> > Q: Do the values of individual forecast grid points get assigned a
new
> > value
> > > based on the values of all the forecast grid points within the
> > neighborhood
> > > before the gridpoint to gridpoint scoring is performed so as to
> > > generate continuous statistics?
> > A: No re-assigining of forecast grid point values.
> >
> > Q: What about the situation where the observation grid
> > > points are assigned new values as well based on the application
of
> > > the
> > same
> > > neighborhood?
> > A:  No re-assigining of observation grid point values.
> >
> > Thanks for taking the time to explain the process.
> >
> > R/
> > John
> >
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Wednesday, June 11, 2014 1:04 PM
> > To: Raby, John W CIV USARMY ARL (US)
> > Subject: Re: [rt.rap.ucar.edu #67570] Questions on Neighborhood
Method
> > (UNCLASSIFIED)
> >
> > John,
> >
> > I read through your email and see that you have several questions
> > about how the neighborhood methods are implemented in the Grid-
Stat
> > tool.  Rather than addressing each individual point in your email,
let
> > me first try to lay out the steps that are applied in Grid-Stat.
> > After reading this, please let me know if you have further
questions.
> >
> > (1) First, neighborhood methods are currently only available in
> > grid_stat since they require both the forecast and observation to
be
> > gridded (and on the same grid).  There are neighborhood methods
> > defined for point observations, but those are not yet available in
MET.
> >
> > (2) The Fractions Skill Score (FSS) and Fractions Brier Score
(FBS)
> > are the two most popular neighborhood methods statistics, and they
are
> > contained in the NBRCNT (Neighborhood Continuous Statistics) line
type.
> >
> > (3) FSS and FBS are computed using two main parameters: a
threshold
> > (cat_thresh in the config file) and a neighborhood size
("nbr.width"
> > in the config file is the "width" of a square box, e.g. width = 3
> > means a 3x3 box containing 9 points).  The neighborhood width must
be
> > odd so that the box can be centered on each grid point.  A NBRCNT
> > output line will be generated for each combination of threshold
and
> > neighborhood width.
> >
> > (4) The first step is to apply one of the thresholds from the
> "cat_thresh"
> > list to both the forecast and observation fields.  These steps are
> > applied equally to both fields - not one or the other.  This
replaces
> > the raw fields with fields of 0's and 1's... 1 where the threshold
> > criteria is met and 0 otherwise.
> >
> > (5) Next, for each grid point in the forecast field, draw an nxn
box
> > around it (where n is the neighborhood width).  Count up the
number of
> > 1's inside that box, and compute a ratio of the number of 1's
divided
> > by the size of the box (nxn) to get a number between 0 and 1.  We
call
> > this number a "fractional coverage" value for that grid point.
For
> > example, if the event occurred at 4 of the 9 grid points, the
> > fractional coverage value would be
> > 4/9 = 0.44.  Doing this for every grid point in the forecast field
> > generates a forecast "fractional coverage" field.
> >
> > As a small side note, for grid points near the edge of the grid or
> > near missing data values, not all of the nxn points in the
> > neighborhood will contain valid data.  The "nbr.vld_thresh" config
> > file setting specifies what percentage of grid points must contain
> > valid data for a fractional coverage value to be computed.  A
value of
> > 1.0 (the default) means that all nxn points must valid data for
> > fractional coverage to be computed at the point.
> >
> > (6) Apply the same logic listed in (5) to the observation field to
> > compute an observation "fractional coverage" field.
> >
> > (7) At each grid point, we now have a forecast fractional coverage
> > value between 0 and 1 and an observation fractional coverage value
> > between 0 and 1.
> >
> > (8) The Fractions Skill Score and Fractions Brier Score are
computed
> > using these fractional coverage values directly.  They are written
out
> > in the NBRCNT line type.
> >
> > (9) MET also contains NBRCTC and NBRCTS line types, which are not
> > commonly used.  These are computed by applying a threshold
> > ("nbrhd.cov_thresh" in the config file) to those fractional
coverage
> > fields.  Applying a threshold converts those fractional coverage
fields
> back
> > into fields of 0's and 1's.
> > A 2x2 contingency table is computed over those thresholded fields.
> > The contingency table counts are stored in the NBRCTC line type,
and
> > the corresponding statistics are stored in the NBRCTS line type.
But
> > as I said, these are not commonly used.
> >
> > As I mentioned, the FSS and FBS are the most commonly used
> > neighborhood methods statistics.  For a given categorical
threshold,
> > you'll find that as you increase the neighborhood size, the FSS
will
> > increase as well.  FSS is often used to determine the scale at
which a
> > forecast is skillful.  For example, a 4km model run will contain a
lot
> > of great detail at the finest scales, but likely won't be good at
> > getting that detail exactly right.  For a particular threshold of
> > interest, you might compute FSS for several neighborhood sizes,
3x3,
> > 5x5, 7x7, and so on.  Suppose you see that the model becomes
skillful
> > at the 5x5 neighborhood size.  So even though the model has great
> > detail at 4km, it's much more skill full at 20km resolution.  If
you
> > were making forecasts to the public, you might choose to upscale
your
> > model output to 20km prior to distributing it to the public
> > - otherwise, you'd be implying more confidence in the details of
the
> > model than you actually have.
> >
> > Sorry for the long email!  Hopefully, that helps explain it.  Just
let
> > us know if you have more questions.
> >
> > Thanks,
> > John
> >
> >
> >
> > On Tue, Jun 10, 2014 at 11:26 AM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Jun 10 11:26:28 2014: Request 67570 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > >        Queue: met_help
> > >      Subject: Questions on Neighborhood Method (UNCLASSIFIED)
> > >        Owner: Nobody
> > >   Requestors: john.w.raby2.civ at mail.mil
> > >       Status: new
> > >  Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > I am trying to understand the application of the MET Grid-Stat
> > neighborhood
> > > method for the calculation of neighborhood continuous statistics
for
> > scalar
> > > variables such as 2m AGL temperature or 2m AGL relative
humidity. I
> > > captured the following paragraph from the MET V4.1 User's Guide.
I
> > > then tried to express in the paragraph below that, my
interpretation
> > > of the User's
> > Guide
> > > explanation. I also have the following questions which arose as
I
> > > tried
> > to
> > > interpret the User's Guide:
> > >
> > > Is the scoring considered a "hit" based on the continuous
statistics
> > error
> > > calculations (i.e. forecast-observation pair errors) or based on
a
> > whether
> > > the category threshold condition was met?
> > >
> > > Do the values of individual forecast grid points get assigned a
new
> > > value based on the values of all the forecast grid points within
the
> > neighborhood
> > > before the gridpoint to gridpoint scoring is performed so as to
> > > generate continuous statistics? What about the situation where
the
> > > observation
> > grid
> > > points are assigned new values as well based on the application
of
> > > the
> > same
> > > neighborhood?
> > >
> > > "MET also incorporates several neighborhood methods to give
credit
> > > to forecasts that are close to the observations, but not
necessarily
> > > exactly matched up in space. Also referred to as "fuzzy"
> > > verification methods, these methods do not just compare a single
> > > forecast at each grid point to a single observation at each grid
> > > point; they compare the forecasts and observations in a
neighborhood
> > > surrounding the point of interest. With the neighborhood method,
the
> > > user chooses a distance
> > within
> > > which the
> > > forecast event can fall from the observed event and still be
> > > considered a hit. In MET this is implemented by defining a
square
> > > search window around each grid point.
> > > Within the search window, the number of observed events is
compared
> > > to
> > the
> > > number of
> > > Forecast events. In this way, credit is given to forecasts that
are
> > > close to the Observations without requiring a strict match
between
> > > forecasted events
> > and
> > > observed
> > > events at any particular grid point. The neighborhood methods
allow
> > > the user to see how forecast skill varies with neighborhood size
and
> > > can help determine the smallest neighborhood size that can be
used
> > > to give sufficiently accurate forecasts."
> > >
> > > My interpretation: The "neighborhood" can be applied to both the
> > > forecast field and the observed field as well as just one or the
> > > other. Within the neighborhood, a number of observed values will
be
> > > compared to the same number of forecast values and the scoring
is
> > > based on those comparisons rather than a single score derived
from
> > > the difference calculated at a particular grid point. Instead of
> > > requiring that a "hit" be defined by
> > the
> > > forecast-observation difference of 0 at each grid point, the use
of
> > > a neighborhood for the observed field allows you to define a
"hit"
> > > if the forecast value falls within a range of the observed
values
> > > included in
> > the
> > > neighborhood. Further, if the same neighborhood also applies to
the
> > > forecast field, then, that too, provides a range of forecast
values
> > > which can be considered "hits' when compared to the range of
> > > observed values.
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > Mr John W. Raby, Meteorologist
> > > U.S. Army Research Laboratory
> > > White Sands Missile Range, NM 88002
> > > (575) 678-2004 DSN 258-2004
> > > FAX (575) 678-1230 DSN 258-1230
> > > Email: john.w.raby2.civ at mail.mil
> > >
> > >
> > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>

------------------------------------------------
Subject: Questions on Neighborhood Method (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Jun 12 10:31:14 2014

Classification: UNCLASSIFIED
Caveats: NONE

John -

All I can say is that this exchange has been very helpful to me and I
appreciate your sharing the knowledge and experience on this topic. We
haven't
used Grid-Stat other than for some limited testing, My intent is to
use it
more to augment the info provided by Point-Stat and MODE on our high
resolution WRF modeling project. Each tool assesses performance, but
in a
unique way which captures a different aspect of performance. For our
work at
high resolution, it is especially important to assess performance
using all
the tools since you are more likely to reveal the true skill of the
model than
if you only use one tool.

R/
John

-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, June 12, 2014 10:19 AM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #67570] Questions on Neighborhood Method
(UNCLASSIFIED)

John,

Yes, since "field = BOTH;" that smoothing filter would be applied to
both the
forecast and observation fields, and then they'd be compared grid-
point by
grid-point.

I our verification work here in the DTC, we have not really used the
"interp"
options much.  Instead, we use the default setting which says to do no
smoothing on either field.  However, we do use the "nbrhd" options to
compute
FSS and see how it varies based on threshold and neighborhood size.

Thanks,
John


On Thu, Jun 12, 2014 at 7:59 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> Thanks for correcting my understanding of the way the nbrhd and the
> interp sections are working independently. I need to think about
this
> to thoroughly grasp what it means. Now, I can start thinking about
the
> two separate types of scoring provided by the settings in these two
> sections.
>
> For the nbrhd section, the results (based on fractional coverage
which
> is computed from hits or misses from categorical forecasts) are
> summarized/captured in the FSS and FBS scores, but for the interp,
> there is no overall "score" to summarize the results other than the
> statistics in the CNT, CTC, and CTS line types which include the ME,
> MAE and RMSE computed over the raw forecast and observation fields
> directly.
>
> For both sections, depending on your settings, it's possible that
the
> individual grid point values are assigned new values.
>
> In your example for the interp section, you mentioned that the "raw
> forecast value would be replaced by the average forecast value using
> the 25 closest points". Since your setting  was "BOTH" wouldn't the
> same happen to the raw observed values?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, June 11, 2014 4:01 PM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #67570] Questions on Neighborhood
Method
> (UNCLASSIFIED)
>
> John,
>
> For your first question on "hits", your answer looks correct.
>
> For your other two questions about whether the grid point gets
> assigned a new value, I would say the answer is yes.  When applying
> the neighborhood methods, the value at each forecast (and
observation)
> grid point gets replaced by the fractional coverage within the
> neighborhood around that point.  The FSS and FBS statistics are
> computed over those fractional coverage values, not the raw forecast
> (and observation) values.  So the stats in the NBRCNT, NBRCTC, and
> NBRCTS line types are computed over the fractional coverage fields
> (derived from the raw forecast and observation fields).
>
> The stats in all the other line types (like CNT, CTC, and CTS) are
> computed over the raw forecast and observation fields directly.
>
> I think you may be confusing two different sections of the config
file:
> "nbrhd" vs "interp".
>
> The nbrhd section controls the logic when computing NBRCNT, NBRCTC,
> and NBRCTS output line types.
>
> The interp section is separate and does not interact with the nbrhd
> section.
> It enables you to smooth either the forecast or observation field or
both.
>  By
> default, its set up to do no smoothing.  Suppose for example, you
> setup the interp section like this:
> interp = {
>    field      = BOTH;
>    vld_thresh = 1.0;
>
>    type = [
>       {
>          method = UW_MEAN;
>          width  = 5;
>       }
>    ];
> };
>
> The would apply a 5x5 smoothing box to each grid point.  In this
case,
> the raw forecast value would be replaced by the average forecast
value
> using the 25 closest points.  Then the output in the CNT, CTC, and
CTS
> line types would be computed over those smoothed values.
>
> So smoothing using the "interp" settings and neighborhood methods
> using the "nbrhd" settings are both available, but are different.
>
> Hope that helps.
>
> Thanks,
> John
>
>
>
> On Wed, Jun 11, 2014 at 1:37 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
> >
> > John -
> >
> > Having your detailed explanation of how the method works is very
helpful.
> > It allows you to slowly understand it by putting each piece
together
> > in the right order.
> >
> > So, my understanding now leads me to answer my questions as
follows:
> >
> > Q:Is the scoring considered a "hit" based on the continuous
> > statistics error
> > > calculations (i.e. forecast-observation pair errors) or based on
a
> > whether
> > > the category threshold condition was met?
> > A: Scoring is based on categorical forecasts and categorical
> observations.
> > (0-miss and 1-hit where a hit is defined as an occurrence when the
> > forecast
> > (observation) meets the threshold criteria.
> >
> > Q: Do the values of individual forecast grid points get assigned a
> > new value
> > > based on the values of all the forecast grid points within the
> > neighborhood
> > > before the gridpoint to gridpoint scoring is performed so as to
> > > generate continuous statistics?
> > A: No re-assigining of forecast grid point values.
> >
> > Q: What about the situation where the observation grid
> > > points are assigned new values as well based on the application
of
> > > the
> > same
> > > neighborhood?
> > A:  No re-assigining of observation grid point values.
> >
> > Thanks for taking the time to explain the process.
> >
> > R/
> > John
> >
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Wednesday, June 11, 2014 1:04 PM
> > To: Raby, John W CIV USARMY ARL (US)
> > Subject: Re: [rt.rap.ucar.edu #67570] Questions on Neighborhood
> > Method
> > (UNCLASSIFIED)
> >
> > John,
> >
> > I read through your email and see that you have several questions
> > about how the neighborhood methods are implemented in the Grid-
Stat
> > tool.  Rather than addressing each individual point in your email,
> > let me first try to lay out the steps that are applied in Grid-
Stat.
> > After reading this, please let me know if you have further
questions.
> >
> > (1) First, neighborhood methods are currently only available in
> > grid_stat since they require both the forecast and observation to
be
> > gridded (and on the same grid).  There are neighborhood methods
> > defined for point observations, but those are not yet available in
MET.
> >
> > (2) The Fractions Skill Score (FSS) and Fractions Brier Score
(FBS)
> > are the two most popular neighborhood methods statistics, and they
> > are contained in the NBRCNT (Neighborhood Continuous Statistics)
line
> > type.
> >
> > (3) FSS and FBS are computed using two main parameters: a
threshold
> > (cat_thresh in the config file) and a neighborhood size
("nbr.width"
> > in the config file is the "width" of a square box, e.g. width = 3
> > means a 3x3 box containing 9 points).  The neighborhood width must
> > be odd so that the box can be centered on each grid point.  A
NBRCNT
> > output line will be generated for each combination of threshold
and
> > neighborhood width.
> >
> > (4) The first step is to apply one of the thresholds from the
> "cat_thresh"
> > list to both the forecast and observation fields.  These steps are
> > applied equally to both fields - not one or the other.  This
> > replaces the raw fields with fields of 0's and 1's... 1 where the
> > threshold criteria is met and 0 otherwise.
> >
> > (5) Next, for each grid point in the forecast field, draw an nxn
box
> > around it (where n is the neighborhood width).  Count up the
number
> > of 1's inside that box, and compute a ratio of the number of 1's
> > divided by the size of the box (nxn) to get a number between 0 and
> > 1.  We call this number a "fractional coverage" value for that
grid
> > point.  For example, if the event occurred at 4 of the 9 grid
> > points, the fractional coverage value would be
> > 4/9 = 0.44.  Doing this for every grid point in the forecast field
> > generates a forecast "fractional coverage" field.
> >
> > As a small side note, for grid points near the edge of the grid or
> > near missing data values, not all of the nxn points in the
> > neighborhood will contain valid data.  The "nbr.vld_thresh" config
> > file setting specifies what percentage of grid points must contain
> > valid data for a fractional coverage value to be computed.  A
value
> > of
> > 1.0 (the default) means that all nxn points must valid data for
> > fractional coverage to be computed at the point.
> >
> > (6) Apply the same logic listed in (5) to the observation field to
> > compute an observation "fractional coverage" field.
> >
> > (7) At each grid point, we now have a forecast fractional coverage
> > value between 0 and 1 and an observation fractional coverage value
> > between 0 and 1.
> >
> > (8) The Fractions Skill Score and Fractions Brier Score are
computed
> > using these fractional coverage values directly.  They are written
> > out in the NBRCNT line type.
> >
> > (9) MET also contains NBRCTC and NBRCTS line types, which are not
> > commonly used.  These are computed by applying a threshold
> > ("nbrhd.cov_thresh" in the config file) to those fractional
coverage
> > fields.  Applying a threshold converts those fractional coverage
> > fields
> back
> > into fields of 0's and 1's.
> > A 2x2 contingency table is computed over those thresholded fields.
> > The contingency table counts are stored in the NBRCTC line type,
and
> > the corresponding statistics are stored in the NBRCTS line type.
> > But as I said, these are not commonly used.
> >
> > As I mentioned, the FSS and FBS are the most commonly used
> > neighborhood methods statistics.  For a given categorical
threshold,
> > you'll find that as you increase the neighborhood size, the FSS
will
> > increase as well.  FSS is often used to determine the scale at
which
> > a forecast is skillful.  For example, a 4km model run will contain
a
> > lot of great detail at the finest scales, but likely won't be good
> > at getting that detail exactly right.  For a particular threshold
of
> > interest, you might compute FSS for several neighborhood sizes,
3x3,
> > 5x5, 7x7, and so on.  Suppose you see that the model becomes
> > skillful at the 5x5 neighborhood size.  So even though the model
has
> > great detail at 4km, it's much more skill full at 20km resolution.
> > If you were making forecasts to the public, you might choose to
> > upscale your model output to 20km prior to distributing it to the
> > public
> > - otherwise, you'd be implying more confidence in the details of
the
> > model than you actually have.
> >
> > Sorry for the long email!  Hopefully, that helps explain it.  Just
> > let us know if you have more questions.
> >
> > Thanks,
> > John
> >
> >
> >
> > On Tue, Jun 10, 2014 at 11:26 AM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Jun 10 11:26:28 2014: Request 67570 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > >        Queue: met_help
> > >      Subject: Questions on Neighborhood Method (UNCLASSIFIED)
> > >        Owner: Nobody
> > >   Requestors: john.w.raby2.civ at mail.mil
> > >       Status: new
> > >  Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > I am trying to understand the application of the MET Grid-Stat
> > neighborhood
> > > method for the calculation of neighborhood continuous statistics
> > > for
> > scalar
> > > variables such as 2m AGL temperature or 2m AGL relative
humidity.
> > > I captured the following paragraph from the MET V4.1 User's
Guide.
> > > I then tried to express in the paragraph below that, my
> > > interpretation of the User's
> > Guide
> > > explanation. I also have the following questions which arose as
I
> > > tried
> > to
> > > interpret the User's Guide:
> > >
> > > Is the scoring considered a "hit" based on the continuous
> > > statistics
> > error
> > > calculations (i.e. forecast-observation pair errors) or based on
a
> > whether
> > > the category threshold condition was met?
> > >
> > > Do the values of individual forecast grid points get assigned a
> > > new value based on the values of all the forecast grid points
> > > within the
> > neighborhood
> > > before the gridpoint to gridpoint scoring is performed so as to
> > > generate continuous statistics? What about the situation where
the
> > > observation
> > grid
> > > points are assigned new values as well based on the application
of
> > > the
> > same
> > > neighborhood?
> > >
> > > "MET also incorporates several neighborhood methods to give
credit
> > > to forecasts that are close to the observations, but not
> > > necessarily exactly matched up in space. Also referred to as
"fuzzy"
> > > verification methods, these methods do not just compare a single
> > > forecast at each grid point to a single observation at each grid
> > > point; they compare the forecasts and observations in a
> > > neighborhood surrounding the point of interest. With the
> > > neighborhood method, the user chooses a distance
> > within
> > > which the
> > > forecast event can fall from the observed event and still be
> > > considered a hit. In MET this is implemented by defining a
square
> > > search window around each grid point.
> > > Within the search window, the number of observed events is
> > > compared to
> > the
> > > number of
> > > Forecast events. In this way, credit is given to forecasts that
> > > are close to the Observations without requiring a strict match
> > > between forecasted events
> > and
> > > observed
> > > events at any particular grid point. The neighborhood methods
> > > allow the user to see how forecast skill varies with
neighborhood
> > > size and can help determine the smallest neighborhood size that
> > > can be used to give sufficiently accurate forecasts."
> > >
> > > My interpretation: The "neighborhood" can be applied to both the
> > > forecast field and the observed field as well as just one or the
> > > other. Within the neighborhood, a number of observed values will
> > > be compared to the same number of forecast values and the
scoring
> > > is based on those comparisons rather than a single score derived
> > > from the difference calculated at a particular grid point.
Instead
> > > of requiring that a "hit" be defined by
> > the
> > > forecast-observation difference of 0 at each grid point, the use
> > > of a neighborhood for the observed field allows you to define a
"hit"
> > > if the forecast value falls within a range of the observed
values
> > > included in
> > the
> > > neighborhood. Further, if the same neighborhood also applies to
> > > the forecast field, then, that too, provides a range of forecast
> > > values which can be considered "hits' when compared to the range
> > > of observed values.
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > Mr John W. Raby, Meteorologist
> > > U.S. Army Research Laboratory
> > > White Sands Missile Range, NM 88002
> > > (575) 678-2004 DSN 258-2004
> > > FAX (575) 678-1230 DSN 258-1230
> > > Email: john.w.raby2.civ at mail.mil
> > >
> > >
> > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>


Classification: UNCLASSIFIED
Caveats: NONE



------------------------------------------------
Subject: Questions on Neighborhood Method (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Jun 12 11:51:56 2014

John,

Sounds good.  I'm glad that you're finding the MET tools to be useful.
If
you have gridded "observation" data available to you, you might also
consider using the series_analysis tool.  Here's how it differs from
Grid-Stat.  Grid-Stat is run at a single point in time to compute
aerial
averages of statistics over some geographic region.  You can choose
the
masking regions to be as large or small as you'd like, but Grid-Stat
is
computing statistics over some number of grid points in that region.

Series-Analysis, on the other hand, computes a statistic for each
individual grid point.  You pass it a list of gridded forecast files
and a
list of matching gridded observation files.  Typically, it'd be a time
series of forecast/observation pairs.  Rather than computing aerial
averages, it computes one or more statistics for each grid point.  If
you
pass it a month's worth of daily data, it would compute statistics for
each
grid point over about 30 matched pairs.

The purpose of doing this is to see how model performance varies
spatially.  You might see that your model does well over certain
topography
and lousy over others.

Hope that helps.

Thanks,
John


On Thu, Jun 12, 2014 at 10:31 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> All I can say is that this exchange has been very helpful to me and
I
> appreciate your sharing the knowledge and experience on this topic.
We
> haven't
> used Grid-Stat other than for some limited testing, My intent is to
use it
> more to augment the info provided by Point-Stat and MODE on our high
> resolution WRF modeling project. Each tool assesses performance, but
in a
> unique way which captures a different aspect of performance. For our
work
> at
> high resolution, it is especially important to assess performance
using all
> the tools since you are more likely to reveal the true skill of the
model
> than
> if you only use one tool.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, June 12, 2014 10:19 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #67570] Questions on Neighborhood
Method
> (UNCLASSIFIED)
>
> John,
>
> Yes, since "field = BOTH;" that smoothing filter would be applied to
both
> the
> forecast and observation fields, and then they'd be compared grid-
point by
> grid-point.
>
> I our verification work here in the DTC, we have not really used the
> "interp"
> options much.  Instead, we use the default setting which says to do
no
> smoothing on either field.  However, we do use the "nbrhd" options
to
> compute
> FSS and see how it varies based on threshold and neighborhood size.
>
> Thanks,
> John
>
>
> On Thu, Jun 12, 2014 at 7:59 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > John -
> >
> > Thanks for correcting my understanding of the way the nbrhd and
the
> > interp sections are working independently. I need to think about
this
> > to thoroughly grasp what it means. Now, I can start thinking about
the
> > two separate types of scoring provided by the settings in these
two
> > sections.
> >
> > For the nbrhd section, the results (based on fractional coverage
which
> > is computed from hits or misses from categorical forecasts) are
> > summarized/captured in the FSS and FBS scores, but for the interp,
> > there is no overall "score" to summarize the results other than
the
> > statistics in the CNT, CTC, and CTS line types which include the
ME,
> > MAE and RMSE computed over the raw forecast and observation fields
> > directly.
> >
> > For both sections, depending on your settings, it's possible that
the
> > individual grid point values are assigned new values.
> >
> > In your example for the interp section, you mentioned that the
"raw
> > forecast value would be replaced by the average forecast value
using
> > the 25 closest points". Since your setting  was "BOTH" wouldn't
the
> > same happen to the raw observed values?
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Wednesday, June 11, 2014 4:01 PM
> > To: Raby, John W CIV USARMY ARL (US)
> > Subject: Re: [rt.rap.ucar.edu #67570] Questions on Neighborhood
Method
> > (UNCLASSIFIED)
> >
> > John,
> >
> > For your first question on "hits", your answer looks correct.
> >
> > For your other two questions about whether the grid point gets
> > assigned a new value, I would say the answer is yes.  When
applying
> > the neighborhood methods, the value at each forecast (and
observation)
> > grid point gets replaced by the fractional coverage within the
> > neighborhood around that point.  The FSS and FBS statistics are
> > computed over those fractional coverage values, not the raw
forecast
> > (and observation) values.  So the stats in the NBRCNT, NBRCTC, and
> > NBRCTS line types are computed over the fractional coverage fields
> > (derived from the raw forecast and observation fields).
> >
> > The stats in all the other line types (like CNT, CTC, and CTS) are
> > computed over the raw forecast and observation fields directly.
> >
> > I think you may be confusing two different sections of the config
file:
> > "nbrhd" vs "interp".
> >
> > The nbrhd section controls the logic when computing NBRCNT,
NBRCTC,
> > and NBRCTS output line types.
> >
> > The interp section is separate and does not interact with the
nbrhd
> > section.
> > It enables you to smooth either the forecast or observation field
or
> both.
> >  By
> > default, its set up to do no smoothing.  Suppose for example, you
> > setup the interp section like this:
> > interp = {
> >    field      = BOTH;
> >    vld_thresh = 1.0;
> >
> >    type = [
> >       {
> >          method = UW_MEAN;
> >          width  = 5;
> >       }
> >    ];
> > };
> >
> > The would apply a 5x5 smoothing box to each grid point.  In this
case,
> > the raw forecast value would be replaced by the average forecast
value
> > using the 25 closest points.  Then the output in the CNT, CTC, and
CTS
> > line types would be computed over those smoothed values.
> >
> > So smoothing using the "interp" settings and neighborhood methods
> > using the "nbrhd" settings are both available, but are different.
> >
> > Hope that helps.
> >
> > Thanks,
> > John
> >
> >
> >
> > On Wed, Jun 11, 2014 at 1:37 PM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
> > >
> > > John -
> > >
> > > Having your detailed explanation of how the method works is very
> helpful.
> > > It allows you to slowly understand it by putting each piece
together
> > > in the right order.
> > >
> > > So, my understanding now leads me to answer my questions as
follows:
> > >
> > > Q:Is the scoring considered a "hit" based on the continuous
> > > statistics error
> > > > calculations (i.e. forecast-observation pair errors) or based
on a
> > > whether
> > > > the category threshold condition was met?
> > > A: Scoring is based on categorical forecasts and categorical
> > observations.
> > > (0-miss and 1-hit where a hit is defined as an occurrence when
the
> > > forecast
> > > (observation) meets the threshold criteria.
> > >
> > > Q: Do the values of individual forecast grid points get assigned
a
> > > new value
> > > > based on the values of all the forecast grid points within the
> > > neighborhood
> > > > before the gridpoint to gridpoint scoring is performed so as
to
> > > > generate continuous statistics?
> > > A: No re-assigining of forecast grid point values.
> > >
> > > Q: What about the situation where the observation grid
> > > > points are assigned new values as well based on the
application of
> > > > the
> > > same
> > > > neighborhood?
> > > A:  No re-assigining of observation grid point values.
> > >
> > > Thanks for taking the time to explain the process.
> > >
> > > R/
> > > John
> > >
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Wednesday, June 11, 2014 1:04 PM
> > > To: Raby, John W CIV USARMY ARL (US)
> > > Subject: Re: [rt.rap.ucar.edu #67570] Questions on Neighborhood
> > > Method
> > > (UNCLASSIFIED)
> > >
> > > John,
> > >
> > > I read through your email and see that you have several
questions
> > > about how the neighborhood methods are implemented in the Grid-
Stat
> > > tool.  Rather than addressing each individual point in your
email,
> > > let me first try to lay out the steps that are applied in Grid-
Stat.
> > > After reading this, please let me know if you have further
questions.
> > >
> > > (1) First, neighborhood methods are currently only available in
> > > grid_stat since they require both the forecast and observation
to be
> > > gridded (and on the same grid).  There are neighborhood methods
> > > defined for point observations, but those are not yet available
in MET.
> > >
> > > (2) The Fractions Skill Score (FSS) and Fractions Brier Score
(FBS)
> > > are the two most popular neighborhood methods statistics, and
they
> > > are contained in the NBRCNT (Neighborhood Continuous Statistics)
line
> > > type.
> > >
> > > (3) FSS and FBS are computed using two main parameters: a
threshold
> > > (cat_thresh in the config file) and a neighborhood size
("nbr.width"
> > > in the config file is the "width" of a square box, e.g. width =
3
> > > means a 3x3 box containing 9 points).  The neighborhood width
must
> > > be odd so that the box can be centered on each grid point.  A
NBRCNT
> > > output line will be generated for each combination of threshold
and
> > > neighborhood width.
> > >
> > > (4) The first step is to apply one of the thresholds from the
> > "cat_thresh"
> > > list to both the forecast and observation fields.  These steps
are
> > > applied equally to both fields - not one or the other.  This
> > > replaces the raw fields with fields of 0's and 1's... 1 where
the
> > > threshold criteria is met and 0 otherwise.
> > >
> > > (5) Next, for each grid point in the forecast field, draw an nxn
box
> > > around it (where n is the neighborhood width).  Count up the
number
> > > of 1's inside that box, and compute a ratio of the number of 1's
> > > divided by the size of the box (nxn) to get a number between 0
and
> > > 1.  We call this number a "fractional coverage" value for that
grid
> > > point.  For example, if the event occurred at 4 of the 9 grid
> > > points, the fractional coverage value would be
> > > 4/9 = 0.44.  Doing this for every grid point in the forecast
field
> > > generates a forecast "fractional coverage" field.
> > >
> > > As a small side note, for grid points near the edge of the grid
or
> > > near missing data values, not all of the nxn points in the
> > > neighborhood will contain valid data.  The "nbr.vld_thresh"
config
> > > file setting specifies what percentage of grid points must
contain
> > > valid data for a fractional coverage value to be computed.  A
value
> > > of
> > > 1.0 (the default) means that all nxn points must valid data for
> > > fractional coverage to be computed at the point.
> > >
> > > (6) Apply the same logic listed in (5) to the observation field
to
> > > compute an observation "fractional coverage" field.
> > >
> > > (7) At each grid point, we now have a forecast fractional
coverage
> > > value between 0 and 1 and an observation fractional coverage
value
> > > between 0 and 1.
> > >
> > > (8) The Fractions Skill Score and Fractions Brier Score are
computed
> > > using these fractional coverage values directly.  They are
written
> > > out in the NBRCNT line type.
> > >
> > > (9) MET also contains NBRCTC and NBRCTS line types, which are
not
> > > commonly used.  These are computed by applying a threshold
> > > ("nbrhd.cov_thresh" in the config file) to those fractional
coverage
> > > fields.  Applying a threshold converts those fractional coverage
> > > fields
> > back
> > > into fields of 0's and 1's.
> > > A 2x2 contingency table is computed over those thresholded
fields.
> > > The contingency table counts are stored in the NBRCTC line type,
and
> > > the corresponding statistics are stored in the NBRCTS line type.
> > > But as I said, these are not commonly used.
> > >
> > > As I mentioned, the FSS and FBS are the most commonly used
> > > neighborhood methods statistics.  For a given categorical
threshold,
> > > you'll find that as you increase the neighborhood size, the FSS
will
> > > increase as well.  FSS is often used to determine the scale at
which
> > > a forecast is skillful.  For example, a 4km model run will
contain a
> > > lot of great detail at the finest scales, but likely won't be
good
> > > at getting that detail exactly right.  For a particular
threshold of
> > > interest, you might compute FSS for several neighborhood sizes,
3x3,
> > > 5x5, 7x7, and so on.  Suppose you see that the model becomes
> > > skillful at the 5x5 neighborhood size.  So even though the model
has
> > > great detail at 4km, it's much more skill full at 20km
resolution.
> > > If you were making forecasts to the public, you might choose to
> > > upscale your model output to 20km prior to distributing it to
the
> > > public
> > > - otherwise, you'd be implying more confidence in the details of
the
> > > model than you actually have.
> > >
> > > Sorry for the long email!  Hopefully, that helps explain it.
Just
> > > let us know if you have more questions.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > > On Tue, Jun 10, 2014 at 11:26 AM, Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Tue Jun 10 11:26:28 2014: Request 67570 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > >        Queue: met_help
> > > >      Subject: Questions on Neighborhood Method (UNCLASSIFIED)
> > > >        Owner: Nobody
> > > >   Requestors: john.w.raby2.civ at mail.mil
> > > >       Status: new
> > > >  Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > > I am trying to understand the application of the MET Grid-Stat
> > > neighborhood
> > > > method for the calculation of neighborhood continuous
statistics
> > > > for
> > > scalar
> > > > variables such as 2m AGL temperature or 2m AGL relative
humidity.
> > > > I captured the following paragraph from the MET V4.1 User's
Guide.
> > > > I then tried to express in the paragraph below that, my
> > > > interpretation of the User's
> > > Guide
> > > > explanation. I also have the following questions which arose
as I
> > > > tried
> > > to
> > > > interpret the User's Guide:
> > > >
> > > > Is the scoring considered a "hit" based on the continuous
> > > > statistics
> > > error
> > > > calculations (i.e. forecast-observation pair errors) or based
on a
> > > whether
> > > > the category threshold condition was met?
> > > >
> > > > Do the values of individual forecast grid points get assigned
a
> > > > new value based on the values of all the forecast grid points
> > > > within the
> > > neighborhood
> > > > before the gridpoint to gridpoint scoring is performed so as
to
> > > > generate continuous statistics? What about the situation where
the
> > > > observation
> > > grid
> > > > points are assigned new values as well based on the
application of
> > > > the
> > > same
> > > > neighborhood?
> > > >
> > > > "MET also incorporates several neighborhood methods to give
credit
> > > > to forecasts that are close to the observations, but not
> > > > necessarily exactly matched up in space. Also referred to as
"fuzzy"
> > > > verification methods, these methods do not just compare a
single
> > > > forecast at each grid point to a single observation at each
grid
> > > > point; they compare the forecasts and observations in a
> > > > neighborhood surrounding the point of interest. With the
> > > > neighborhood method, the user chooses a distance
> > > within
> > > > which the
> > > > forecast event can fall from the observed event and still be
> > > > considered a hit. In MET this is implemented by defining a
square
> > > > search window around each grid point.
> > > > Within the search window, the number of observed events is
> > > > compared to
> > > the
> > > > number of
> > > > Forecast events. In this way, credit is given to forecasts
that
> > > > are close to the Observations without requiring a strict match
> > > > between forecasted events
> > > and
> > > > observed
> > > > events at any particular grid point. The neighborhood methods
> > > > allow the user to see how forecast skill varies with
neighborhood
> > > > size and can help determine the smallest neighborhood size
that
> > > > can be used to give sufficiently accurate forecasts."
> > > >
> > > > My interpretation: The "neighborhood" can be applied to both
the
> > > > forecast field and the observed field as well as just one or
the
> > > > other. Within the neighborhood, a number of observed values
will
> > > > be compared to the same number of forecast values and the
scoring
> > > > is based on those comparisons rather than a single score
derived
> > > > from the difference calculated at a particular grid point.
Instead
> > > > of requiring that a "hit" be defined by
> > > the
> > > > forecast-observation difference of 0 at each grid point, the
use
> > > > of a neighborhood for the observed field allows you to define
a "hit"
> > > > if the forecast value falls within a range of the observed
values
> > > > included in
> > > the
> > > > neighborhood. Further, if the same neighborhood also applies
to
> > > > the forecast field, then, that too, provides a range of
forecast
> > > > values which can be considered "hits' when compared to the
range
> > > > of observed values.
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > >
> > > > Mr John W. Raby, Meteorologist
> > > > U.S. Army Research Laboratory
> > > > White Sands Missile Range, NM 88002
> > > > (575) 678-2004 DSN 258-2004
> > > > FAX (575) 678-1230 DSN 258-1230
> > > > Email: john.w.raby2.civ at mail.mil
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>

------------------------------------------------
Subject: Questions on Neighborhood Method (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Jun 12 13:01:56 2014

Classification: UNCLASSIFIED
Caveats: NONE

John -

I have used the NWS RTMA product with a 2.5 km resolution as gridded
observations. It is the NWS analysis of record intended for model
verification
and is independent enough to not to cause concern for being the "model
analysis grid" which is not considered appropriate for use as a true
gridded
observation data set.

I appreciate your explanation of the Series-Analysis tool. It really
lays it
out clearer than I had understood from seeing the notes for the last
MET release or the User's Guide. Now, you've peaked my interest and
I'll go
back to the User's Guide to learn more. We are attempting to try
verification
over sub-regions determined by land use types, terrain slope and other
geographic features to shoe the true skill of high resolution models.
We are
ingesting the MPR output of Point-Stat into a GIS to figure out a way
to
analyze the errors to see what they tell us about what the appropriate
definitions of a sub-domain rather than arbitrarily picking it based
on
geographical type or topography.

Anxious to try Series-Analysis.

Thanks for sharing this info.

R/
John

-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, June 12, 2014 11:52 AM
To: Raby, John W CIV USARMY ARL (US)
Subject: Re: [rt.rap.ucar.edu #67570] Questions on Neighborhood Method
(UNCLASSIFIED)

John,

Sounds good.  I'm glad that you're finding the MET tools to be useful.
If you
have gridded "observation" data available to you, you might also
consider
using the series_analysis tool.  Here's how it differs from Grid-Stat.
Grid-Stat is run at a single point in time to compute aerial averages
of
statistics over some geographic region.  You can choose the masking
regions to
be as large or small as you'd like, but Grid-Stat is computing
statistics over
some number of grid points in that region.

Series-Analysis, on the other hand, computes a statistic for each
individual
grid point.  You pass it a list of gridded forecast files and a list
of
matching gridded observation files.  Typically, it'd be a time series
of
forecast/observation pairs.  Rather than computing aerial averages, it
computes one or more statistics for each grid point.  If you pass it a
month's
worth of daily data, it would compute statistics for each grid point
over
about 30 matched pairs.

The purpose of doing this is to see how model performance varies
spatially.
You might see that your model does well over certain topography and
lousy over
others.

Hope that helps.

Thanks,
John


On Thu, Jun 12, 2014 at 10:31 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> All I can say is that this exchange has been very helpful to me and
I
> appreciate your sharing the knowledge and experience on this topic.
We
> haven't used Grid-Stat other than for some limited testing, My
intent
> is to use it more to augment the info provided by Point-Stat and
MODE
> on our high resolution WRF modeling project. Each tool assesses
> performance, but in a unique way which captures a different aspect
of
> performance. For our work at high resolution, it is especially
> important to assess performance using all the tools since you are
more
> likely to reveal the true skill of the model than if you only use
one
> tool.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, June 12, 2014 10:19 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #67570] Questions on Neighborhood
Method
> (UNCLASSIFIED)
>
> John,
>
> Yes, since "field = BOTH;" that smoothing filter would be applied to
> both the forecast and observation fields, and then they'd be
compared
> grid-point by grid-point.
>
> I our verification work here in the DTC, we have not really used the
> "interp"
> options much.  Instead, we use the default setting which says to do
no
> smoothing on either field.  However, we do use the "nbrhd" options
to
> compute FSS and see how it varies based on threshold and
neighborhood
> size.
>
> Thanks,
> John
>
>
> On Thu, Jun 12, 2014 at 7:59 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > John -
> >
> > Thanks for correcting my understanding of the way the nbrhd and
the
> > interp sections are working independently. I need to think about
> > this to thoroughly grasp what it means. Now, I can start thinking
> > about the two separate types of scoring provided by the settings
in
> > these two sections.
> >
> > For the nbrhd section, the results (based on fractional coverage
> > which is computed from hits or misses from categorical forecasts)
> > are summarized/captured in the FSS and FBS scores, but for the
> > interp, there is no overall "score" to summarize the results other
> > than the statistics in the CNT, CTC, and CTS line types which
> > include the ME, MAE and RMSE computed over the raw forecast and
> > observation fields directly.
> >
> > For both sections, depending on your settings, it's possible that
> > the individual grid point values are assigned new values.
> >
> > In your example for the interp section, you mentioned that the
"raw
> > forecast value would be replaced by the average forecast value
using
> > the 25 closest points". Since your setting  was "BOTH" wouldn't
the
> > same happen to the raw observed values?
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Wednesday, June 11, 2014 4:01 PM
> > To: Raby, John W CIV USARMY ARL (US)
> > Subject: Re: [rt.rap.ucar.edu #67570] Questions on Neighborhood
> > Method
> > (UNCLASSIFIED)
> >
> > John,
> >
> > For your first question on "hits", your answer looks correct.
> >
> > For your other two questions about whether the grid point gets
> > assigned a new value, I would say the answer is yes.  When
applying
> > the neighborhood methods, the value at each forecast (and
> > observation) grid point gets replaced by the fractional coverage
> > within the neighborhood around that point.  The FSS and FBS
> > statistics are computed over those fractional coverage values, not
> > the raw forecast (and observation) values.  So the stats in the
> > NBRCNT, NBRCTC, and NBRCTS line types are computed over the
> > fractional coverage fields (derived from the raw forecast and
observation
> > fields).
> >
> > The stats in all the other line types (like CNT, CTC, and CTS) are
> > computed over the raw forecast and observation fields directly.
> >
> > I think you may be confusing two different sections of the config
file:
> > "nbrhd" vs "interp".
> >
> > The nbrhd section controls the logic when computing NBRCNT,
NBRCTC,
> > and NBRCTS output line types.
> >
> > The interp section is separate and does not interact with the
nbrhd
> > section.
> > It enables you to smooth either the forecast or observation field
or
> both.
> >  By
> > default, its set up to do no smoothing.  Suppose for example, you
> > setup the interp section like this:
> > interp = {
> >    field      = BOTH;
> >    vld_thresh = 1.0;
> >
> >    type = [
> >       {
> >          method = UW_MEAN;
> >          width  = 5;
> >       }
> >    ];
> > };
> >
> > The would apply a 5x5 smoothing box to each grid point.  In this
> > case, the raw forecast value would be replaced by the average
> > forecast value using the 25 closest points.  Then the output in
the
> > CNT, CTC, and CTS line types would be computed over those smoothed
values.
> >
> > So smoothing using the "interp" settings and neighborhood methods
> > using the "nbrhd" settings are both available, but are different.
> >
> > Hope that helps.
> >
> > Thanks,
> > John
> >
> >
> >
> > On Wed, Jun 11, 2014 at 1:37 PM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
> > >
> > > John -
> > >
> > > Having your detailed explanation of how the method works is very
> helpful.
> > > It allows you to slowly understand it by putting each piece
> > > together in the right order.
> > >
> > > So, my understanding now leads me to answer my questions as
follows:
> > >
> > > Q:Is the scoring considered a "hit" based on the continuous
> > > statistics error
> > > > calculations (i.e. forecast-observation pair errors) or based
on
> > > > a
> > > whether
> > > > the category threshold condition was met?
> > > A: Scoring is based on categorical forecasts and categorical
> > observations.
> > > (0-miss and 1-hit where a hit is defined as an occurrence when
the
> > > forecast
> > > (observation) meets the threshold criteria.
> > >
> > > Q: Do the values of individual forecast grid points get assigned
a
> > > new value
> > > > based on the values of all the forecast grid points within the
> > > neighborhood
> > > > before the gridpoint to gridpoint scoring is performed so as
to
> > > > generate continuous statistics?
> > > A: No re-assigining of forecast grid point values.
> > >
> > > Q: What about the situation where the observation grid
> > > > points are assigned new values as well based on the
application
> > > > of the
> > > same
> > > > neighborhood?
> > > A:  No re-assigining of observation grid point values.
> > >
> > > Thanks for taking the time to explain the process.
> > >
> > > R/
> > > John
> > >
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Wednesday, June 11, 2014 1:04 PM
> > > To: Raby, John W CIV USARMY ARL (US)
> > > Subject: Re: [rt.rap.ucar.edu #67570] Questions on Neighborhood
> > > Method
> > > (UNCLASSIFIED)
> > >
> > > John,
> > >
> > > I read through your email and see that you have several
questions
> > > about how the neighborhood methods are implemented in the
> > > Grid-Stat tool.  Rather than addressing each individual point in
> > > your email, let me first try to lay out the steps that are
applied in
> > > Grid-Stat.
> > > After reading this, please let me know if you have further
questions.
> > >
> > > (1) First, neighborhood methods are currently only available in
> > > grid_stat since they require both the forecast and observation
to
> > > be gridded (and on the same grid).  There are neighborhood
methods
> > > defined for point observations, but those are not yet available
in MET.
> > >
> > > (2) The Fractions Skill Score (FSS) and Fractions Brier Score
> > > (FBS) are the two most popular neighborhood methods statistics,
> > > and they are contained in the NBRCNT (Neighborhood Continuous
> > > Statistics) line type.
> > >
> > > (3) FSS and FBS are computed using two main parameters: a
> > > threshold (cat_thresh in the config file) and a neighborhood
size
> > > ("nbr.width"
> > > in the config file is the "width" of a square box, e.g. width =
3
> > > means a 3x3 box containing 9 points).  The neighborhood width
must
> > > be odd so that the box can be centered on each grid point.  A
> > > NBRCNT output line will be generated for each combination of
> > > threshold and neighborhood width.
> > >
> > > (4) The first step is to apply one of the thresholds from the
> > "cat_thresh"
> > > list to both the forecast and observation fields.  These steps
are
> > > applied equally to both fields - not one or the other.  This
> > > replaces the raw fields with fields of 0's and 1's... 1 where
the
> > > threshold criteria is met and 0 otherwise.
> > >
> > > (5) Next, for each grid point in the forecast field, draw an nxn
> > > box around it (where n is the neighborhood width).  Count up the
> > > number of 1's inside that box, and compute a ratio of the number
> > > of 1's divided by the size of the box (nxn) to get a number
> > > between 0 and 1.  We call this number a "fractional coverage"
> > > value for that grid point.  For example, if the event occurred
at
> > > 4 of the 9 grid points, the fractional coverage value would be
> > > 4/9 = 0.44.  Doing this for every grid point in the forecast
field
> > > generates a forecast "fractional coverage" field.
> > >
> > > As a small side note, for grid points near the edge of the grid
or
> > > near missing data values, not all of the nxn points in the
> > > neighborhood will contain valid data.  The "nbr.vld_thresh"
config
> > > file setting specifies what percentage of grid points must
contain
> > > valid data for a fractional coverage value to be computed.  A
> > > value of
> > > 1.0 (the default) means that all nxn points must valid data for
> > > fractional coverage to be computed at the point.
> > >
> > > (6) Apply the same logic listed in (5) to the observation field
to
> > > compute an observation "fractional coverage" field.
> > >
> > > (7) At each grid point, we now have a forecast fractional
coverage
> > > value between 0 and 1 and an observation fractional coverage
value
> > > between 0 and 1.
> > >
> > > (8) The Fractions Skill Score and Fractions Brier Score are
> > > computed using these fractional coverage values directly.  They
> > > are written out in the NBRCNT line type.
> > >
> > > (9) MET also contains NBRCTC and NBRCTS line types, which are
not
> > > commonly used.  These are computed by applying a threshold
> > > ("nbrhd.cov_thresh" in the config file) to those fractional
> > > coverage fields.  Applying a threshold converts those fractional
> > > coverage fields
> > back
> > > into fields of 0's and 1's.
> > > A 2x2 contingency table is computed over those thresholded
fields.
> > > The contingency table counts are stored in the NBRCTC line type,
> > > and the corresponding statistics are stored in the NBRCTS line
type.
> > > But as I said, these are not commonly used.
> > >
> > > As I mentioned, the FSS and FBS are the most commonly used
> > > neighborhood methods statistics.  For a given categorical
> > > threshold, you'll find that as you increase the neighborhood
size,
> > > the FSS will increase as well.  FSS is often used to determine
the
> > > scale at which a forecast is skillful.  For example, a 4km model
> > > run will contain a lot of great detail at the finest scales, but
> > > likely won't be good at getting that detail exactly right.  For
a
> > > particular threshold of interest, you might compute FSS for
> > > several neighborhood sizes, 3x3, 5x5, 7x7, and so on.  Suppose
you
> > > see that the model becomes skillful at the 5x5 neighborhood
size.
> > > So even though the model has great detail at 4km, it's much more
skill
> > > full at 20km resolution.
> > > If you were making forecasts to the public, you might choose to
> > > upscale your model output to 20km prior to distributing it to
the
> > > public
> > > - otherwise, you'd be implying more confidence in the details of
> > > the model than you actually have.
> > >
> > > Sorry for the long email!  Hopefully, that helps explain it.
Just
> > > let us know if you have more questions.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > > On Tue, Jun 10, 2014 at 11:26 AM, Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Tue Jun 10 11:26:28 2014: Request 67570 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > >        Queue: met_help
> > > >      Subject: Questions on Neighborhood Method (UNCLASSIFIED)
> > > >        Owner: Nobody
> > > >   Requestors: john.w.raby2.civ at mail.mil
> > > >       Status: new
> > > >  Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > > I am trying to understand the application of the MET Grid-Stat
> > > neighborhood
> > > > method for the calculation of neighborhood continuous
statistics
> > > > for
> > > scalar
> > > > variables such as 2m AGL temperature or 2m AGL relative
humidity.
> > > > I captured the following paragraph from the MET V4.1 User's
Guide.
> > > > I then tried to express in the paragraph below that, my
> > > > interpretation of the User's
> > > Guide
> > > > explanation. I also have the following questions which arose
as
> > > > I tried
> > > to
> > > > interpret the User's Guide:
> > > >
> > > > Is the scoring considered a "hit" based on the continuous
> > > > statistics
> > > error
> > > > calculations (i.e. forecast-observation pair errors) or based
on
> > > > a
> > > whether
> > > > the category threshold condition was met?
> > > >
> > > > Do the values of individual forecast grid points get assigned
a
> > > > new value based on the values of all the forecast grid points
> > > > within the
> > > neighborhood
> > > > before the gridpoint to gridpoint scoring is performed so as
to
> > > > generate continuous statistics? What about the situation where
> > > > the observation
> > > grid
> > > > points are assigned new values as well based on the
application
> > > > of the
> > > same
> > > > neighborhood?
> > > >
> > > > "MET also incorporates several neighborhood methods to give
> > > > credit to forecasts that are close to the observations, but
not
> > > > necessarily exactly matched up in space. Also referred to as
"fuzzy"
> > > > verification methods, these methods do not just compare a
single
> > > > forecast at each grid point to a single observation at each
grid
> > > > point; they compare the forecasts and observations in a
> > > > neighborhood surrounding the point of interest. With the
> > > > neighborhood method, the user chooses a distance
> > > within
> > > > which the
> > > > forecast event can fall from the observed event and still be
> > > > considered a hit. In MET this is implemented by defining a
> > > > square search window around each grid point.
> > > > Within the search window, the number of observed events is
> > > > compared to
> > > the
> > > > number of
> > > > Forecast events. In this way, credit is given to forecasts
that
> > > > are close to the Observations without requiring a strict match
> > > > between forecasted events
> > > and
> > > > observed
> > > > events at any particular grid point. The neighborhood methods
> > > > allow the user to see how forecast skill varies with
> > > > neighborhood size and can help determine the smallest
> > > > neighborhood size that can be used to give sufficiently
accurate
> > > > forecasts."
> > > >
> > > > My interpretation: The "neighborhood" can be applied to both
the
> > > > forecast field and the observed field as well as just one or
the
> > > > other. Within the neighborhood, a number of observed values
will
> > > > be compared to the same number of forecast values and the
> > > > scoring is based on those comparisons rather than a single
score
> > > > derived from the difference calculated at a particular grid
> > > > point. Instead of requiring that a "hit" be defined by
> > > the
> > > > forecast-observation difference of 0 at each grid point, the
use
> > > > of a neighborhood for the observed field allows you to define
a "hit"
> > > > if the forecast value falls within a range of the observed
> > > > values included in
> > > the
> > > > neighborhood. Further, if the same neighborhood also applies
to
> > > > the forecast field, then, that too, provides a range of
forecast
> > > > values which can be considered "hits' when compared to the
range
> > > > of observed values.
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > >
> > > > Mr John W. Raby, Meteorologist
> > > > U.S. Army Research Laboratory
> > > > White Sands Missile Range, NM 88002
> > > > (575) 678-2004 DSN 258-2004
> > > > FAX (575) 678-1230 DSN 258-1230
> > > > Email: john.w.raby2.civ at mail.mil
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Classification: UNCLASSIFIED
> > > > Caveats: NONE
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>


Classification: UNCLASSIFIED
Caveats: NONE



------------------------------------------------
Subject: Questions on Neighborhood Method (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Jun 12 13:43:47 2014

John,

Happy to help.  I'll go ahead and resolve this ticket.  Please just
write
us a new email if more questions arise.

Thanks,
John


On Thu, Jun 12, 2014 at 1:01 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> John -
>
> I have used the NWS RTMA product with a 2.5 km resolution as gridded
> observations. It is the NWS analysis of record intended for model
> verification
> and is independent enough to not to cause concern for being the
"model
> analysis grid" which is not considered appropriate for use as a true
> gridded
> observation data set.
>
> I appreciate your explanation of the Series-Analysis tool. It really
lays
> it
> out clearer than I had understood from seeing the notes for the last
> MET release or the User's Guide. Now, you've peaked my interest and
I'll go
> back to the User's Guide to learn more. We are attempting to try
> verification
> over sub-regions determined by land use types, terrain slope and
other
> geographic features to shoe the true skill of high resolution
models. We
> are
> ingesting the MPR output of Point-Stat into a GIS to figure out a
way to
> analyze the errors to see what they tell us about what the
appropriate
> definitions of a sub-domain rather than arbitrarily picking it based
on
> geographical type or topography.
>
> Anxious to try Series-Analysis.
>
> Thanks for sharing this info.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, June 12, 2014 11:52 AM
> To: Raby, John W CIV USARMY ARL (US)
> Subject: Re: [rt.rap.ucar.edu #67570] Questions on Neighborhood
Method
> (UNCLASSIFIED)
>
> John,
>
> Sounds good.  I'm glad that you're finding the MET tools to be
useful.  If
> you
> have gridded "observation" data available to you, you might also
consider
> using the series_analysis tool.  Here's how it differs from Grid-
Stat.
> Grid-Stat is run at a single point in time to compute aerial
averages of
> statistics over some geographic region.  You can choose the masking
> regions to
> be as large or small as you'd like, but Grid-Stat is computing
statistics
> over
> some number of grid points in that region.
>
> Series-Analysis, on the other hand, computes a statistic for each
> individual
> grid point.  You pass it a list of gridded forecast files and a list
of
> matching gridded observation files.  Typically, it'd be a time
series of
> forecast/observation pairs.  Rather than computing aerial averages,
it
> computes one or more statistics for each grid point.  If you pass it
a
> month's
> worth of daily data, it would compute statistics for each grid point
over
> about 30 matched pairs.
>
> The purpose of doing this is to see how model performance varies
spatially.
> You might see that your model does well over certain topography and
lousy
> over
> others.
>
> Hope that helps.
>
> Thanks,
> John
>
>
> On Thu, Jun 12, 2014 at 10:31 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> > John -
> >
> > All I can say is that this exchange has been very helpful to me
and I
> > appreciate your sharing the knowledge and experience on this
topic. We
> > haven't used Grid-Stat other than for some limited testing, My
intent
> > is to use it more to augment the info provided by Point-Stat and
MODE
> > on our high resolution WRF modeling project. Each tool assesses
> > performance, but in a unique way which captures a different aspect
of
> > performance. For our work at high resolution, it is especially
> > important to assess performance using all the tools since you are
more
> > likely to reveal the true skill of the model than if you only use
one
> > tool.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Thursday, June 12, 2014 10:19 AM
> > To: Raby, John W CIV USARMY ARL (US)
> > Subject: Re: [rt.rap.ucar.edu #67570] Questions on Neighborhood
Method
> > (UNCLASSIFIED)
> >
> > John,
> >
> > Yes, since "field = BOTH;" that smoothing filter would be applied
to
> > both the forecast and observation fields, and then they'd be
compared
> > grid-point by grid-point.
> >
> > I our verification work here in the DTC, we have not really used
the
> > "interp"
> > options much.  Instead, we use the default setting which says to
do no
> > smoothing on either field.  However, we do use the "nbrhd" options
to
> > compute FSS and see how it varies based on threshold and
neighborhood
> > size.
> >
> > Thanks,
> > John
> >
> >
> > On Thu, Jun 12, 2014 at 7:59 AM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > > John -
> > >
> > > Thanks for correcting my understanding of the way the nbrhd and
the
> > > interp sections are working independently. I need to think about
> > > this to thoroughly grasp what it means. Now, I can start
thinking
> > > about the two separate types of scoring provided by the settings
in
> > > these two sections.
> > >
> > > For the nbrhd section, the results (based on fractional coverage
> > > which is computed from hits or misses from categorical
forecasts)
> > > are summarized/captured in the FSS and FBS scores, but for the
> > > interp, there is no overall "score" to summarize the results
other
> > > than the statistics in the CNT, CTC, and CTS line types which
> > > include the ME, MAE and RMSE computed over the raw forecast and
> > > observation fields directly.
> > >
> > > For both sections, depending on your settings, it's possible
that
> > > the individual grid point values are assigned new values.
> > >
> > > In your example for the interp section, you mentioned that the
"raw
> > > forecast value would be replaced by the average forecast value
using
> > > the 25 closest points". Since your setting  was "BOTH" wouldn't
the
> > > same happen to the raw observed values?
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Wednesday, June 11, 2014 4:01 PM
> > > To: Raby, John W CIV USARMY ARL (US)
> > > Subject: Re: [rt.rap.ucar.edu #67570] Questions on Neighborhood
> > > Method
> > > (UNCLASSIFIED)
> > >
> > > John,
> > >
> > > For your first question on "hits", your answer looks correct.
> > >
> > > For your other two questions about whether the grid point gets
> > > assigned a new value, I would say the answer is yes.  When
applying
> > > the neighborhood methods, the value at each forecast (and
> > > observation) grid point gets replaced by the fractional coverage
> > > within the neighborhood around that point.  The FSS and FBS
> > > statistics are computed over those fractional coverage values,
not
> > > the raw forecast (and observation) values.  So the stats in the
> > > NBRCNT, NBRCTC, and NBRCTS line types are computed over the
> > > fractional coverage fields (derived from the raw forecast and
> observation
> > > fields).
> > >
> > > The stats in all the other line types (like CNT, CTC, and CTS)
are
> > > computed over the raw forecast and observation fields directly.
> > >
> > > I think you may be confusing two different sections of the
config file:
> > > "nbrhd" vs "interp".
> > >
> > > The nbrhd section controls the logic when computing NBRCNT,
NBRCTC,
> > > and NBRCTS output line types.
> > >
> > > The interp section is separate and does not interact with the
nbrhd
> > > section.
> > > It enables you to smooth either the forecast or observation
field or
> > both.
> > >  By
> > > default, its set up to do no smoothing.  Suppose for example,
you
> > > setup the interp section like this:
> > > interp = {
> > >    field      = BOTH;
> > >    vld_thresh = 1.0;
> > >
> > >    type = [
> > >       {
> > >          method = UW_MEAN;
> > >          width  = 5;
> > >       }
> > >    ];
> > > };
> > >
> > > The would apply a 5x5 smoothing box to each grid point.  In this
> > > case, the raw forecast value would be replaced by the average
> > > forecast value using the 25 closest points.  Then the output in
the
> > > CNT, CTC, and CTS line types would be computed over those
smoothed
> values.
> > >
> > > So smoothing using the "interp" settings and neighborhood
methods
> > > using the "nbrhd" settings are both available, but are
different.
> > >
> > > Hope that helps.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > > On Wed, Jun 11, 2014 at 1:37 PM, Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570
>
> > > >
> > > > John -
> > > >
> > > > Having your detailed explanation of how the method works is
very
> > helpful.
> > > > It allows you to slowly understand it by putting each piece
> > > > together in the right order.
> > > >
> > > > So, my understanding now leads me to answer my questions as
follows:
> > > >
> > > > Q:Is the scoring considered a "hit" based on the continuous
> > > > statistics error
> > > > > calculations (i.e. forecast-observation pair errors) or
based on
> > > > > a
> > > > whether
> > > > > the category threshold condition was met?
> > > > A: Scoring is based on categorical forecasts and categorical
> > > observations.
> > > > (0-miss and 1-hit where a hit is defined as an occurrence when
the
> > > > forecast
> > > > (observation) meets the threshold criteria.
> > > >
> > > > Q: Do the values of individual forecast grid points get
assigned a
> > > > new value
> > > > > based on the values of all the forecast grid points within
the
> > > > neighborhood
> > > > > before the gridpoint to gridpoint scoring is performed so as
to
> > > > > generate continuous statistics?
> > > > A: No re-assigining of forecast grid point values.
> > > >
> > > > Q: What about the situation where the observation grid
> > > > > points are assigned new values as well based on the
application
> > > > > of the
> > > > same
> > > > > neighborhood?
> > > > A:  No re-assigining of observation grid point values.
> > > >
> > > > Thanks for taking the time to explain the process.
> > > >
> > > > R/
> > > > John
> > > >
> > > > ________________________________________
> > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > Sent: Wednesday, June 11, 2014 1:04 PM
> > > > To: Raby, John W CIV USARMY ARL (US)
> > > > Subject: Re: [rt.rap.ucar.edu #67570] Questions on
Neighborhood
> > > > Method
> > > > (UNCLASSIFIED)
> > > >
> > > > John,
> > > >
> > > > I read through your email and see that you have several
questions
> > > > about how the neighborhood methods are implemented in the
> > > > Grid-Stat tool.  Rather than addressing each individual point
in
> > > > your email, let me first try to lay out the steps that are
applied in
> > > > Grid-Stat.
> > > > After reading this, please let me know if you have further
questions.
> > > >
> > > > (1) First, neighborhood methods are currently only available
in
> > > > grid_stat since they require both the forecast and observation
to
> > > > be gridded (and on the same grid).  There are neighborhood
methods
> > > > defined for point observations, but those are not yet
available in
> MET.
> > > >
> > > > (2) The Fractions Skill Score (FSS) and Fractions Brier Score
> > > > (FBS) are the two most popular neighborhood methods
statistics,
> > > > and they are contained in the NBRCNT (Neighborhood Continuous
> > > > Statistics) line type.
> > > >
> > > > (3) FSS and FBS are computed using two main parameters: a
> > > > threshold (cat_thresh in the config file) and a neighborhood
size
> > > > ("nbr.width"
> > > > in the config file is the "width" of a square box, e.g. width
= 3
> > > > means a 3x3 box containing 9 points).  The neighborhood width
must
> > > > be odd so that the box can be centered on each grid point.  A
> > > > NBRCNT output line will be generated for each combination of
> > > > threshold and neighborhood width.
> > > >
> > > > (4) The first step is to apply one of the thresholds from the
> > > "cat_thresh"
> > > > list to both the forecast and observation fields.  These steps
are
> > > > applied equally to both fields - not one or the other.  This
> > > > replaces the raw fields with fields of 0's and 1's... 1 where
the
> > > > threshold criteria is met and 0 otherwise.
> > > >
> > > > (5) Next, for each grid point in the forecast field, draw an
nxn
> > > > box around it (where n is the neighborhood width).  Count up
the
> > > > number of 1's inside that box, and compute a ratio of the
number
> > > > of 1's divided by the size of the box (nxn) to get a number
> > > > between 0 and 1.  We call this number a "fractional coverage"
> > > > value for that grid point.  For example, if the event occurred
at
> > > > 4 of the 9 grid points, the fractional coverage value would be
> > > > 4/9 = 0.44.  Doing this for every grid point in the forecast
field
> > > > generates a forecast "fractional coverage" field.
> > > >
> > > > As a small side note, for grid points near the edge of the
grid or
> > > > near missing data values, not all of the nxn points in the
> > > > neighborhood will contain valid data.  The "nbr.vld_thresh"
config
> > > > file setting specifies what percentage of grid points must
contain
> > > > valid data for a fractional coverage value to be computed.  A
> > > > value of
> > > > 1.0 (the default) means that all nxn points must valid data
for
> > > > fractional coverage to be computed at the point.
> > > >
> > > > (6) Apply the same logic listed in (5) to the observation
field to
> > > > compute an observation "fractional coverage" field.
> > > >
> > > > (7) At each grid point, we now have a forecast fractional
coverage
> > > > value between 0 and 1 and an observation fractional coverage
value
> > > > between 0 and 1.
> > > >
> > > > (8) The Fractions Skill Score and Fractions Brier Score are
> > > > computed using these fractional coverage values directly.
They
> > > > are written out in the NBRCNT line type.
> > > >
> > > > (9) MET also contains NBRCTC and NBRCTS line types, which are
not
> > > > commonly used.  These are computed by applying a threshold
> > > > ("nbrhd.cov_thresh" in the config file) to those fractional
> > > > coverage fields.  Applying a threshold converts those
fractional
> > > > coverage fields
> > > back
> > > > into fields of 0's and 1's.
> > > > A 2x2 contingency table is computed over those thresholded
fields.
> > > > The contingency table counts are stored in the NBRCTC line
type,
> > > > and the corresponding statistics are stored in the NBRCTS line
type.
> > > > But as I said, these are not commonly used.
> > > >
> > > > As I mentioned, the FSS and FBS are the most commonly used
> > > > neighborhood methods statistics.  For a given categorical
> > > > threshold, you'll find that as you increase the neighborhood
size,
> > > > the FSS will increase as well.  FSS is often used to determine
the
> > > > scale at which a forecast is skillful.  For example, a 4km
model
> > > > run will contain a lot of great detail at the finest scales,
but
> > > > likely won't be good at getting that detail exactly right.
For a
> > > > particular threshold of interest, you might compute FSS for
> > > > several neighborhood sizes, 3x3, 5x5, 7x7, and so on.  Suppose
you
> > > > see that the model becomes skillful at the 5x5 neighborhood
size.
> > > > So even though the model has great detail at 4km, it's much
more
> skill
> > > > full at 20km resolution.
> > > > If you were making forecasts to the public, you might choose
to
> > > > upscale your model output to 20km prior to distributing it to
the
> > > > public
> > > > - otherwise, you'd be implying more confidence in the details
of
> > > > the model than you actually have.
> > > >
> > > > Sorry for the long email!  Hopefully, that helps explain it.
Just
> > > > let us know if you have more questions.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > >
> > > > On Tue, Jun 10, 2014 at 11:26 AM, Raby, John W USA CIV via RT
<
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Tue Jun 10 11:26:28 2014: Request 67570 was acted upon.
> > > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > >        Queue: met_help
> > > > >      Subject: Questions on Neighborhood Method
(UNCLASSIFIED)
> > > > >        Owner: Nobody
> > > > >   Requestors: john.w.raby2.civ at mail.mil
> > > > >       Status: new
> > > > >  Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=67570 >
> > > > >
> > > > >
> > > > > Classification: UNCLASSIFIED
> > > > > Caveats: NONE
> > > > >
> > > > > I am trying to understand the application of the MET Grid-
Stat
> > > > neighborhood
> > > > > method for the calculation of neighborhood continuous
statistics
> > > > > for
> > > > scalar
> > > > > variables such as 2m AGL temperature or 2m AGL relative
humidity.
> > > > > I captured the following paragraph from the MET V4.1 User's
Guide.
> > > > > I then tried to express in the paragraph below that, my
> > > > > interpretation of the User's
> > > > Guide
> > > > > explanation. I also have the following questions which arose
as
> > > > > I tried
> > > > to
> > > > > interpret the User's Guide:
> > > > >
> > > > > Is the scoring considered a "hit" based on the continuous
> > > > > statistics
> > > > error
> > > > > calculations (i.e. forecast-observation pair errors) or
based on
> > > > > a
> > > > whether
> > > > > the category threshold condition was met?
> > > > >
> > > > > Do the values of individual forecast grid points get
assigned a
> > > > > new value based on the values of all the forecast grid
points
> > > > > within the
> > > > neighborhood
> > > > > before the gridpoint to gridpoint scoring is performed so as
to
> > > > > generate continuous statistics? What about the situation
where
> > > > > the observation
> > > > grid
> > > > > points are assigned new values as well based on the
application
> > > > > of the
> > > > same
> > > > > neighborhood?
> > > > >
> > > > > "MET also incorporates several neighborhood methods to give
> > > > > credit to forecasts that are close to the observations, but
not
> > > > > necessarily exactly matched up in space. Also referred to as
> "fuzzy"
> > > > > verification methods, these methods do not just compare a
single
> > > > > forecast at each grid point to a single observation at each
grid
> > > > > point; they compare the forecasts and observations in a
> > > > > neighborhood surrounding the point of interest. With the
> > > > > neighborhood method, the user chooses a distance
> > > > within
> > > > > which the
> > > > > forecast event can fall from the observed event and still be
> > > > > considered a hit. In MET this is implemented by defining a
> > > > > square search window around each grid point.
> > > > > Within the search window, the number of observed events is
> > > > > compared to
> > > > the
> > > > > number of
> > > > > Forecast events. In this way, credit is given to forecasts
that
> > > > > are close to the Observations without requiring a strict
match
> > > > > between forecasted events
> > > > and
> > > > > observed
> > > > > events at any particular grid point. The neighborhood
methods
> > > > > allow the user to see how forecast skill varies with
> > > > > neighborhood size and can help determine the smallest
> > > > > neighborhood size that can be used to give sufficiently
accurate
> > > > > forecasts."
> > > > >
> > > > > My interpretation: The "neighborhood" can be applied to both
the
> > > > > forecast field and the observed field as well as just one or
the
> > > > > other. Within the neighborhood, a number of observed values
will
> > > > > be compared to the same number of forecast values and the
> > > > > scoring is based on those comparisons rather than a single
score
> > > > > derived from the difference calculated at a particular grid
> > > > > point. Instead of requiring that a "hit" be defined by
> > > > the
> > > > > forecast-observation difference of 0 at each grid point, the
use
> > > > > of a neighborhood for the observed field allows you to
define a
> "hit"
> > > > > if the forecast value falls within a range of the observed
> > > > > values included in
> > > > the
> > > > > neighborhood. Further, if the same neighborhood also applies
to
> > > > > the forecast field, then, that too, provides a range of
forecast
> > > > > values which can be considered "hits' when compared to the
range
> > > > > of observed values.
> > > > >
> > > > > Thanks.
> > > > >
> > > > > R/
> > > > > John
> > > > >
> > > > > Mr John W. Raby, Meteorologist
> > > > > U.S. Army Research Laboratory
> > > > > White Sands Missile Range, NM 88002
> > > > > (575) 678-2004 DSN 258-2004
> > > > > FAX (575) 678-1230 DSN 258-1230
> > > > > Email: john.w.raby2.civ at mail.mil
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Classification: UNCLASSIFIED
> > > > > Caveats: NONE
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > Classification: UNCLASSIFIED
> > > Caveats: NONE
> > >
> > >
> > >
> > >
> >
> >
> > Classification: UNCLASSIFIED
> > Caveats: NONE
> >
> >
> >
> >
>
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
>
>

------------------------------------------------


More information about the Met_help mailing list