[Met_help] [rt.rap.ucar.edu #56984] History for Neighbourhood method for grid-stat tool

[John Halley Gotway via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hi,

When using the cov_thres[] in the neighbourhood method to calculate
traditional contingency table statistics:

Are only the forecast grids subjected to the algorithm/calculations that
use cov_thres[] (i.e., to define yes/no forecast within the search window)?

Or do the observation grids have to go through the "transformation" to
define within that window if it's a yes/no observation as well?

In other words, is it the case that only the observation value at the
centre the window is chosen to be compared to the "transformed" forecast
grids?

Thanks,
Raizan


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Re: [rt.rap.ucar.edu #56984] Neighbourhood method for grid-stat tool
From: John Halley Gotway
Time: Thu Jun 14 07:27:08 2012

Raizan,

Good question.

The answer is that the smoothing is done to both the forecast and
observation fields prior to computing neighborhood statistics.

Here are the basic steps that are performed in Grid-Stat for
neighborhood stats:

(1) For each field being verified, loop through the event thresholds
defined in "fcst_thresh" and "obs_thresh".
(2) For each threshold, compute a 0/1 bitmap field for the forecast
and observation fields - 1 at grid points where the threshold criteria
is met, and 0 otherwise.
(3) For each choice of neighborhood size defined in "nbr_width",
transform the forecast and observation bitmap fields into what we've
called "fractional coverage" fields - basically, the fraction of
points in the neighborhood that are 1.
(4) Compute continuous neighborhood statistics by comparing the
forecast and observation fractional coverage fields directly.  For
example, that's were the fractions skill score is defined.
(5) Lastly, apply the coverage threshold defined in "cov_thresh" to
those fractional coverage fields.  That defines another 0/1 bitmap
field.  And the neighborhood contingency table counts (NBRCTC)
and statistics (NBRCTS) are computed from those fields.

That's a lot of steps in the process - but they are applied in the
same way to both the forecast and observation fields!  I can tell you
that the FSS and FBS stats contained in the NBRCNT line are the
more conventional and typical neighborhood statistics that are used.
We added that last step of computing NBRCTC and NBRCTS because it was
a logical extension of the method.  But it's up to you to
interpret the results.

Hope that helps clarify.

Thanks,
John Halley Gotway
met_help at ucar.edu

On 06/14/2012 05:41 AM, Zan Rahmat via RT wrote:
>
> Thu Jun 14 05:41:48 2012: Request 56984 was acted upon.
> Transaction: Ticket created by raizan.rahmat at gmail.com
>         Queue: met_help
>       Subject: Neighbourhood method for grid-stat tool
>         Owner: Nobody
>    Requestors: raizan.rahmat at gmail.com
>        Status: new
>   Ticket<URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56984>
>
>
> Hi,
>
> When using the cov_thres[] in the neighbourhood method to calculate
> traditional contingency table statistics:
>
> Are only the forecast grids subjected to the algorithm/calculations
that
> use cov_thres[] (i.e., to define yes/no forecast within the search
window)?
>
> Or do the observation grids have to go through the "transformation"
to
> define within that window if it's a yes/no observation as well?
>
> In other words, is it the case that only the observation value at
the
> centre the window is chosen to be compared to the "transformed"
forecast
> grids?
>
> Thanks,
> Raizan

------------------------------------------------
Subject: Neighbourhood method for grid-stat tool
From: Zan Rahmat
Time: Thu Jun 14 07:41:41 2012

Dear John,

Thanks for the prompt and elaborate reply. Yes, that clarifies and
point
no. 3 answered my question. So, in a way, the methodology is less
strict
then since the transformation is both done on both forecast and
observation
grids. I would like to suggest that the Met User guide be updated to
reflect this fact if not too much of a hassle so that future users of
the
guide would benefit from the added clarity.

And it helps that you mentioned the FSS and FBS stats are more natural
scores to use. Because I'm still contemplating on what scores to use
since
there are a large number of statistics for me to look at given the
time so
that will help me to be more focused with my approach.

Anyhow, thanks for the great software as well!

Warm Regards,
Raizan

On 14 June 2012 14:27, John Halley Gotway via RT <met_help at ucar.edu>
wrote:

> Raizan,
>
> Good question.
>
> The answer is that the smoothing is done to both the forecast and
> observation fields prior to computing neighborhood statistics.
>
> Here are the basic steps that are performed in Grid-Stat for
neighborhood
> stats:
>
> (1) For each field being verified, loop through the event thresholds
> defined in "fcst_thresh" and "obs_thresh".
> (2) For each threshold, compute a 0/1 bitmap field for the forecast
and
> observation fields - 1 at grid points where the threshold criteria
is met,
> and 0 otherwise.
> (3) For each choice of neighborhood size defined in "nbr_width",
transform
> the forecast and observation bitmap fields into what we've called
> "fractional coverage" fields - basically, the fraction of
> points in the neighborhood that are 1.
> (4) Compute continuous neighborhood statistics by comparing the
forecast
> and observation fractional coverage fields directly.  For example,
that's
> were the fractions skill score is defined.
> (5) Lastly, apply the coverage threshold defined in "cov_thresh" to
those
> fractional coverage fields.  That defines another 0/1 bitmap field.
And
> the neighborhood contingency table counts (NBRCTC)
> and statistics (NBRCTS) are computed from those fields.
>
> That's a lot of steps in the process - but they are applied in the
same
> way to both the forecast and observation fields!  I can tell you
that the
> FSS and FBS stats contained in the NBRCNT line are the
> more conventional and typical neighborhood statistics that are used.
We
> added that last step of computing NBRCTC and NBRCTS because it was a
> logical extension of the method.  But it's up to you to
> interpret the results.
>
> Hope that helps clarify.
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu
>
> On 06/14/2012 05:41 AM, Zan Rahmat via RT wrote:
> >
> > Thu Jun 14 05:41:48 2012: Request 56984 was acted upon.
> > Transaction: Ticket created by raizan.rahmat at gmail.com
> >         Queue: met_help
> >       Subject: Neighbourhood method for grid-stat tool
> >         Owner: Nobody
> >    Requestors: raizan.rahmat at gmail.com
> >        Status: new
> >   Ticket<URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56984>
> >
> >
> > Hi,
> >
> > When using the cov_thres[] in the neighbourhood method to
calculate
> > traditional contingency table statistics:
> >
> > Are only the forecast grids subjected to the
algorithm/calculations that
> > use cov_thres[] (i.e., to define yes/no forecast within the search
> window)?
> >
> > Or do the observation grids have to go through the
"transformation" to
> > define within that window if it's a yes/no observation as well?
> >
> > In other words, is it the case that only the observation value at
the
> > centre the window is chosen to be compared to the "transformed"
forecast
> > grids?
> >
> > Thanks,
> > Raizan
>
>

------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #56984] Neighbourhood method for grid-stat tool
From: John Halley Gotway
Time: Thu Jun 14 07:46:24 2012

Raizan,

Glad that clarified it.  I've passed along your request for more
detail in the user's guide and the computation of neighborhood stats.

Thanks,
John

On 06/14/2012 07:41 AM, Zan Rahmat via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56984>
>
> Dear John,
>
> Thanks for the prompt and elaborate reply. Yes, that clarifies and
point
> no. 3 answered my question. So, in a way, the methodology is less
strict
> then since the transformation is both done on both forecast and
observation
> grids. I would like to suggest that the Met User guide be updated to
> reflect this fact if not too much of a hassle so that future users
of the
> guide would benefit from the added clarity.
>
> And it helps that you mentioned the FSS and FBS stats are more
natural
> scores to use. Because I'm still contemplating on what scores to use
since
> there are a large number of statistics for me to look at given the
time so
> that will help me to be more focused with my approach.
>
> Anyhow, thanks for the great software as well!
>
> Warm Regards,
> Raizan
>
> On 14 June 2012 14:27, John Halley Gotway via RT<met_help at ucar.edu>
wrote:
>
>> Raizan,
>>
>> Good question.
>>
>> The answer is that the smoothing is done to both the forecast and
>> observation fields prior to computing neighborhood statistics.
>>
>> Here are the basic steps that are performed in Grid-Stat for
neighborhood
>> stats:
>>
>> (1) For each field being verified, loop through the event
thresholds
>> defined in "fcst_thresh" and "obs_thresh".
>> (2) For each threshold, compute a 0/1 bitmap field for the forecast
and
>> observation fields - 1 at grid points where the threshold criteria
is met,
>> and 0 otherwise.
>> (3) For each choice of neighborhood size defined in "nbr_width",
transform
>> the forecast and observation bitmap fields into what we've called
>> "fractional coverage" fields - basically, the fraction of
>> points in the neighborhood that are 1.
>> (4) Compute continuous neighborhood statistics by comparing the
forecast
>> and observation fractional coverage fields directly.  For example,
that's
>> were the fractions skill score is defined.
>> (5) Lastly, apply the coverage threshold defined in "cov_thresh" to
those
>> fractional coverage fields.  That defines another 0/1 bitmap field.
And
>> the neighborhood contingency table counts (NBRCTC)
>> and statistics (NBRCTS) are computed from those fields.
>>
>> That's a lot of steps in the process - but they are applied in the
same
>> way to both the forecast and observation fields!  I can tell you
that the
>> FSS and FBS stats contained in the NBRCNT line are the
>> more conventional and typical neighborhood statistics that are
used.  We
>> added that last step of computing NBRCTC and NBRCTS because it was
a
>> logical extension of the method.  But it's up to you to
>> interpret the results.
>>
>> Hope that helps clarify.
>>
>> Thanks,
>> John Halley Gotway
>> met_help at ucar.edu
>>
>> On 06/14/2012 05:41 AM, Zan Rahmat via RT wrote:
>>>
>>> Thu Jun 14 05:41:48 2012: Request 56984 was acted upon.
>>> Transaction: Ticket created by raizan.rahmat at gmail.com
>>>          Queue: met_help
>>>        Subject: Neighbourhood method for grid-stat tool
>>>          Owner: Nobody
>>>     Requestors: raizan.rahmat at gmail.com
>>>         Status: new
>>>    Ticket<URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=56984>
>>>
>>>
>>> Hi,
>>>
>>> When using the cov_thres[] in the neighbourhood method to
calculate
>>> traditional contingency table statistics:
>>>
>>> Are only the forecast grids subjected to the
algorithm/calculations that
>>> use cov_thres[] (i.e., to define yes/no forecast within the search
>> window)?
>>>
>>> Or do the observation grids have to go through the
"transformation" to
>>> define within that window if it's a yes/no observation as well?
>>>
>>> In other words, is it the case that only the observation value at
the
>>> centre the window is chosen to be compared to the "transformed"
forecast
>>> grids?
>>>
>>> Thanks,
>>> Raizan
>>
>>

------------------------------------------------