[Met_help] [rt.rap.ucar.edu #46931] History for MODE: Gilbert Skill Score question
RAL HelpDesk {for John Halley Gotway}
met_help at ucar.edu
Fri Jun 17 14:11:41 MDT 2011
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello,
I just want to clarify that, in the output file from mode that contains the
CTSs, the first line of data (after the headings) contains the scores for
the RAW, unprocessed data. Thus, the GSS I see (equivalent to the equitable
threat score) is the true ETS based on standard contingency table statistics
from the unprocessed precip field, and wasn't computed from the precip field
after the threshold was applied, right? That's what the second line of data
that is called FILTER, right?
Jeff Duda
--
Jeff Duda
Iowa State University
Meteorology Graduate Student
3134 Agronomy Hall
www.meteor.iastate.edu/~jdduda
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #46931] MODE: Gilbert Skill Score question
From: John Halley Gotway
Time: Thu May 19 09:07:17 2011
Jeff,
The MODE CTS text file will always contain 4 lines in it:
(1) A header row
(2) A line for the "RAW" scores
(3) A line for the "FILTER" scores
(4) And a line for the "OBJECT" scores
Unless you've set the "fcst_raw_thresh" and "obs_raw_thresh" settings
in the MODE config file, the scores in the RAW and FILTER lines will
be identical. So the RAW scores are computed by looking at
the raw forecast and observation fields, applying the
"fcst_conv_thresh" and "obs_conv_thresh" thresholds to create 0/1
fields, computing a 2x2 contingency table, and then deriving
statistics from the
contingency table. The FILTER scores are computed in an identical
way, except that before thresholding the raw values, MODE first
applies the "fcst_raw_thresh" and "obs_raw_thresh" to zero out values
that don't meet those threshold criteria. The OBJECT scores are also
computed on a grid-point by grid-point basis, where each grid point is
considered "on" if it is inside an object and "off" if it
is not. Again those counts are put into a 2x2 contingency table and
statistics are derived.
The application of the "fcst_raw_thresh" and "obs_raw_thresh" may seem
odd but there's a good reason for them. At one point, we were
applying MODE to forecasts of reflectivity. However the algorithm
only forecasted areas of reflectivity >= 35 dBZ, whereas the
observations contained reflectivity values well below 35 dBZ.
Basically, the forecasts had areas of reflectivity >= 35 dBZ
surrounded by a
bunch of zeros, whereas the observations had areas of reflectivity >=
35 dBZ surrounded by values of 30, 25, 20, ..., and so on. The
problem arose in the MODE convolution step. Convolving the
forecast values with a bunch of zeros produced much smaller objects
that convolving the observations with non-zero values. So we used
these settings to filter out observations < 35 dBZ to make the
observation field look more similar to the forecast field. That way
the MODE object-definition process would work the same on both fields.
Sorry for all that detail, but that's the reason for it.
Most users will not need to use these settings unless they're in a
similar situation to what I've described.
And in general, since object-based verification approaches are still
pretty new, we provided the CTS scores as a way of comparing your
object-based results to more traditional metrics.
Hope that helps.
John
On 05/18/2011 10:03 PM, RAL HelpDesk {for jdduda at iastate.edu} wrote:
>
> Wed May 18 22:03:33 2011: Request 46931 was acted upon.
> Transaction: Ticket created by jdduda at iastate.edu
> Queue: met_help
> Subject: MODE: Gilbert Skill Score question
> Owner: Nobody
> Requestors: jdduda at iastate.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=46931 >
>
>
> Hello,
> I just want to clarify that, in the output file from mode that
contains the
> CTSs, the first line of data (after the headings) contains the
scores for
> the RAW, unprocessed data. Thus, the GSS I see (equivalent to the
equitable
> threat score) is the true ETS based on standard contingency table
statistics
> from the unprocessed precip field, and wasn't computed from the
precip field
> after the threshold was applied, right? That's what the second line
of data
> that is called FILTER, right?
>
> Jeff Duda
>
------------------------------------------------
Subject: MODE: Gilbert Skill Score question
From: jdduda at iastate.edu
Time: Thu May 19 09:41:24 2011
John,
Let me make sure I understand this. The "fcst_conv_thresh" and
"obs_conv_thresh" are applied to the raw forecast and observation
fields
without any convolution being applied to them first in the RAW line,
right?
So the GSS and FBIAS scores are being computed in the traditional way
in
this line, right?
Jeff
On Thu, May 19, 2011 at 10:07 AM, RAL HelpDesk {for John Halley
Gotway} <
met_help at ucar.edu> wrote:
> Jeff,
>
> The MODE CTS text file will always contain 4 lines in it:
> (1) A header row
> (2) A line for the "RAW" scores
> (3) A line for the "FILTER" scores
> (4) And a line for the "OBJECT" scores
>
> Unless you've set the "fcst_raw_thresh" and "obs_raw_thresh"
settings in
> the MODE config file, the scores in the RAW and FILTER lines will be
> identical. So the RAW scores are computed by looking at
> the raw forecast and observation fields, applying the
"fcst_conv_thresh"
> and "obs_conv_thresh" thresholds to create 0/1 fields, computing a
2x2
> contingency table, and then deriving statistics from the
> contingency table. The FILTER scores are computed in an identical
way,
> except that before thresholding the raw values, MODE first applies
the
> "fcst_raw_thresh" and "obs_raw_thresh" to zero out values
> that don't meet those threshold criteria. The OBJECT scores are
also
> computed on a grid-point by grid-point basis, where each grid point
is
> considered "on" if it is inside an object and "off" if it
> is not. Again those counts are put into a 2x2 contingency table and
> statistics are derived.
>
> The application of the "fcst_raw_thresh" and "obs_raw_thresh" may
seem odd
> but there's a good reason for them. At one point, we were applying
MODE to
> forecasts of reflectivity. However the algorithm
> only forecasted areas of reflectivity >= 35 dBZ, whereas the
observations
> contained reflectivity values well below 35 dBZ. Basically, the
forecasts
> had areas of reflectivity >= 35 dBZ surrounded by a
> bunch of zeros, whereas the observations had areas of reflectivity
>= 35
> dBZ surrounded by values of 30, 25, 20, ..., and so on. The problem
arose
> in the MODE convolution step. Convolving the
> forecast values with a bunch of zeros produced much smaller objects
that
> convolving the observations with non-zero values. So we used these
settings
> to filter out observations < 35 dBZ to make the
> observation field look more similar to the forecast field. That way
the
> MODE object-definition process would work the same on both fields.
Sorry
> for all that detail, but that's the reason for it.
> Most users will not need to use these settings unless they're in a
similar
> situation to what I've described.
>
> And in general, since object-based verification approaches are still
pretty
> new, we provided the CTS scores as a way of comparing your object-
based
> results to more traditional metrics.
>
> Hope that helps.
>
> John
>
> On 05/18/2011 10:03 PM, RAL HelpDesk {for jdduda at iastate.edu} wrote:
> >
> > Wed May 18 22:03:33 2011: Request 46931 was acted upon.
> > Transaction: Ticket created by jdduda at iastate.edu
> > Queue: met_help
> > Subject: MODE: Gilbert Skill Score question
> > Owner: Nobody
> > Requestors: jdduda at iastate.edu
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=46931 >
> >
> >
> > Hello,
> > I just want to clarify that, in the output file from mode that
contains
> the
> > CTSs, the first line of data (after the headings) contains the
scores for
> > the RAW, unprocessed data. Thus, the GSS I see (equivalent to the
> equitable
> > threat score) is the true ETS based on standard contingency table
> statistics
> > from the unprocessed precip field, and wasn't computed from the
precip
> field
> > after the threshold was applied, right? That's what the second
line of
> data
> > that is called FILTER, right?
> >
> > Jeff Duda
> >
>
>
--
Jeff Duda
Iowa State University
Meteorology Graduate Student
3134 Agronomy Hall
www.meteor.iastate.edu/~jdduda
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #46931] MODE: Gilbert Skill Score question
From: John Halley Gotway
Time: Thu May 19 09:48:58 2011
Jeff,
Yes, that's correct.
John
On 05/19/2011 09:41 AM, RAL HelpDesk {for jdduda at iastate.edu} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=46931 >
>
> John,
> Let me make sure I understand this. The "fcst_conv_thresh" and
> "obs_conv_thresh" are applied to the raw forecast and observation
fields
> without any convolution being applied to them first in the RAW line,
right?
> So the GSS and FBIAS scores are being computed in the traditional
way in
> this line, right?
>
> Jeff
>
> On Thu, May 19, 2011 at 10:07 AM, RAL HelpDesk {for John Halley
Gotway} <
> met_help at ucar.edu> wrote:
>
>> Jeff,
>>
>> The MODE CTS text file will always contain 4 lines in it:
>> (1) A header row
>> (2) A line for the "RAW" scores
>> (3) A line for the "FILTER" scores
>> (4) And a line for the "OBJECT" scores
>>
>> Unless you've set the "fcst_raw_thresh" and "obs_raw_thresh"
settings in
>> the MODE config file, the scores in the RAW and FILTER lines will
be
>> identical. So the RAW scores are computed by looking at
>> the raw forecast and observation fields, applying the
"fcst_conv_thresh"
>> and "obs_conv_thresh" thresholds to create 0/1 fields, computing a
2x2
>> contingency table, and then deriving statistics from the
>> contingency table. The FILTER scores are computed in an identical
way,
>> except that before thresholding the raw values, MODE first applies
the
>> "fcst_raw_thresh" and "obs_raw_thresh" to zero out values
>> that don't meet those threshold criteria. The OBJECT scores are
also
>> computed on a grid-point by grid-point basis, where each grid point
is
>> considered "on" if it is inside an object and "off" if it
>> is not. Again those counts are put into a 2x2 contingency table
and
>> statistics are derived.
>>
>> The application of the "fcst_raw_thresh" and "obs_raw_thresh" may
seem odd
>> but there's a good reason for them. At one point, we were applying
MODE to
>> forecasts of reflectivity. However the algorithm
>> only forecasted areas of reflectivity >= 35 dBZ, whereas the
observations
>> contained reflectivity values well below 35 dBZ. Basically, the
forecasts
>> had areas of reflectivity >= 35 dBZ surrounded by a
>> bunch of zeros, whereas the observations had areas of reflectivity
>= 35
>> dBZ surrounded by values of 30, 25, 20, ..., and so on. The
problem arose
>> in the MODE convolution step. Convolving the
>> forecast values with a bunch of zeros produced much smaller objects
that
>> convolving the observations with non-zero values. So we used these
settings
>> to filter out observations < 35 dBZ to make the
>> observation field look more similar to the forecast field. That
way the
>> MODE object-definition process would work the same on both fields.
Sorry
>> for all that detail, but that's the reason for it.
>> Most users will not need to use these settings unless they're in a
similar
>> situation to what I've described.
>>
>> And in general, since object-based verification approaches are
still pretty
>> new, we provided the CTS scores as a way of comparing your object-
based
>> results to more traditional metrics.
>>
>> Hope that helps.
>>
>> John
>>
>> On 05/18/2011 10:03 PM, RAL HelpDesk {for jdduda at iastate.edu}
wrote:
>>>
>>> Wed May 18 22:03:33 2011: Request 46931 was acted upon.
>>> Transaction: Ticket created by jdduda at iastate.edu
>>> Queue: met_help
>>> Subject: MODE: Gilbert Skill Score question
>>> Owner: Nobody
>>> Requestors: jdduda at iastate.edu
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=46931 >
>>>
>>>
>>> Hello,
>>> I just want to clarify that, in the output file from mode that
contains
>> the
>>> CTSs, the first line of data (after the headings) contains the
scores for
>>> the RAW, unprocessed data. Thus, the GSS I see (equivalent to the
>> equitable
>>> threat score) is the true ETS based on standard contingency table
>> statistics
>>> from the unprocessed precip field, and wasn't computed from the
precip
>> field
>>> after the threshold was applied, right? That's what the second
line of
>> data
>>> that is called FILTER, right?
>>>
>>> Jeff Duda
>>>
>>
>>
>
>
------------------------------------------------
More information about the Met_help
mailing list