[Met_help] [rt.rap.ucar.edu #83265] History for Issue with generating FSS statistics for probabilities
John Halley Gotway via RT
met_help at ucar.edu
Tue Jul 9 12:04:01 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi,
I am attempting to generate FSS statistics on probabilistic output (chance
of 6-h accumulated precip exceeding 12.7 mm or 0.5") using Grid Stat and
the NBRCNT line type. However, I don't think what I am doing is correct
because my FSS values are extremely low.
In my Grid Stat Config file (attached), I set prob_as_scalar = TRUE so that
neighborhood verification statistics can be computed. For verifying
observations, I use Stage-IV 06-h accumulated precip. The probabilities
are point probabilities (not neighborhood probs) so I do not need to modify
the Stage IV values. However, I do re-grid the Stage IV file to grid 227
prior to running MET so that the fcst and obs files are on the same grid.
I've also attached a nbrcnt.txt file which may help to determine what is
wrong. I noticed the F_RATE is equal to 1 while the O_RATE is much lower,
so I think I am not handling the probabilities correctly in my config file.
Thanks,
Ben
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Issue with generating FSS statistics for probabilities
From: John Halley Gotway
Time: Mon Dec 11 11:19:59 2017
Ben,
I see you have some questions about Fractions Skill Score in Grid-
Stat.
You are passing a forecast field of probabilities into MET but using
prob_as_scalar to tell MET to not process them as probabilities.
In general, the logic for computing FSS works as follows...
(1) Read scalar fields of forecast and observation data.
(2) Apply thresholds to both fields to define 0/1 bitmap.
(3) Use the neighborhood shape and width to convert that bitmap field
into
a "fractional coverage" field which contains at each grid point the
fraction of the points within the neighborhood that have a value of 1
(i.e.
they meet the threshold criteria).
(4) Use those forecast and observation fractional coverage values to
compute fractions skill score.
Looking through that logic, the piece that is missing in your config
file
is the specification of a categorical threshold value (cat_thresh).
Looking in your output, I see that FCST_THRESH = NA. A threshold of
type
NA always evaluates to TRUE, which is why the forecast rate (F_RATE)
column
is 1.
So I'd suggest adding the following to your fcst.field entry:
cat_thresh = [ >=0.5 ];
However, I suspect this may not be the verification logic you'd
actually
like to apply. The forecast probability field already contains values
between 0 and 1. Are you actually trying to treat those forecast
probability values as fractional coverage values... and then compare
them
to the StageIV observation values? If so, there is no current logic
defined in Grid-Stat for that processing flow.
FYI, while you're testing, I'd suggest turning on the NetCDF matched
pairs
output file, setting "nbrhd" to true. That will write out the
Fractional
Coverage Fields that are computed during the application of the
neighborhood methods.
nc_pairs_flag = {
latlon = TRUE;
raw = TRUE;
diff = TRUE;
climo = TRUE;
weight = FALSE;
nbrhd = TRUE;
apply_mask = TRUE;
}
Thanks,
John
On Mon, Dec 11, 2017 at 10:18 AM, Benjamin Blake - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> Mon Dec 11 10:18:39 2017: Request 83265 was acted upon.
> Transaction: Ticket created by benjamin.blake at noaa.gov
> Queue: met_help
> Subject: Issue with generating FSS statistics for probabilities
> Owner: Nobody
> Requestors: benjamin.blake at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83265 >
>
>
> Hi,
>
> I am attempting to generate FSS statistics on probabilistic output
(chance
> of 6-h accumulated precip exceeding 12.7 mm or 0.5") using Grid Stat
and
> the NBRCNT line type. However, I don't think what I am doing is
correct
> because my FSS values are extremely low.
>
> In my Grid Stat Config file (attached), I set prob_as_scalar = TRUE
so that
> neighborhood verification statistics can be computed. For verifying
> observations, I use Stage-IV 06-h accumulated precip. The
probabilities
> are point probabilities (not neighborhood probs) so I do not need to
modify
> the Stage IV values. However, I do re-grid the Stage IV file to
grid 227
> prior to running MET so that the fcst and obs files are on the same
grid.
>
> I've also attached a nbrcnt.txt file which may help to determine
what is
> wrong. I noticed the F_RATE is equal to 1 while the O_RATE is much
lower,
> so I think I am not handling the probabilities correctly in my
config file.
>
> Thanks,
> Ben
>
>
------------------------------------------------
Subject: Issue with generating FSS statistics for probabilities
From: Benjamin Blake - NOAA Affiliate
Time: Mon Dec 11 12:55:56 2017
Hi John,
Thank you for the clarification. I now understand what I was doing
incorrectly.
I need to think about this more, but using a cat_thresh of 0.5 may be
sufficient for what I am trying to accomplish. The probabilities
represent
the chance of exceeding a threshold at any given point. I re-ran a
FSS
case using cat_thresh = [ >=0.5 ] and got much more realistic values
for
FSS (one forecast hour):
0.56871 (20-km), 0.62674 (40-km), 0.66494 (60-km), 0.68863 (80-km),
0.70386
(100-km)
Thanks,
Ben
On Mon, Dec 11, 2017 at 1:20 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Ben,
>
> I see you have some questions about Fractions Skill Score in Grid-
Stat.
> You are passing a forecast field of probabilities into MET but using
> prob_as_scalar to tell MET to not process them as probabilities.
>
> In general, the logic for computing FSS works as follows...
> (1) Read scalar fields of forecast and observation data.
> (2) Apply thresholds to both fields to define 0/1 bitmap.
> (3) Use the neighborhood shape and width to convert that bitmap
field into
> a "fractional coverage" field which contains at each grid point the
> fraction of the points within the neighborhood that have a value of
1 (i.e.
> they meet the threshold criteria).
> (4) Use those forecast and observation fractional coverage values to
> compute fractions skill score.
>
>
> Looking through that logic, the piece that is missing in your config
file
> is the specification of a categorical threshold value (cat_thresh).
>
> Looking in your output, I see that FCST_THRESH = NA. A threshold of
type
> NA always evaluates to TRUE, which is why the forecast rate (F_RATE)
column
> is 1.
>
> So I'd suggest adding the following to your fcst.field entry:
> cat_thresh = [ >=0.5 ];
>
> However, I suspect this may not be the verification logic you'd
actually
> like to apply. The forecast probability field already contains
values
> between 0 and 1. Are you actually trying to treat those forecast
> probability values as fractional coverage values... and then compare
them
> to the StageIV observation values? If so, there is no current logic
> defined in Grid-Stat for that processing flow.
>
> FYI, while you're testing, I'd suggest turning on the NetCDF matched
pairs
> output file, setting "nbrhd" to true. That will write out the
Fractional
> Coverage Fields that are computed during the application of the
> neighborhood methods.
>
> nc_pairs_flag = {
> latlon = TRUE;
> raw = TRUE;
> diff = TRUE;
> climo = TRUE;
> weight = FALSE;
> nbrhd = TRUE;
> apply_mask = TRUE;
> }
>
> Thanks,
> John
>
>
>
>
>
> On Mon, Dec 11, 2017 at 10:18 AM, Benjamin Blake - NOAA Affiliate
via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Dec 11 10:18:39 2017: Request 83265 was acted upon.
> > Transaction: Ticket created by benjamin.blake at noaa.gov
> > Queue: met_help
> > Subject: Issue with generating FSS statistics for
probabilities
> > Owner: Nobody
> > Requestors: benjamin.blake at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83265 >
> >
> >
> > Hi,
> >
> > I am attempting to generate FSS statistics on probabilistic output
> (chance
> > of 6-h accumulated precip exceeding 12.7 mm or 0.5") using Grid
Stat and
> > the NBRCNT line type. However, I don't think what I am doing is
correct
> > because my FSS values are extremely low.
> >
> > In my Grid Stat Config file (attached), I set prob_as_scalar =
TRUE so
> that
> > neighborhood verification statistics can be computed. For
verifying
> > observations, I use Stage-IV 06-h accumulated precip. The
probabilities
> > are point probabilities (not neighborhood probs) so I do not need
to
> modify
> > the Stage IV values. However, I do re-grid the Stage IV file to
grid 227
> > prior to running MET so that the fcst and obs files are on the
same grid.
> >
> > I've also attached a nbrcnt.txt file which may help to determine
what is
> > wrong. I noticed the F_RATE is equal to 1 while the O_RATE is
much
> lower,
> > so I think I am not handling the probabilities correctly in my
config
> file.
> >
> > Thanks,
> > Ben
> >
> >
>
>
------------------------------------------------
Subject: Issue with generating FSS statistics for probabilities
From: John Halley Gotway
Time: Mon Dec 11 14:34:31 2017
Ben,
OK, sounds good. I'll resolve this ticket. Just let us know what
questions or issues arise.
Thanks,
John
On Mon, Dec 11, 2017 at 12:55 PM, Benjamin Blake - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83265 >
>
> Hi John,
>
> Thank you for the clarification. I now understand what I was doing
> incorrectly.
>
> I need to think about this more, but using a cat_thresh of 0.5 may
be
> sufficient for what I am trying to accomplish. The probabilities
represent
> the chance of exceeding a threshold at any given point. I re-ran a
FSS
> case using cat_thresh = [ >=0.5 ] and got much more realistic values
for
> FSS (one forecast hour):
> 0.56871 (20-km), 0.62674 (40-km), 0.66494 (60-km), 0.68863 (80-km),
0.70386
> (100-km)
>
> Thanks,
> Ben
>
> On Mon, Dec 11, 2017 at 1:20 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Ben,
> >
> > I see you have some questions about Fractions Skill Score in Grid-
Stat.
> > You are passing a forecast field of probabilities into MET but
using
> > prob_as_scalar to tell MET to not process them as probabilities.
> >
> > In general, the logic for computing FSS works as follows...
> > (1) Read scalar fields of forecast and observation data.
> > (2) Apply thresholds to both fields to define 0/1 bitmap.
> > (3) Use the neighborhood shape and width to convert that bitmap
field
> into
> > a "fractional coverage" field which contains at each grid point
the
> > fraction of the points within the neighborhood that have a value
of 1
> (i.e.
> > they meet the threshold criteria).
> > (4) Use those forecast and observation fractional coverage values
to
> > compute fractions skill score.
> >
> >
> > Looking through that logic, the piece that is missing in your
config file
> > is the specification of a categorical threshold value
(cat_thresh).
> >
> > Looking in your output, I see that FCST_THRESH = NA. A threshold
of type
> > NA always evaluates to TRUE, which is why the forecast rate
(F_RATE)
> column
> > is 1.
> >
> > So I'd suggest adding the following to your fcst.field entry:
> > cat_thresh = [ >=0.5 ];
> >
> > However, I suspect this may not be the verification logic you'd
actually
> > like to apply. The forecast probability field already contains
values
> > between 0 and 1. Are you actually trying to treat those forecast
> > probability values as fractional coverage values... and then
compare them
> > to the StageIV observation values? If so, there is no current
logic
> > defined in Grid-Stat for that processing flow.
> >
> > FYI, while you're testing, I'd suggest turning on the NetCDF
matched
> pairs
> > output file, setting "nbrhd" to true. That will write out the
Fractional
> > Coverage Fields that are computed during the application of the
> > neighborhood methods.
> >
> > nc_pairs_flag = {
> > latlon = TRUE;
> > raw = TRUE;
> > diff = TRUE;
> > climo = TRUE;
> > weight = FALSE;
> > nbrhd = TRUE;
> > apply_mask = TRUE;
> > }
> >
> > Thanks,
> > John
> >
> >
> >
> >
> >
> > On Mon, Dec 11, 2017 at 10:18 AM, Benjamin Blake - NOAA Affiliate
via RT
> <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Mon Dec 11 10:18:39 2017: Request 83265 was acted upon.
> > > Transaction: Ticket created by benjamin.blake at noaa.gov
> > > Queue: met_help
> > > Subject: Issue with generating FSS statistics for
probabilities
> > > Owner: Nobody
> > > Requestors: benjamin.blake at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83265
> >
> > >
> > >
> > > Hi,
> > >
> > > I am attempting to generate FSS statistics on probabilistic
output
> > (chance
> > > of 6-h accumulated precip exceeding 12.7 mm or 0.5") using Grid
Stat
> and
> > > the NBRCNT line type. However, I don't think what I am doing is
> correct
> > > because my FSS values are extremely low.
> > >
> > > In my Grid Stat Config file (attached), I set prob_as_scalar =
TRUE so
> > that
> > > neighborhood verification statistics can be computed. For
verifying
> > > observations, I use Stage-IV 06-h accumulated precip. The
> probabilities
> > > are point probabilities (not neighborhood probs) so I do not
need to
> > modify
> > > the Stage IV values. However, I do re-grid the Stage IV file to
grid
> 227
> > > prior to running MET so that the fcst and obs files are on the
same
> grid.
> > >
> > > I've also attached a nbrcnt.txt file which may help to determine
what
> is
> > > wrong. I noticed the F_RATE is equal to 1 while the O_RATE is
much
> > lower,
> > > so I think I am not handling the probabilities correctly in my
config
> > file.
> > >
> > > Thanks,
> > > Ben
> > >
> > >
> >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list