[Met_help] [rt.rap.ucar.edu #98347] History for Grid Stat Neighborhood Methods on Probability Data
John Halley Gotway via RT
met_help at ucar.edu
Wed Feb 10 09:07:23 MST 2021
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
John, I am verifying ensemble probabilities of wind speed > 15kts and using a gridded analysis as ground truth. I have defined a series of neighborhoods. I found that to get this to work I had to set the prob_as_scalar flag. When I do this am I comparing a forecast field of 0-1 to a observation field of either 0 (winds < 15) or 1 (winds > 15). Or, is does MET not allow verification of neighborhoods of probability data?
Thanks
Bob
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Grid Stat Neighborhood Methods on Probability Data
From: David Fillmore
Time: Fri Jan 22 10:51:36 2021
Hi Bob - I am forwarding your Grid Stat question to John to make sure
he
sees it,
David
On Fri, Jan 22, 2021 at 9:58 AM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> Fri Jan 22 09:58:07 2021: Request 98347 was acted upon.
> Transaction: Ticket created by robert.craig.2 at us.af.mil
> Queue: met_help
> Subject: Grid Stat Neighborhood Methods on Probability Data
> Owner: Nobody
> Requestors: robert.craig.2 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98347 >
>
>
> John, I am verifying ensemble probabilities of wind speed > 15kts
and
> using a gridded analysis as ground truth. I have defined a series
of
> neighborhoods. I found that to get this to work I had to set the
> prob_as_scalar flag. When I do this am I comparing a forecast field
of 0-1
> to a observation field of either 0 (winds < 15) or 1 (winds > 15).
Or, is
> does MET not allow verification of neighborhoods of probability
data?
>
> Thanks
> Bob
>
>
------------------------------------------------
Subject: Grid Stat Neighborhood Methods on Probability Data
From: John Halley Gotway
Time: Fri Jan 22 13:15:37 2021
Hi Bob,
You are correct that the neighborhood methods in Grid-Stat are only
applied
to scalar fields, and not probabilities.
And you're right in using "prob_as_scalar = TRUE" to process
probability data as scalars.
As a brief overview, the neighborhood probability logic in Grid-Stat
goes
like this:
- Process scalar fields for which categorical threshold(s) are
defined.
- Apply the categorical threshold to convert the raw data into a
binary
field of 1's and 0's.
- Apply the neighborhood size and shape to convert that binary field
into a
fractional coverage field of numbers between 0 and 1.
- At each grid point, the fractional coverage value represents the
event
frequency within the neighborhood.
- After doing this for both the forecast and observation data, compare
those fractional coverage fields to compute neighborhood stats, like
Fractions Skill Score (FSS).
As you can see, the neighborhood logic is very tied to a single event
definition threshold.
I'm wondering exactly how you were hoping Grid-Stat would apply
neighborhoods to the verification of probabilities? If you actually
just
want to SMOOTH the probability fields prior to verifying them, Grid-
Stat
can already do that using the "interp" dictionary. But if it's
something
else, I'd need more details to understand exactly what logic you're
looking
for.
Thanks,
John
On Fri, Jan 22, 2021 at 11:46 AM David Fillmore via RT
<met_help at ucar.edu>
wrote:
>
> Fri Jan 22 11:40:38 2021: Request 98347 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by fillmore
> Queue: met_help
> Subject: Grid Stat Neighborhood Methods on Probability Data
> Owner: johnhg
> Requestors: robert.craig.2 at us.af.mil
> Status: open
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98347 >
>
>
> This transaction appears to have no content
>
------------------------------------------------
Subject: Grid Stat Neighborhood Methods on Probability Data
From: robert.craig.2 at us.af.mil
Time: Fri Jan 22 13:36:40 2021
John, what I was thinking of doing was to take the probability fcst
field of an event (wind >15) and convert the field to a 1 if the
probability was greater than an threshold or 0 if it wasn't and
compare to the ob grid with a threshold of wind > 15 then use the
neighborhood logic. But from your description of the logic, that
probably wouldn't work as expected.
What I need to do is calculate FSS for 4km probability fields and
compare to FSS for 55km probability field.
Another idea was to use MODE where I have defined an object where the
prob of an event greater than a threshold and then compare to an
gridded analysis where the event has been thresholded.
Bob
________________________________
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Friday, January 22, 2021 2:15 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXD
<robert.craig.2 at us.af.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #98347] Grid Stat
Neighborhood Methods on Probability Data
Hi Bob,
You are correct that the neighborhood methods in Grid-Stat are only
applied
to scalar fields, and not probabilities.
And you're right in using "prob_as_scalar = TRUE" to process
probability data as scalars.
As a brief overview, the neighborhood probability logic in Grid-Stat
goes
like this:
- Process scalar fields for which categorical threshold(s) are
defined.
- Apply the categorical threshold to convert the raw data into a
binary
field of 1's and 0's.
- Apply the neighborhood size and shape to convert that binary field
into a
fractional coverage field of numbers between 0 and 1.
- At each grid point, the fractional coverage value represents the
event
frequency within the neighborhood.
- After doing this for both the forecast and observation data, compare
those fractional coverage fields to compute neighborhood stats, like
Fractions Skill Score (FSS).
As you can see, the neighborhood logic is very tied to a single event
definition threshold.
I'm wondering exactly how you were hoping Grid-Stat would apply
neighborhoods to the verification of probabilities? If you actually
just
want to SMOOTH the probability fields prior to verifying them, Grid-
Stat
can already do that using the "interp" dictionary. But if it's
something
else, I'd need more details to understand exactly what logic you're
looking
for.
Thanks,
John
On Fri, Jan 22, 2021 at 11:46 AM David Fillmore via RT
<met_help at ucar.edu>
wrote:
>
> Fri Jan 22 11:40:38 2021: Request 98347 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by fillmore
> Queue: met_help
> Subject: Grid Stat Neighborhood Methods on Probability Data
> Owner: johnhg
> Requestors: robert.craig.2 at us.af.mil
> Status: open
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98347 >
>
>
> This transaction appears to have no content
>
------------------------------------------------
Subject: Grid Stat Neighborhood Methods on Probability Data
From: robert.craig.2 at us.af.mil
Time: Fri Jan 22 14:10:05 2021
John, I forgot to ask - If I use the interp capability to smooth the
grid field, can I still get a FSS calculated?
Bob
________________________________
From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXD
<robert.craig.2 at us.af.mil>
Sent: Friday, January 22, 2021 2:34 PM
To: met_help at ucar.edu <met_help at ucar.edu>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #98347] Grid Stat
Neighborhood Methods on Probability Data
John, what I was thinking of doing was to take the probability fcst
field of an event (wind >15) and convert the field to a 1 if the
probability was greater than an threshold or 0 if it wasn't and
compare to the ob grid with a threshold of wind > 15 then use the
neighborhood logic. But from your description of the logic, that
probably wouldn't work as expected.
What I need to do is calculate FSS for 4km probability fields and
compare to FSS for 55km probability field.
Another idea was to use MODE where I have defined an object where the
prob of an event greater than a threshold and then compare to an
gridded analysis where the event has been thresholded.
Bob
________________________________
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Friday, January 22, 2021 2:15 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXD
<robert.craig.2 at us.af.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #98347] Grid Stat
Neighborhood Methods on Probability Data
Hi Bob,
You are correct that the neighborhood methods in Grid-Stat are only
applied
to scalar fields, and not probabilities.
And you're right in using "prob_as_scalar = TRUE" to process
probability data as scalars.
As a brief overview, the neighborhood probability logic in Grid-Stat
goes
like this:
- Process scalar fields for which categorical threshold(s) are
defined.
- Apply the categorical threshold to convert the raw data into a
binary
field of 1's and 0's.
- Apply the neighborhood size and shape to convert that binary field
into a
fractional coverage field of numbers between 0 and 1.
- At each grid point, the fractional coverage value represents the
event
frequency within the neighborhood.
- After doing this for both the forecast and observation data, compare
those fractional coverage fields to compute neighborhood stats, like
Fractions Skill Score (FSS).
As you can see, the neighborhood logic is very tied to a single event
definition threshold.
I'm wondering exactly how you were hoping Grid-Stat would apply
neighborhoods to the verification of probabilities? If you actually
just
want to SMOOTH the probability fields prior to verifying them, Grid-
Stat
can already do that using the "interp" dictionary. But if it's
something
else, I'd need more details to understand exactly what logic you're
looking
for.
Thanks,
John
On Fri, Jan 22, 2021 at 11:46 AM David Fillmore via RT
<met_help at ucar.edu>
wrote:
>
> Fri Jan 22 11:40:38 2021: Request 98347 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by fillmore
> Queue: met_help
> Subject: Grid Stat Neighborhood Methods on Probability Data
> Owner: johnhg
> Requestors: robert.craig.2 at us.af.mil
> Status: open
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98347 >
>
>
> This transaction appears to have no content
>
------------------------------------------------
Subject: Grid Stat Neighborhood Methods on Probability Data
From: John Halley Gotway
Time: Fri Jan 22 15:07:22 2021
Bob,
You should definitely be able to do what you described by setting
prob_as_scalar = TRUE.
In the fcst dictionary, set...
prob_as_scalar = TRUE;
cat_thresh = [ >=0.5 ]; // or whatever probability threshold(s) you
choose
And in the obs dictionary, set...
cat_thresh = [ >15 ];
Those thresholds will convert the fcst and obs data into 0/1 binary
fields
on which the neighborhood methods will be run.
I would not recommend doing the smoothing step (using the interp
dictionary) and the neighborhood methods at the same time. That's a
lot of
processing of the data and would make it difficult to interpret the
results. For that reason, the neighborhood methods are applied to the
RAW
data and not to the smoothed output from the interp step. So those are
separate processes.
If for some reason, you wanted to play around with this, you could
always
run "regrid_data_plane" to smooth your data prior to passing it into
Grid-Stat.
John
On Fri, Jan 22, 2021 at 2:10 PM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98347 >
>
> John, I forgot to ask - If I use the interp capability to smooth the
grid
> field, can I still get a FSS calculated?
>
> Bob
>
> ________________________________
> From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXD
<robert.craig.2 at us.af.mil>
> Sent: Friday, January 22, 2021 2:34 PM
> To: met_help at ucar.edu <met_help at ucar.edu>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #98347] Grid Stat
> Neighborhood Methods on Probability Data
>
> John, what I was thinking of doing was to take the probability fcst
> field of an event (wind >15) and convert the field to a 1 if the
> probability was greater than an threshold or 0 if it wasn't and
compare to
> the ob grid with a threshold of wind > 15 then use the neighborhood
> logic. But from your description of the logic, that probably
wouldn't work
> as expected.
>
> What I need to do is calculate FSS for 4km probability fields and
compare
> to FSS for 55km probability field.
>
> Another idea was to use MODE where I have defined an object where
the prob
> of an event greater than a threshold and then compare to an gridded
> analysis where the event has been thresholded.
>
> Bob
>
> ________________________________
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Friday, January 22, 2021 2:15 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXD
<robert.craig.2 at us.af.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #98347] Grid Stat
> Neighborhood Methods on Probability Data
>
> Hi Bob,
>
> You are correct that the neighborhood methods in Grid-Stat are only
applied
> to scalar fields, and not probabilities.
> And you're right in using "prob_as_scalar = TRUE" to process
> probability data as scalars.
>
> As a brief overview, the neighborhood probability logic in Grid-Stat
goes
> like this:
> - Process scalar fields for which categorical threshold(s) are
defined.
> - Apply the categorical threshold to convert the raw data into a
binary
> field of 1's and 0's.
> - Apply the neighborhood size and shape to convert that binary field
into a
> fractional coverage field of numbers between 0 and 1.
> - At each grid point, the fractional coverage value represents the
event
> frequency within the neighborhood.
> - After doing this for both the forecast and observation data,
compare
> those fractional coverage fields to compute neighborhood stats, like
> Fractions Skill Score (FSS).
>
> As you can see, the neighborhood logic is very tied to a single
event
> definition threshold.
>
> I'm wondering exactly how you were hoping Grid-Stat would apply
> neighborhoods to the verification of probabilities? If you actually
just
> want to SMOOTH the probability fields prior to verifying them, Grid-
Stat
> can already do that using the "interp" dictionary. But if it's
something
> else, I'd need more details to understand exactly what logic you're
looking
> for.
>
> Thanks,
> John
>
> On Fri, Jan 22, 2021 at 11:46 AM David Fillmore via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Fri Jan 22 11:40:38 2021: Request 98347 was acted upon.
> > Transaction: Given to johnhg (John Halley Gotway) by fillmore
> > Queue: met_help
> > Subject: Grid Stat Neighborhood Methods on Probability Data
> > Owner: johnhg
> > Requestors: robert.craig.2 at us.af.mil
> > Status: open
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98347 >
> >
> >
> > This transaction appears to have no content
> >
>
>
>
------------------------------------------------
Subject: Grid Stat Neighborhood Methods on Probability Data
From: robert.craig.2 at us.af.mil
Time: Fri Jan 22 15:41:14 2021
Okay thanks, I will give that a shot.
Bob
________________________________
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: Friday, January 22, 2021 4:07 PM
To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXD
<robert.craig.2 at us.af.mil>
Subject: Re: Fw: [Non-DoD Source] Re: [rt.rap.ucar.edu #98347] Grid
Stat Neighborhood Methods on Probability Data
Bob,
You should definitely be able to do what you described by setting
prob_as_scalar = TRUE.
In the fcst dictionary, set...
prob_as_scalar = TRUE;
cat_thresh = [ >=0.5 ]; // or whatever probability threshold(s) you
choose
And in the obs dictionary, set...
cat_thresh = [ >15 ];
Those thresholds will convert the fcst and obs data into 0/1 binary
fields
on which the neighborhood methods will be run.
I would not recommend doing the smoothing step (using the interp
dictionary) and the neighborhood methods at the same time. That's a
lot of
processing of the data and would make it difficult to interpret the
results. For that reason, the neighborhood methods are applied to the
RAW
data and not to the smoothed output from the interp step. So those are
separate processes.
If for some reason, you wanted to play around with this, you could
always
run "regrid_data_plane" to smooth your data prior to passing it into
Grid-Stat.
John
On Fri, Jan 22, 2021 at 2:10 PM robert.craig.2 at us.af.mil via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98347 >
>
> John, I forgot to ask - If I use the interp capability to smooth the
grid
> field, can I still get a FSS calculated?
>
> Bob
>
> ________________________________
> From: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXD
<robert.craig.2 at us.af.mil>
> Sent: Friday, January 22, 2021 2:34 PM
> To: met_help at ucar.edu <met_help at ucar.edu>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #98347] Grid Stat
> Neighborhood Methods on Probability Data
>
> John, what I was thinking of doing was to take the probability fcst
> field of an event (wind >15) and convert the field to a 1 if the
> probability was greater than an threshold or 0 if it wasn't and
compare to
> the ob grid with a threshold of wind > 15 then use the neighborhood
> logic. But from your description of the logic, that probably
wouldn't work
> as expected.
>
> What I need to do is calculate FSS for 4km probability fields and
compare
> to FSS for 55km probability field.
>
> Another idea was to use MODE where I have defined an object where
the prob
> of an event greater than a threshold and then compare to an gridded
> analysis where the event has been thresholded.
>
> Bob
>
> ________________________________
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: Friday, January 22, 2021 2:15 PM
> To: CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXD
<robert.craig.2 at us.af.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #98347] Grid Stat
> Neighborhood Methods on Probability Data
>
> Hi Bob,
>
> You are correct that the neighborhood methods in Grid-Stat are only
applied
> to scalar fields, and not probabilities.
> And you're right in using "prob_as_scalar = TRUE" to process
> probability data as scalars.
>
> As a brief overview, the neighborhood probability logic in Grid-Stat
goes
> like this:
> - Process scalar fields for which categorical threshold(s) are
defined.
> - Apply the categorical threshold to convert the raw data into a
binary
> field of 1's and 0's.
> - Apply the neighborhood size and shape to convert that binary field
into a
> fractional coverage field of numbers between 0 and 1.
> - At each grid point, the fractional coverage value represents the
event
> frequency within the neighborhood.
> - After doing this for both the forecast and observation data,
compare
> those fractional coverage fields to compute neighborhood stats, like
> Fractions Skill Score (FSS).
>
> As you can see, the neighborhood logic is very tied to a single
event
> definition threshold.
>
> I'm wondering exactly how you were hoping Grid-Stat would apply
> neighborhoods to the verification of probabilities? If you actually
just
> want to SMOOTH the probability fields prior to verifying them, Grid-
Stat
> can already do that using the "interp" dictionary. But if it's
something
> else, I'd need more details to understand exactly what logic you're
looking
> for.
>
> Thanks,
> John
>
> On Fri, Jan 22, 2021 at 11:46 AM David Fillmore via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Fri Jan 22 11:40:38 2021: Request 98347 was acted upon.
> > Transaction: Given to johnhg (John Halley Gotway) by fillmore
> > Queue: met_help
> > Subject: Grid Stat Neighborhood Methods on Probability Data
> > Owner: johnhg
> > Requestors: robert.craig.2 at us.af.mil
> > Status: open
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98347 >
> >
> >
> > This transaction appears to have no content
> >
>
>
>
------------------------------------------------
More information about the Met_help
mailing list