[Met_help] [rt.rap.ucar.edu #84104] History for Computation of AFSS

[John Halley Gotway via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hi John,

I have been computing Fractions Skill Score (FSS) on my own which I have done in the past, just not in Python.  I'm relatively confident that my results are correct for the sample time period I have been using for testing purposes.
However, I have never computed the Asymptotic Fractions Skill Score (AFSS) before.   I have double checked the logic and have been using Eq. 8 in Roberts and Lean (2008) as my reference point.  Still, I get the impression that AFSS should always be higher than the FSS I am computing for a specific neighborhood.   However, in this case, I am getting lower values based on model-forecast and observed frequency.   Is this even possible?

Thanks,
Chris

//SIGNED//

Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693





----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Computation of AFSS
From: Tressa Fowler
Time: Fri Feb 16 10:18:13 2018

Hi Chris,

Unless you have some type of edge effect, AFSS should be higher than
FSS. This assumes that you compute FSS over each smaller neighborhood
in the identical domain that you are calculating AFSS over.

Have you tried using your observation file as both the forecast and
observed data to make sure you get 1 for your AFSS and average FSS
values?

Sometimes in MET, neighborhoods with missing data at domain boundaries
are not included in calculations, which gives interior points greater
"weight" than boundary points and could lead to slight differences.
Could this be happening? It is more likely to matter with large
neighborhoods and small domains.

Tressa

On Fri Feb 16 09:18:15 2018, christopher.melick at us.af.mil wrote:
> Hi John,
>
> I have been computing Fractions Skill Score (FSS) on my own which I
> have done in the past, just not in Python.  I'm relatively confident
> that my results are correct for the sample time period I have been
> using for testing purposes.
> However, I have never computed the Asymptotic Fractions Skill Score
> (AFSS) before.   I have double checked the logic and have been using
> Eq. 8 in Roberts and Lean (2008) as my reference point.  Still, I
get
> the impression that AFSS should always be higher than the FSS I am
> computing for a specific neighborhood.   However, in this case, I am
> getting lower values based on model-forecast and observed frequency.
> Is this even possible?
>
> Thanks,
> Chris
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian
> Meteorologist, Verification Exploitation Team
> 16th Weather Squadron (16 WS/WXEV)
> 557th Weather Wing, Offutt AFB, NE
> DSN 271-3693; Comm (402) 294-3693


------------------------------------------------
Subject: RE: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation of AFSS
From: christopher.melick at us.af.mil
Time: Fri Feb 16 10:37:21 2018

Hi Tressa,

For each neighborhood size, I ensure that AFSS and FSS are computed on
the same domain area by masking any grid points that are
missing/undefined for either the forecast or observed grids.   I
double checked this computation a couple of ways in Python to ensure
the value I obtained matched one another.  Is this what you are
talking about?  Is the forecast/observed frequency just the fractional
coverage of forecast/observed events over the entire domain area
(excluding undefined grid points in the mask)?

I will have to try your suggestion.   I have not done that yet. That
is a great way to ensure my approach and logic are working properly.

Thanks,
Chris

//SIGNED//

Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693



-----Original Message-----
From: Tressa Fowler via RT [mailto:met_help at ucar.edu]
Sent: Friday, February 16, 2018 11:18 AM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation of AFSS

Hi Chris,

Unless you have some type of edge effect, AFSS should be higher than
FSS. This assumes that you compute FSS over each smaller neighborhood
in the identical domain that you are calculating AFSS over.

Have you tried using your observation file as both the forecast and
observed data to make sure you get 1 for your AFSS and average FSS
values?

Sometimes in MET, neighborhoods with missing data at domain boundaries
are not included in calculations, which gives interior points greater
"weight" than boundary points and could lead to slight differences.
Could this be happening? It is more likely to matter with large
neighborhoods and small domains.

Tressa

On Fri Feb 16 09:18:15 2018, christopher.melick at us.af.mil wrote:
> Hi John,
>
> I have been computing Fractions Skill Score (FSS) on my own which I
> have done in the past, just not in Python.  I'm relatively confident
> that my results are correct for the sample time period I have been
> using for testing purposes.
> However, I have never computed the Asymptotic Fractions Skill Score
> (AFSS) before.   I have double checked the logic and have been using
> Eq. 8 in Roberts and Lean (2008) as my reference point.  Still, I
get
> the impression that AFSS should always be higher than the FSS I am
> computing for a specific neighborhood.   However, in this case, I am
> getting lower values based on model-forecast and observed frequency.
> Is this even possible?
>
> Thanks,
> Chris
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693





------------------------------------------------
Subject: Computation of AFSS
From: Tressa Fowler
Time: Fri Feb 16 12:00:37 2018

Hi Chris,

The missing grid locations are not exactly what I meant.

Let's say for example that you have a domain of 5 x 5 points and are
interested in computing FSS over square neighborhoods of width 3 (e.g.
9
grid points). At each of your 25 grid locations, you will find the
proportion of events in both forecast and observed fields at the grid
location and the surrounding 8 grid points that form a box around it.
You
will calculate your score over 25 (overlapping) neighborhoods,
preferably
of size 9 points. However, sixteen of your grid locations are on the
edge
of your domain and will be missing part of the 3x3 neighborhood. A
corner
will have only 4 of the 9 neighborhood points. Are those 4 used in the
FSS
or are any locations with less than 9 neighborhood points ignored?
Regardless, that corner grid point gets counted less often.  It can
only
get used for neighborhoods centered on itself and its 3 neighbors. It
might
be used only once or it could be used 4 times, depending on how you
handle
less than 9 neighbors. The center point of the grid gets used in
calculating the fraction of events in 9 neighborhoods, that centered
on
itself and each of it's 8 immediately surrounding points.

When you do AFSS, there is no problem with edges because you have only
one
neighborhood that is the entire domain of 25 points and you just
figure out
your event proportion for each of the forecast and observed fields.

Does that make sense?

Tressa




On Fri, Feb 16, 2018 at 10:37 AM, christopher.melick at us.af.mil via RT
<
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104 >
>
> Hi Tressa,
>
> For each neighborhood size, I ensure that AFSS and FSS are computed
on the
> same domain area by masking any grid points that are
missing/undefined for
> either the forecast or observed grids.   I double checked this
computation
> a couple of ways in Python to ensure the value I obtained matched
one
> another.  Is this what you are talking about?  Is the
forecast/observed
> frequency just the fractional coverage of forecast/observed events
over the
> entire domain area (excluding undefined grid points in the mask)?
>
> I will have to try your suggestion.   I have not done that yet. That
is a
> great way to ensure my approach and logic are working properly.
>
> Thanks,
> Chris
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian
> Meteorologist, Verification Exploitation Team
> 16th Weather Squadron (16 WS/WXEV)
> 557th Weather Wing, Offutt AFB, NE
> DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: Tressa Fowler via RT [mailto:met_help at ucar.edu]
> Sent: Friday, February 16, 2018 11:18 AM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation of
AFSS
>
> Hi Chris,
>
> Unless you have some type of edge effect, AFSS should be higher than
FSS.
> This assumes that you compute FSS over each smaller neighborhood in
the
> identical domain that you are calculating AFSS over.
>
> Have you tried using your observation file as both the forecast and
> observed data to make sure you get 1 for your AFSS and average FSS
values?
>
> Sometimes in MET, neighborhoods with missing data at domain
boundaries are
> not included in calculations, which gives interior points greater
"weight"
> than boundary points and could lead to slight differences. Could
this be
> happening? It is more likely to matter with large neighborhoods and
small
> domains.
>
> Tressa
>
> On Fri Feb 16 09:18:15 2018, christopher.melick at us.af.mil wrote:
> > Hi John,
> >
> > I have been computing Fractions Skill Score (FSS) on my own which
I
> > have done in the past, just not in Python.  I'm relatively
confident
> > that my results are correct for the sample time period I have been
> > using for testing purposes.
> > However, I have never computed the Asymptotic Fractions Skill
Score
> > (AFSS) before.   I have double checked the logic and have been
using
> > Eq. 8 in Roberts and Lean (2008) as my reference point.  Still, I
get
> > the impression that AFSS should always be higher than the FSS I am
> > computing for a specific neighborhood.   However, in this case, I
am
> > getting lower values based on model-forecast and observed
frequency.
> > Is this even possible?
> >
> > Thanks,
> > Chris
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
>
>
>
>

------------------------------------------------
Subject: RE: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation of AFSS
From: christopher.melick at us.af.mil
Time: Fri Feb 16 12:58:58 2018

Hi Tressa,

I think I know what you are saying but I am still a little confused.
Is the idea to calculate a Fractions Skill Score (FSS) for each
neighborhood box on the grid domain and then to average the result?  I
have never approached it that way:  Form what I understood in Eqs. 5
and 6 of Roberts and Lean (2008),  I figured it was a tally of all the
probability differences across the domain based on different
neighborhood sizes.   As for your main question, I don't eliminate
certain grid points in the computation domain based on whether the
neighborhood box has the right number of grid points.   Would that
affect the result for FSS that much?

For AFSS, I am using  (2 * fo *fm) / [(fo*fo) + (fm*fm)] from Roberts
and Lean (2008).   Is that the same formula that MET uses?   Wouldn't
this need to be re-calculated for each successive increase in the
neighborhood length since I have more undefined grid boxes?   The
addition of missing grid points come about because I am using MET to
do interpolation for different neighborhood sizes in grid_stat.

Thanks,
Chris

//SIGNED//

Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693



-----Original Message-----
From: Tressa Fowler via RT [mailto:met_help at ucar.edu]
Sent: Friday, February 16, 2018 1:01 PM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation of
AFSS

Hi Chris,

The missing grid locations are not exactly what I meant.

Let's say for example that you have a domain of 5 x 5 points and are
interested in computing FSS over square neighborhoods of width 3 (e.g.
9 grid points). At each of your 25 grid locations, you will find the
proportion of events in both forecast and observed fields at the grid
location and the surrounding 8 grid points that form a box around it.
You will calculate your score over 25 (overlapping) neighborhoods,
preferably of size 9 points. However, sixteen of your grid locations
are on the edge of your domain and will be missing part of the 3x3
neighborhood. A corner will have only 4 of the 9 neighborhood points.
Are those 4 used in the FSS or are any locations with less than 9
neighborhood points ignored?
Regardless, that corner grid point gets counted less often.  It can
only get used for neighborhoods centered on itself and its 3
neighbors. It might be used only once or it could be used 4 times,
depending on how you handle less than 9 neighbors. The center point of
the grid gets used in calculating the fraction of events in 9
neighborhoods, that centered on itself and each of it's 8 immediately
surrounding points.

When you do AFSS, there is no problem with edges because you have only
one neighborhood that is the entire domain of 25 points and you just
figure out your event proportion for each of the forecast and observed
fields.

Does that make sense?

Tressa




On Fri, Feb 16, 2018 at 10:37 AM, christopher.melick at us.af.mil via RT
< met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104 >
>
> Hi Tressa,
>
> For each neighborhood size, I ensure that AFSS and FSS are computed
on
> the same domain area by masking any grid points that are
missing/undefined for
> either the forecast or observed grids.   I double checked this
computation
> a couple of ways in Python to ensure the value I obtained matched
one
> another.  Is this what you are talking about?  Is the
> forecast/observed frequency just the fractional coverage of
> forecast/observed events over the entire domain area (excluding
undefined grid points in the mask)?
>
> I will have to try your suggestion.   I have not done that yet. That
is a
> great way to ensure my approach and logic are working properly.
>
> Thanks,
> Chris
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: Tressa Fowler via RT [mailto:met_help at ucar.edu]
> Sent: Friday, February 16, 2018 11:18 AM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation of
AFSS
>
> Hi Chris,
>
> Unless you have some type of edge effect, AFSS should be higher than
FSS.
> This assumes that you compute FSS over each smaller neighborhood in
the
> identical domain that you are calculating AFSS over.
>
> Have you tried using your observation file as both the forecast and
> observed data to make sure you get 1 for your AFSS and average FSS
values?
>
> Sometimes in MET, neighborhoods with missing data at domain
boundaries are
> not included in calculations, which gives interior points greater
"weight"
> than boundary points and could lead to slight differences. Could
this be
> happening? It is more likely to matter with large neighborhoods and
small
> domains.
>
> Tressa
>
> On Fri Feb 16 09:18:15 2018, christopher.melick at us.af.mil wrote:
> > Hi John,
> >
> > I have been computing Fractions Skill Score (FSS) on my own which
I
> > have done in the past, just not in Python.  I'm relatively
confident
> > that my results are correct for the sample time period I have been
> > using for testing purposes.
> > However, I have never computed the Asymptotic Fractions Skill
Score
> > (AFSS) before.   I have double checked the logic and have been
using
> > Eq. 8 in Roberts and Lean (2008) as my reference point.  Still, I
get
> > the impression that AFSS should always be higher than the FSS I am
> > computing for a specific neighborhood.   However, in this case, I
am
> > getting lower values based on model-forecast and observed
frequency.
> > Is this even possible?
> >
> > Thanks,
> > Chris
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
>
>
>
>




------------------------------------------------
Subject: Computation of AFSS
From: John Halley Gotway
Time: Fri Feb 16 13:16:06 2018

Chris,

Yes, that's the same equation that MET uses to compute AFSS.  Take a
look
in met-6.1/src/libcode/vx_statistics/met_stats.cc at the function
named
"compute_afss()".  It is a function of the forecast and observed rate
of
the event over the verification region.

Hope that helps.

Thanks,
John

On Fri, Feb 16, 2018 at 12:58 PM, christopher.melick at us.af.mil via RT
<
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104 >
>
> Hi Tressa,
>
> I think I know what you are saying but I am still a little confused.
Is
> the idea to calculate a Fractions Skill Score (FSS) for each
neighborhood
> box on the grid domain and then to average the result?  I have never
> approached it that way:  Form what I understood in Eqs. 5 and 6 of
Roberts
> and Lean (2008),  I figured it was a tally of all the probability
> differences across the domain based on different neighborhood sizes.
As
> for your main question, I don't eliminate certain grid points in the
> computation domain based on whether the neighborhood box has the
right
> number of grid points.   Would that affect the result for FSS that
much?
>
> For AFSS, I am using  (2 * fo *fm) / [(fo*fo) + (fm*fm)] from
Roberts and
> Lean (2008).   Is that the same formula that MET uses?   Wouldn't
this need
> to be re-calculated for each successive increase in the neighborhood
length
> since I have more undefined grid boxes?   The addition of missing
grid
> points come about because I am using MET to do interpolation for
different
> neighborhood sizes in grid_stat.
>
> Thanks,
> Chris
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian
> Meteorologist, Verification Exploitation Team
> 16th Weather Squadron (16 WS/WXEV)
> 557th Weather Wing, Offutt AFB, NE
> DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: Tressa Fowler via RT [mailto:met_help at ucar.edu]
> Sent: Friday, February 16, 2018 1:01 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation
of AFSS
>
> Hi Chris,
>
> The missing grid locations are not exactly what I meant.
>
> Let's say for example that you have a domain of 5 x 5 points and are
> interested in computing FSS over square neighborhoods of width 3
(e.g. 9
> grid points). At each of your 25 grid locations, you will find the
> proportion of events in both forecast and observed fields at the
grid
> location and the surrounding 8 grid points that form a box around
it. You
> will calculate your score over 25 (overlapping) neighborhoods,
preferably
> of size 9 points. However, sixteen of your grid locations are on the
edge
> of your domain and will be missing part of the 3x3 neighborhood. A
corner
> will have only 4 of the 9 neighborhood points. Are those 4 used in
the FSS
> or are any locations with less than 9 neighborhood points ignored?
> Regardless, that corner grid point gets counted less often.  It can
only
> get used for neighborhoods centered on itself and its 3 neighbors.
It might
> be used only once or it could be used 4 times, depending on how you
handle
> less than 9 neighbors. The center point of the grid gets used in
> calculating the fraction of events in 9 neighborhoods, that centered
on
> itself and each of it's 8 immediately surrounding points.
>
> When you do AFSS, there is no problem with edges because you have
only one
> neighborhood that is the entire domain of 25 points and you just
figure out
> your event proportion for each of the forecast and observed fields.
>
> Does that make sense?
>
> Tressa
>
>
>
>
> On Fri, Feb 16, 2018 at 10:37 AM, christopher.melick at us.af.mil via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104 >
> >
> > Hi Tressa,
> >
> > For each neighborhood size, I ensure that AFSS and FSS are
computed on
> > the same domain area by masking any grid points that are
> missing/undefined for
> > either the forecast or observed grids.   I double checked this
> computation
> > a couple of ways in Python to ensure the value I obtained matched
one
> > another.  Is this what you are talking about?  Is the
> > forecast/observed frequency just the fractional coverage of
> > forecast/observed events over the entire domain area (excluding
> undefined grid points in the mask)?
> >
> > I will have to try your suggestion.   I have not done that yet.
That is a
> > great way to ensure my approach and logic are working properly.
> >
> > Thanks,
> > Chris
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> > -----Original Message-----
> > From: Tressa Fowler via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, February 16, 2018 11:18 AM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation of
AFSS
> >
> > Hi Chris,
> >
> > Unless you have some type of edge effect, AFSS should be higher
than FSS.
> > This assumes that you compute FSS over each smaller neighborhood
in the
> > identical domain that you are calculating AFSS over.
> >
> > Have you tried using your observation file as both the forecast
and
> > observed data to make sure you get 1 for your AFSS and average FSS
> values?
> >
> > Sometimes in MET, neighborhoods with missing data at domain
boundaries
> are
> > not included in calculations, which gives interior points greater
> "weight"
> > than boundary points and could lead to slight differences. Could
this be
> > happening? It is more likely to matter with large neighborhoods
and small
> > domains.
> >
> > Tressa
> >
> > On Fri Feb 16 09:18:15 2018, christopher.melick at us.af.mil wrote:
> > > Hi John,
> > >
> > > I have been computing Fractions Skill Score (FSS) on my own
which I
> > > have done in the past, just not in Python.  I'm relatively
confident
> > > that my results are correct for the sample time period I have
been
> > > using for testing purposes.
> > > However, I have never computed the Asymptotic Fractions Skill
Score
> > > (AFSS) before.   I have double checked the logic and have been
using
> > > Eq. 8 in Roberts and Lean (2008) as my reference point.  Still,
I get
> > > the impression that AFSS should always be higher than the FSS I
am
> > > computing for a specific neighborhood.   However, in this case,
I am
> > > getting lower values based on model-forecast and observed
frequency.
> > > Is this even possible?
> > >
> > > Thanks,
> > > Chris
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> >
> >
> >
>
>
>
>
>

------------------------------------------------
Subject: RE: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation of AFSS
From: christopher.melick at us.af.mil
Time: Fri Feb 16 13:24:25 2018

Hi John,

We are having computer access problems on our end so I can't access
the source code.   Is there any way you could please send the module
via e-mail to me?  If not, no worries -- I can check next week.

Thanks,
Chris

//SIGNED//

Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693



-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, February 16, 2018 2:16 PM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation of
AFSS

Chris,

Yes, that's the same equation that MET uses to compute AFSS.  Take a
look in met-6.1/src/libcode/vx_statistics/met_stats.cc at the function
named "compute_afss()".  It is a function of the forecast and observed
rate of the event over the verification region.

Hope that helps.

Thanks,
John

On Fri, Feb 16, 2018 at 12:58 PM, christopher.melick at us.af.mil via RT
< met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104 >
>
> Hi Tressa,
>
> I think I know what you are saying but I am still a little confused.
Is
> the idea to calculate a Fractions Skill Score (FSS) for each
> neighborhood box on the grid domain and then to average the result?
I
> have never approached it that way:  Form what I understood in Eqs. 5
> and 6 of Roberts and Lean (2008),  I figured it was a tally of all
the probability
> differences across the domain based on different neighborhood sizes.
As
> for your main question, I don't eliminate certain grid points in the
> computation domain based on whether the neighborhood box has the
right
> number of grid points.   Would that affect the result for FSS that
much?
>
> For AFSS, I am using  (2 * fo *fm) / [(fo*fo) + (fm*fm)] from
Roberts and
> Lean (2008).   Is that the same formula that MET uses?   Wouldn't
this need
> to be re-calculated for each successive increase in the neighborhood
length
> since I have more undefined grid boxes?   The addition of missing
grid
> points come about because I am using MET to do interpolation for
> different neighborhood sizes in grid_stat.
>
> Thanks,
> Chris
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: Tressa Fowler via RT [mailto:met_help at ucar.edu]
> Sent: Friday, February 16, 2018 1:01 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation
of AFSS
>
> Hi Chris,
>
> The missing grid locations are not exactly what I meant.
>
> Let's say for example that you have a domain of 5 x 5 points and are
> interested in computing FSS over square neighborhoods of width 3
(e.g. 9
> grid points). At each of your 25 grid locations, you will find the
> proportion of events in both forecast and observed fields at the
grid
> location and the surrounding 8 grid points that form a box around
it. You
> will calculate your score over 25 (overlapping) neighborhoods,
preferably
> of size 9 points. However, sixteen of your grid locations are on the
edge
> of your domain and will be missing part of the 3x3 neighborhood. A
corner
> will have only 4 of the 9 neighborhood points. Are those 4 used in
the FSS
> or are any locations with less than 9 neighborhood points ignored?
> Regardless, that corner grid point gets counted less often.  It can
only
> get used for neighborhoods centered on itself and its 3 neighbors.
It might
> be used only once or it could be used 4 times, depending on how you
handle
> less than 9 neighbors. The center point of the grid gets used in
> calculating the fraction of events in 9 neighborhoods, that centered
on
> itself and each of it's 8 immediately surrounding points.
>
> When you do AFSS, there is no problem with edges because you have
only one
> neighborhood that is the entire domain of 25 points and you just
figure out
> your event proportion for each of the forecast and observed fields.
>
> Does that make sense?
>
> Tressa
>
>
>
>
> On Fri, Feb 16, 2018 at 10:37 AM, christopher.melick at us.af.mil via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104 >
> >
> > Hi Tressa,
> >
> > For each neighborhood size, I ensure that AFSS and FSS are
computed on
> > the same domain area by masking any grid points that are
> missing/undefined for
> > either the forecast or observed grids.   I double checked this
> computation
> > a couple of ways in Python to ensure the value I obtained matched
one
> > another.  Is this what you are talking about?  Is the
> > forecast/observed frequency just the fractional coverage of
> > forecast/observed events over the entire domain area (excluding
> undefined grid points in the mask)?
> >
> > I will have to try your suggestion.   I have not done that yet.
That is a
> > great way to ensure my approach and logic are working properly.
> >
> > Thanks,
> > Chris
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> > -----Original Message-----
> > From: Tressa Fowler via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, February 16, 2018 11:18 AM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation of
AFSS
> >
> > Hi Chris,
> >
> > Unless you have some type of edge effect, AFSS should be higher
than FSS.
> > This assumes that you compute FSS over each smaller neighborhood
in the
> > identical domain that you are calculating AFSS over.
> >
> > Have you tried using your observation file as both the forecast
and
> > observed data to make sure you get 1 for your AFSS and average FSS
> values?
> >
> > Sometimes in MET, neighborhoods with missing data at domain
boundaries
> are
> > not included in calculations, which gives interior points greater
> "weight"
> > than boundary points and could lead to slight differences. Could
this be
> > happening? It is more likely to matter with large neighborhoods
and small
> > domains.
> >
> > Tressa
> >
> > On Fri Feb 16 09:18:15 2018, christopher.melick at us.af.mil wrote:
> > > Hi John,
> > >
> > > I have been computing Fractions Skill Score (FSS) on my own
which I
> > > have done in the past, just not in Python.  I'm relatively
confident
> > > that my results are correct for the sample time period I have
been
> > > using for testing purposes.
> > > However, I have never computed the Asymptotic Fractions Skill
Score
> > > (AFSS) before.   I have double checked the logic and have been
using
> > > Eq. 8 in Roberts and Lean (2008) as my reference point.  Still,
I get
> > > the impression that AFSS should always be higher than the FSS I
am
> > > computing for a specific neighborhood.   However, in this case,
I am
> > > getting lower values based on model-forecast and observed
frequency.
> > > Is this even possible?
> > >
> > > Thanks,
> > > Chris
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> >
> >
> >
>
>
>
>
>




------------------------------------------------
Subject: Computation of AFSS
From: John Halley Gotway
Time: Fri Feb 16 13:32:57 2018

Chris,

Sure thing.  I've attached that source code file, but I know that
sometimes
attachments don't come through.  So here's the functions for
Asymptotic and
Uniform Fractions Skill score:

////////////////////////////////////////////////////////////////////////


double compute_afss(double f_rate, double o_rate) {

   double num, den, afss;


   //

   // Compute Asymptotic Fractions Skill Score

   //

   num = 2.0*f_rate*o_rate;

   den = f_rate*f_rate + o_rate*o_rate;


   if(is_bad_data(f_rate) || is_bad_data(o_rate) || is_eq(den, 0.0)) {

      afss = bad_data_double;

   }

   else {

      afss = num/den;

   }


   return(afss);

}


////////////////////////////////////////////////////////////////////////


double compute_ufss(double o_rate) {

   double ufss;


   //

   // Compute Uniform Fractions Skill Score

   //

   if(is_bad_data(o_rate)) ufss = bad_data_double;

   else                    ufss = 0.5 + o_rate/2.0;


   return(ufss);

}


On Fri, Feb 16, 2018 at 1:24 PM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104 >
>
> Hi John,
>
> We are having computer access problems on our end so I can't access
the
> source code.   Is there any way you could please send the module via
e-mail
> to me?  If not, no worries -- I can check next week.
>
> Thanks,
> Chris
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian
> Meteorologist, Verification Exploitation Team
> 16th Weather Squadron (16 WS/WXEV)
> 557th Weather Wing, Offutt AFB, NE
> DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, February 16, 2018 2:16 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation
of AFSS
>
> Chris,
>
> Yes, that's the same equation that MET uses to compute AFSS.  Take a
look
> in met-6.1/src/libcode/vx_statistics/met_stats.cc at the function
named
> "compute_afss()".  It is a function of the forecast and observed
rate of
> the event over the verification region.
>
> Hope that helps.
>
> Thanks,
> John
>
> On Fri, Feb 16, 2018 at 12:58 PM, christopher.melick at us.af.mil via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104 >
> >
> > Hi Tressa,
> >
> > I think I know what you are saying but I am still a little
confused.   Is
> > the idea to calculate a Fractions Skill Score (FSS) for each
> > neighborhood box on the grid domain and then to average the
result?  I
> > have never approached it that way:  Form what I understood in Eqs.
5
> > and 6 of Roberts and Lean (2008),  I figured it was a tally of all
the
> probability
> > differences across the domain based on different neighborhood
sizes.   As
> > for your main question, I don't eliminate certain grid points in
the
> > computation domain based on whether the neighborhood box has the
right
> > number of grid points.   Would that affect the result for FSS that
much?
> >
> > For AFSS, I am using  (2 * fo *fm) / [(fo*fo) + (fm*fm)] from
Roberts and
> > Lean (2008).   Is that the same formula that MET uses?   Wouldn't
this
> need
> > to be re-calculated for each successive increase in the
neighborhood
> length
> > since I have more undefined grid boxes?   The addition of missing
grid
> > points come about because I am using MET to do interpolation for
> > different neighborhood sizes in grid_stat.
> >
> > Thanks,
> > Chris
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> > -----Original Message-----
> > From: Tressa Fowler via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, February 16, 2018 1:01 PM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation
of
> AFSS
> >
> > Hi Chris,
> >
> > The missing grid locations are not exactly what I meant.
> >
> > Let's say for example that you have a domain of 5 x 5 points and
are
> > interested in computing FSS over square neighborhoods of width 3
(e.g. 9
> > grid points). At each of your 25 grid locations, you will find the
> > proportion of events in both forecast and observed fields at the
grid
> > location and the surrounding 8 grid points that form a box around
it. You
> > will calculate your score over 25 (overlapping) neighborhoods,
preferably
> > of size 9 points. However, sixteen of your grid locations are on
the edge
> > of your domain and will be missing part of the 3x3 neighborhood. A
corner
> > will have only 4 of the 9 neighborhood points. Are those 4 used in
the
> FSS
> > or are any locations with less than 9 neighborhood points ignored?
> > Regardless, that corner grid point gets counted less often.  It
can only
> > get used for neighborhoods centered on itself and its 3 neighbors.
It
> might
> > be used only once or it could be used 4 times, depending on how
you
> handle
> > less than 9 neighbors. The center point of the grid gets used in
> > calculating the fraction of events in 9 neighborhoods, that
centered on
> > itself and each of it's 8 immediately surrounding points.
> >
> > When you do AFSS, there is no problem with edges because you have
only
> one
> > neighborhood that is the entire domain of 25 points and you just
figure
> out
> > your event proportion for each of the forecast and observed
fields.
> >
> > Does that make sense?
> >
> > Tressa
> >
> >
> >
> >
> > On Fri, Feb 16, 2018 at 10:37 AM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104 >
> > >
> > > Hi Tressa,
> > >
> > > For each neighborhood size, I ensure that AFSS and FSS are
computed on
> > > the same domain area by masking any grid points that are
> > missing/undefined for
> > > either the forecast or observed grids.   I double checked this
> > computation
> > > a couple of ways in Python to ensure the value I obtained
matched one
> > > another.  Is this what you are talking about?  Is the
> > > forecast/observed frequency just the fractional coverage of
> > > forecast/observed events over the entire domain area (excluding
> > undefined grid points in the mask)?
> > >
> > > I will have to try your suggestion.   I have not done that yet.
That
> is a
> > > great way to ensure my approach and logic are working properly.
> > >
> > > Thanks,
> > > Chris
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Tressa Fowler via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, February 16, 2018 11:18 AM
> > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > christopher.melick at us.af.mil>
> > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > matthew.dawson.8 at us.af.mil>;
> > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > Subject: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation
of AFSS
> > >
> > > Hi Chris,
> > >
> > > Unless you have some type of edge effect, AFSS should be higher
than
> FSS.
> > > This assumes that you compute FSS over each smaller neighborhood
in the
> > > identical domain that you are calculating AFSS over.
> > >
> > > Have you tried using your observation file as both the forecast
and
> > > observed data to make sure you get 1 for your AFSS and average
FSS
> > values?
> > >
> > > Sometimes in MET, neighborhoods with missing data at domain
boundaries
> > are
> > > not included in calculations, which gives interior points
greater
> > "weight"
> > > than boundary points and could lead to slight differences. Could
this
> be
> > > happening? It is more likely to matter with large neighborhoods
and
> small
> > > domains.
> > >
> > > Tressa
> > >
> > > On Fri Feb 16 09:18:15 2018, christopher.melick at us.af.mil wrote:
> > > > Hi John,
> > > >
> > > > I have been computing Fractions Skill Score (FSS) on my own
which I
> > > > have done in the past, just not in Python.  I'm relatively
confident
> > > > that my results are correct for the sample time period I have
been
> > > > using for testing purposes.
> > > > However, I have never computed the Asymptotic Fractions Skill
Score
> > > > (AFSS) before.   I have double checked the logic and have been
using
> > > > Eq. 8 in Roberts and Lean (2008) as my reference point.
Still, I get
> > > > the impression that AFSS should always be higher than the FSS
I am
> > > > computing for a specific neighborhood.   However, in this
case, I am
> > > > getting lower values based on model-forecast and observed
frequency.
> > > > Is this even possible?
> > > >
> > > > Thanks,
> > > > Chris
> > > >
> > > > //SIGNED//
> > > >
> > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>

------------------------------------------------
Subject: RE: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation of AFSS
From: christopher.melick at us.af.mil
Time: Fri Feb 16 14:37:28 2018

John,

Thank you very much!   I am using the same logic for afss. I just
haven't gotten around to calculating ufss but it appears to be
straightforward.

I did have a couple of questions with this C subroutine below:
1.)  Are the "bar"s in reference to an average?  If so, is this where
the computation for the input parameters is done over the neighborhood
boxes and then averaged?
2.)  There are a couple of instances like this:  I take it that
sl1l2_info.ffbar is the forecast probability squared?  I figured it
was done in another module.
3.)   Why is "n" multiplied in the numerator ("num") but then divided
by "n" in the denominator for calculating fbs?  I take it "n =
sl1l2_info.scount" is the count of grid points over the domain.
4.)  Are the "f_rate" and "o_rate" an average of some sort for every
neighborhood area over the domain?  Are weights used in this as well?
I figured the forecast/observed event was just the relative frequency
of defined events across the whole domain (excluding any undefined
grid points)?

Thanks again for your assistance,
Chris

  ////////////////////////////////////////////////////////////////////////

void NBRCNTInfo::compute_stats() {
   double num, den;
   int n;

   //
   // Compute FBS
   //
   n   = sl1l2_info.scount;
   num = sl1l2_info.ffbar*n + sl1l2_info.oobar*n -
2.0*sl1l2_info.fobar*n;

   if(n == 0) fbs.v = bad_data_double;
   else       fbs.v = (double) num/n;

   //
   // Compute FSS
   //
   num = fbs.v;
   den = sl1l2_info.ffbar + sl1l2_info.oobar;

   if(is_eq(den, 0.0)) fss.v = bad_data_double;
   else                fss.v = 1.0 - (num/den);

   //
   // Compute F_RATE and O_RATE
   //
   afss.v = compute_afss(f_rate.v, o_rate.v);
   ufss.v = compute_ufss(o_rate.v);

   return;
}

################################

//SIGNED//

Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693



-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, February 16, 2018 2:33 PM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation of
AFSS

Chris,

Sure thing.  I've attached that source code file, but I know that
sometimes attachments don't come through.  So here's the functions for
Asymptotic and Uniform Fractions Skill score:

////////////////////////////////////////////////////////////////////////


double compute_afss(double f_rate, double o_rate) {

   double num, den, afss;


   //

   // Compute Asymptotic Fractions Skill Score

   //

   num = 2.0*f_rate*o_rate;

   den = f_rate*f_rate + o_rate*o_rate;


   if(is_bad_data(f_rate) || is_bad_data(o_rate) || is_eq(den, 0.0)) {

      afss = bad_data_double;

   }

   else {

      afss = num/den;

   }


   return(afss);

}


////////////////////////////////////////////////////////////////////////


double compute_ufss(double o_rate) {

   double ufss;


   //

   // Compute Uniform Fractions Skill Score

   //

   if(is_bad_data(o_rate)) ufss = bad_data_double;

   else                    ufss = 0.5 + o_rate/2.0;


   return(ufss);

}


On Fri, Feb 16, 2018 at 1:24 PM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104 >
>
> Hi John,
>
> We are having computer access problems on our end so I can't access
the
> source code.   Is there any way you could please send the module via
e-mail
> to me?  If not, no worries -- I can check next week.
>
> Thanks,
> Chris
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, February 16, 2018 2:16 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation
of AFSS
>
> Chris,
>
> Yes, that's the same equation that MET uses to compute AFSS.  Take a
look
> in met-6.1/src/libcode/vx_statistics/met_stats.cc at the function
named
> "compute_afss()".  It is a function of the forecast and observed
rate of
> the event over the verification region.
>
> Hope that helps.
>
> Thanks,
> John
>
> On Fri, Feb 16, 2018 at 12:58 PM, christopher.melick at us.af.mil via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104 >
> >
> > Hi Tressa,
> >
> > I think I know what you are saying but I am still a little
confused.   Is
> > the idea to calculate a Fractions Skill Score (FSS) for each
> > neighborhood box on the grid domain and then to average the
result?  I
> > have never approached it that way:  Form what I understood in Eqs.
5
> > and 6 of Roberts and Lean (2008),  I figured it was a tally of all
the
> probability
> > differences across the domain based on different neighborhood
sizes.   As
> > for your main question, I don't eliminate certain grid points in
the
> > computation domain based on whether the neighborhood box has the
right
> > number of grid points.   Would that affect the result for FSS that
much?
> >
> > For AFSS, I am using  (2 * fo *fm) / [(fo*fo) + (fm*fm)] from
Roberts and
> > Lean (2008).   Is that the same formula that MET uses?   Wouldn't
this
> need
> > to be re-calculated for each successive increase in the
neighborhood
> length
> > since I have more undefined grid boxes?   The addition of missing
grid
> > points come about because I am using MET to do interpolation for
> > different neighborhood sizes in grid_stat.
> >
> > Thanks,
> > Chris
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> > -----Original Message-----
> > From: Tressa Fowler via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, February 16, 2018 1:01 PM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation
of
> AFSS
> >
> > Hi Chris,
> >
> > The missing grid locations are not exactly what I meant.
> >
> > Let's say for example that you have a domain of 5 x 5 points and
are
> > interested in computing FSS over square neighborhoods of width 3
(e.g. 9
> > grid points). At each of your 25 grid locations, you will find the
> > proportion of events in both forecast and observed fields at the
grid
> > location and the surrounding 8 grid points that form a box around
it. You
> > will calculate your score over 25 (overlapping) neighborhoods,
preferably
> > of size 9 points. However, sixteen of your grid locations are on
the edge
> > of your domain and will be missing part of the 3x3 neighborhood. A
corner
> > will have only 4 of the 9 neighborhood points. Are those 4 used in
the
> FSS
> > or are any locations with less than 9 neighborhood points ignored?
> > Regardless, that corner grid point gets counted less often.  It
can only
> > get used for neighborhoods centered on itself and its 3 neighbors.
It
> might
> > be used only once or it could be used 4 times, depending on how
you
> handle
> > less than 9 neighbors. The center point of the grid gets used in
> > calculating the fraction of events in 9 neighborhoods, that
centered on
> > itself and each of it's 8 immediately surrounding points.
> >
> > When you do AFSS, there is no problem with edges because you have
only
> one
> > neighborhood that is the entire domain of 25 points and you just
figure
> out
> > your event proportion for each of the forecast and observed
fields.
> >
> > Does that make sense?
> >
> > Tressa
> >
> >
> >
> >
> > On Fri, Feb 16, 2018 at 10:37 AM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104 >
> > >
> > > Hi Tressa,
> > >
> > > For each neighborhood size, I ensure that AFSS and FSS are
computed on
> > > the same domain area by masking any grid points that are
> > missing/undefined for
> > > either the forecast or observed grids.   I double checked this
> > computation
> > > a couple of ways in Python to ensure the value I obtained
matched one
> > > another.  Is this what you are talking about?  Is the
> > > forecast/observed frequency just the fractional coverage of
> > > forecast/observed events over the entire domain area (excluding
> > undefined grid points in the mask)?
> > >
> > > I will have to try your suggestion.   I have not done that yet.
That
> is a
> > > great way to ensure my approach and logic are working properly.
> > >
> > > Thanks,
> > > Chris
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Tressa Fowler via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, February 16, 2018 11:18 AM
> > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > christopher.melick at us.af.mil>
> > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > matthew.dawson.8 at us.af.mil>;
> > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > Subject: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation
of AFSS
> > >
> > > Hi Chris,
> > >
> > > Unless you have some type of edge effect, AFSS should be higher
than
> FSS.
> > > This assumes that you compute FSS over each smaller neighborhood
in the
> > > identical domain that you are calculating AFSS over.
> > >
> > > Have you tried using your observation file as both the forecast
and
> > > observed data to make sure you get 1 for your AFSS and average
FSS
> > values?
> > >
> > > Sometimes in MET, neighborhoods with missing data at domain
boundaries
> > are
> > > not included in calculations, which gives interior points
greater
> > "weight"
> > > than boundary points and could lead to slight differences. Could
this
> be
> > > happening? It is more likely to matter with large neighborhoods
and
> small
> > > domains.
> > >
> > > Tressa
> > >
> > > On Fri Feb 16 09:18:15 2018, christopher.melick at us.af.mil wrote:
> > > > Hi John,
> > > >
> > > > I have been computing Fractions Skill Score (FSS) on my own
which I
> > > > have done in the past, just not in Python.  I'm relatively
confident
> > > > that my results are correct for the sample time period I have
been
> > > > using for testing purposes.
> > > > However, I have never computed the Asymptotic Fractions Skill
Score
> > > > (AFSS) before.   I have double checked the logic and have been
using
> > > > Eq. 8 in Roberts and Lean (2008) as my reference point.
Still, I get
> > > > the impression that AFSS should always be higher than the FSS
I am
> > > > computing for a specific neighborhood.   However, in this
case, I am
> > > > getting lower values based on model-forecast and observed
frequency.
> > > > Is this even possible?
> > > >
> > > > Thanks,
> > > > Chris
> > > >
> > > > //SIGNED//
> > > >
> > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
>



------------------------------------------------
Subject: Computation of AFSS
From: John Halley Gotway
Time: Fri Feb 16 14:52:49 2018

Chris,

1.)  Are the "bar"s in reference to an average?  If so, is this where
the
computation for the input parameters is done over the neighborhood
boxes
and then averaged?

Yes, "bar" means an average.  The f's and o's here are the fractional
coverage values that have already been computed for a particular
threshold
and neighborhood size.  The partial sums (fbar, obar, ffbar, oobar,
and
fobar) are the mean of the fractional coverage values for all grid
points
falling inside the current spatial masking area.

2.)  There are a couple of instances like this:  I take it that
sl1l2_info.ffbar is the forecast probability squared?  I figured it
was
done in another module.i

Yes, ffbar is the mean(f*f) for all grid points in the current spatial
masking area.

3.)   Why is "n" multiplied in the numerator ("num") but then divided
by
"n" in the denominator for calculating fbs?  I take it "n =
sl1l2_info.scount" is the count of grid points over the domain.

Good point!  There's no point in multiplying the numerator and
denominator
by n.  Not sure why that's there.

When aggregating results over multiple cases in STAT-Analysis we
compute
weighted averages, where the weight is determined by n.  But that's
not
what's going on here.

4.)  Are the "f_rate" and "o_rate" an average of some sort for every
neighborhood area over the domain?  Are weights used in this as well?
I
figured the forecast/observed event was just the relative frequency of
defined events across the whole domain (excluding any undefined grid
points)?

For a given spatial masking area, f_rate and o_rate are just the ratio
of
grid points in that area where the event is occurring.  So the f_rate
and
o_rate are determined by the event threshold.  Looking at the output
from
MET, you may notice that the f_rate and o_rate change slightly as you
increase the neighborhood size.  But look closely at the number of
matched
pairs the TOTAL column.  As you increase the neighborhood size, you
get
bigger edge effects, and reduce the number of matched pairs that are
included.

Thanks,
John


On Fri, Feb 16, 2018 at 2:37 PM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104 >
>
> John,
>
> Thank you very much!   I am using the same logic for afss. I just
haven't
> gotten around to calculating ufss but it appears to be
straightforward.
>
> I did have a couple of questions with this C subroutine below:
> 1.)  Are the "bar"s in reference to an average?  If so, is this
where the
> computation for the input parameters is done over the neighborhood
boxes
> and then averaged?
> 2.)  There are a couple of instances like this:  I take it that
> sl1l2_info.ffbar is the forecast probability squared?  I figured it
was
> done in another module.
> 3.)   Why is "n" multiplied in the numerator ("num") but then
divided by
> "n" in the denominator for calculating fbs?  I take it "n =
> sl1l2_info.scount" is the count of grid points over the domain.
> 4.)  Are the "f_rate" and "o_rate" an average of some sort for every
> neighborhood area over the domain?  Are weights used in this as
well?  I
> figured the forecast/observed event was just the relative frequency
of
> defined events across the whole domain (excluding any undefined grid
> points)?
>
> Thanks again for your assistance,
> Chris
>
>
////////////////////////////////////////////////////////////////////////
>
> void NBRCNTInfo::compute_stats() {
>    double num, den;
>    int n;
>
>    //
>    // Compute FBS
>    //
>    n   = sl1l2_info.scount;
>    num = sl1l2_info.ffbar*n + sl1l2_info.oobar*n -
2.0*sl1l2_info.fobar*n;
>
>    if(n == 0) fbs.v = bad_data_double;
>    else       fbs.v = (double) num/n;
>
>    //
>    // Compute FSS
>    //
>    num = fbs.v;
>    den = sl1l2_info.ffbar + sl1l2_info.oobar;
>
>    if(is_eq(den, 0.0)) fss.v = bad_data_double;
>    else                fss.v = 1.0 - (num/den);
>
>    //
>    // Compute F_RATE and O_RATE
>    //
>    afss.v = compute_afss(f_rate.v, o_rate.v);
>    ufss.v = compute_ufss(o_rate.v);
>
>    return;
> }
>
> ################################
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian
> Meteorologist, Verification Exploitation Team
> 16th Weather Squadron (16 WS/WXEV)
> 557th Weather Wing, Offutt AFB, NE
> DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, February 16, 2018 2:33 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation
of AFSS
>
> Chris,
>
> Sure thing.  I've attached that source code file, but I know that
> sometimes attachments don't come through.  So here's the functions
for
> Asymptotic and Uniform Fractions Skill score:
>
>
////////////////////////////////////////////////////////////////////////
>
>
> double compute_afss(double f_rate, double o_rate) {
>
>    double num, den, afss;
>
>
>    //
>
>    // Compute Asymptotic Fractions Skill Score
>
>    //
>
>    num = 2.0*f_rate*o_rate;
>
>    den = f_rate*f_rate + o_rate*o_rate;
>
>
>    if(is_bad_data(f_rate) || is_bad_data(o_rate) || is_eq(den, 0.0))
{
>
>       afss = bad_data_double;
>
>    }
>
>    else {
>
>       afss = num/den;
>
>    }
>
>
>    return(afss);
>
> }
>
>
>
////////////////////////////////////////////////////////////////////////
>
>
> double compute_ufss(double o_rate) {
>
>    double ufss;
>
>
>    //
>
>    // Compute Uniform Fractions Skill Score
>
>    //
>
>    if(is_bad_data(o_rate)) ufss = bad_data_double;
>
>    else                    ufss = 0.5 + o_rate/2.0;
>
>
>    return(ufss);
>
> }
>
>
> On Fri, Feb 16, 2018 at 1:24 PM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104 >
> >
> > Hi John,
> >
> > We are having computer access problems on our end so I can't
access the
> > source code.   Is there any way you could please send the module
via
> e-mail
> > to me?  If not, no worries -- I can check next week.
> >
> > Thanks,
> > Chris
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, February 16, 2018 2:16 PM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation
of
> AFSS
> >
> > Chris,
> >
> > Yes, that's the same equation that MET uses to compute AFSS.  Take
a look
> > in met-6.1/src/libcode/vx_statistics/met_stats.cc at the function
named
> > "compute_afss()".  It is a function of the forecast and observed
rate of
> > the event over the verification region.
> >
> > Hope that helps.
> >
> > Thanks,
> > John
> >
> > On Fri, Feb 16, 2018 at 12:58 PM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104 >
> > >
> > > Hi Tressa,
> > >
> > > I think I know what you are saying but I am still a little
confused.
>  Is
> > > the idea to calculate a Fractions Skill Score (FSS) for each
> > > neighborhood box on the grid domain and then to average the
result?  I
> > > have never approached it that way:  Form what I understood in
Eqs. 5
> > > and 6 of Roberts and Lean (2008),  I figured it was a tally of
all the
> > probability
> > > differences across the domain based on different neighborhood
sizes.
>  As
> > > for your main question, I don't eliminate certain grid points in
the
> > > computation domain based on whether the neighborhood box has the
right
> > > number of grid points.   Would that affect the result for FSS
that
> much?
> > >
> > > For AFSS, I am using  (2 * fo *fm) / [(fo*fo) + (fm*fm)] from
Roberts
> and
> > > Lean (2008).   Is that the same formula that MET uses?
Wouldn't this
> > need
> > > to be re-calculated for each successive increase in the
neighborhood
> > length
> > > since I have more undefined grid boxes?   The addition of
missing grid
> > > points come about because I am using MET to do interpolation for
> > > different neighborhood sizes in grid_stat.
> > >
> > > Thanks,
> > > Chris
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Tressa Fowler via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, February 16, 2018 1:01 PM
> > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > christopher.melick at us.af.mil>
> > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > matthew.dawson.8 at us.af.mil>;
> > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #84104]
Computation of
> > AFSS
> > >
> > > Hi Chris,
> > >
> > > The missing grid locations are not exactly what I meant.
> > >
> > > Let's say for example that you have a domain of 5 x 5 points and
are
> > > interested in computing FSS over square neighborhoods of width 3
(e.g.
> 9
> > > grid points). At each of your 25 grid locations, you will find
the
> > > proportion of events in both forecast and observed fields at the
grid
> > > location and the surrounding 8 grid points that form a box
around it.
> You
> > > will calculate your score over 25 (overlapping) neighborhoods,
> preferably
> > > of size 9 points. However, sixteen of your grid locations are on
the
> edge
> > > of your domain and will be missing part of the 3x3 neighborhood.
A
> corner
> > > will have only 4 of the 9 neighborhood points. Are those 4 used
in the
> > FSS
> > > or are any locations with less than 9 neighborhood points
ignored?
> > > Regardless, that corner grid point gets counted less often.  It
can
> only
> > > get used for neighborhoods centered on itself and its 3
neighbors. It
> > might
> > > be used only once or it could be used 4 times, depending on how
you
> > handle
> > > less than 9 neighbors. The center point of the grid gets used in
> > > calculating the fraction of events in 9 neighborhoods, that
centered on
> > > itself and each of it's 8 immediately surrounding points.
> > >
> > > When you do AFSS, there is no problem with edges because you
have only
> > one
> > > neighborhood that is the entire domain of 25 points and you just
figure
> > out
> > > your event proportion for each of the forecast and observed
fields.
> > >
> > > Does that make sense?
> > >
> > > Tressa
> > >
> > >
> > >
> > >
> > > On Fri, Feb 16, 2018 at 10:37 AM, christopher.melick at us.af.mil
via RT
> <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104
>
> > > >
> > > > Hi Tressa,
> > > >
> > > > For each neighborhood size, I ensure that AFSS and FSS are
computed
> on
> > > > the same domain area by masking any grid points that are
> > > missing/undefined for
> > > > either the forecast or observed grids.   I double checked this
> > > computation
> > > > a couple of ways in Python to ensure the value I obtained
matched one
> > > > another.  Is this what you are talking about?  Is the
> > > > forecast/observed frequency just the fractional coverage of
> > > > forecast/observed events over the entire domain area
(excluding
> > > undefined grid points in the mask)?
> > > >
> > > > I will have to try your suggestion.   I have not done that
yet. That
> > is a
> > > > great way to ensure my approach and logic are working
properly.
> > > >
> > > > Thanks,
> > > > Chris
> > > >
> > > > //SIGNED//
> > > >
> > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Tressa Fowler via RT [mailto:met_help at ucar.edu]
> > > > Sent: Friday, February 16, 2018 11:18 AM
> > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > christopher.melick at us.af.mil>
> > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > matthew.dawson.8 at us.af.mil>;
> > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > > Subject: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation
of
> AFSS
> > > >
> > > > Hi Chris,
> > > >
> > > > Unless you have some type of edge effect, AFSS should be
higher than
> > FSS.
> > > > This assumes that you compute FSS over each smaller
neighborhood in
> the
> > > > identical domain that you are calculating AFSS over.
> > > >
> > > > Have you tried using your observation file as both the
forecast and
> > > > observed data to make sure you get 1 for your AFSS and average
FSS
> > > values?
> > > >
> > > > Sometimes in MET, neighborhoods with missing data at domain
> boundaries
> > > are
> > > > not included in calculations, which gives interior points
greater
> > > "weight"
> > > > than boundary points and could lead to slight differences.
Could this
> > be
> > > > happening? It is more likely to matter with large
neighborhoods and
> > small
> > > > domains.
> > > >
> > > > Tressa
> > > >
> > > > On Fri Feb 16 09:18:15 2018, christopher.melick at us.af.mil
wrote:
> > > > > Hi John,
> > > > >
> > > > > I have been computing Fractions Skill Score (FSS) on my own
which I
> > > > > have done in the past, just not in Python.  I'm relatively
> confident
> > > > > that my results are correct for the sample time period I
have been
> > > > > using for testing purposes.
> > > > > However, I have never computed the Asymptotic Fractions
Skill Score
> > > > > (AFSS) before.   I have double checked the logic and have
been
> using
> > > > > Eq. 8 in Roberts and Lean (2008) as my reference point.
Still, I
> get
> > > > > the impression that AFSS should always be higher than the
FSS I am
> > > > > computing for a specific neighborhood.   However, in this
case, I
> am
> > > > > getting lower values based on model-forecast and observed
> frequency.
> > > > > Is this even possible?
> > > > >
> > > > > Thanks,
> > > > > Chris
> > > > >
> > > > > //SIGNED//
> > > > >
> > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>

------------------------------------------------
Subject: RE: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation of AFSS
From: christopher.melick at us.af.mil
Time: Fri Feb 16 15:23:47 2018

Hi John,

Thank you for the detailed responses!

1)   I had never considered performing the computation for FSS that
way.  I guess what is nice about that way is I suppose you would be
able to display results per grid box based on statistics you get for
the neighborhood area.   The only problem I see is that the average
might mask out some of the good details in the domain since there
could be several areas with results of zero for the components
(especially for a rare type of forecast).   Was this the way that
Roberts and Lean (2008) were inferring in computing FSS?  The other
point is that the neighborhood probabilities that are computed might
not be the traditional way of a fractional coverage of an event or
explicitly represented.  I think the SPC probability contours from an
outlook is a good example which falls into the latter case.   It is
generally implicit that the probability values mean within 25 miles/40
km of a point.    In this scenario, the squared probability
differences between observations and forecast are just summed across
the domain to compute the FBS.

I don't really have any other questions for the other answers you
provided me.   I am still kind of perplexed as to why the AFSS is
lower than the FSS.   I will have a couple of colleagues here take a
look at my code just as a double check.  If there are still some
issues and not a solution, I might send along.   Would you or Tressa
mind a proofread and providing some critique?

Thanks,
Chris

//SIGNED//

Dr. Christopher J. Melick, DAF Civilian
Meteorologist, Verification Exploitation Team
16th Weather Squadron (16 WS/WXEV)
557th Weather Wing, Offutt AFB, NE
DSN 271-3693; Comm (402) 294-3693



-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, February 16, 2018 3:53 PM
To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES
<christopher.melick at us.af.mil>
Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>; CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation of
AFSS

Chris,

1.)  Are the "bar"s in reference to an average?  If so, is this where
the computation for the input parameters is done over the neighborhood
boxes and then averaged?

Yes, "bar" means an average.  The f's and o's here are the fractional
coverage values that have already been computed for a particular
threshold and neighborhood size.  The partial sums (fbar, obar, ffbar,
oobar, and
fobar) are the mean of the fractional coverage values for all grid
points falling inside the current spatial masking area.

2.)  There are a couple of instances like this:  I take it that
sl1l2_info.ffbar is the forecast probability squared?  I figured it
was done in another module.i

Yes, ffbar is the mean(f*f) for all grid points in the current spatial
masking area.

3.)   Why is "n" multiplied in the numerator ("num") but then divided
by
"n" in the denominator for calculating fbs?  I take it "n =
sl1l2_info.scount" is the count of grid points over the domain.

Good point!  There's no point in multiplying the numerator and
denominator by n.  Not sure why that's there.

When aggregating results over multiple cases in STAT-Analysis we
compute weighted averages, where the weight is determined by n.  But
that's not what's going on here.

4.)  Are the "f_rate" and "o_rate" an average of some sort for every
neighborhood area over the domain?  Are weights used in this as well?
I figured the forecast/observed event was just the relative frequency
of defined events across the whole domain (excluding any undefined
grid points)?

For a given spatial masking area, f_rate and o_rate are just the ratio
of grid points in that area where the event is occurring.  So the
f_rate and o_rate are determined by the event threshold.  Looking at
the output from MET, you may notice that the f_rate and o_rate change
slightly as you increase the neighborhood size.  But look closely at
the number of matched pairs the TOTAL column.  As you increase the
neighborhood size, you get bigger edge effects, and reduce the number
of matched pairs that are included.

Thanks,
John


On Fri, Feb 16, 2018 at 2:37 PM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104 >
>
> John,
>
> Thank you very much!   I am using the same logic for afss. I just
haven't
> gotten around to calculating ufss but it appears to be
straightforward.
>
> I did have a couple of questions with this C subroutine below:
> 1.)  Are the "bar"s in reference to an average?  If so, is this
where
> the computation for the input parameters is done over the
neighborhood
> boxes and then averaged?
> 2.)  There are a couple of instances like this:  I take it that
> sl1l2_info.ffbar is the forecast probability squared?  I figured it
> was done in another module.
> 3.)   Why is "n" multiplied in the numerator ("num") but then
divided by
> "n" in the denominator for calculating fbs?  I take it "n =
> sl1l2_info.scount" is the count of grid points over the domain.
> 4.)  Are the "f_rate" and "o_rate" an average of some sort for every
> neighborhood area over the domain?  Are weights used in this as
well?
> I figured the forecast/observed event was just the relative
frequency
> of defined events across the whole domain (excluding any undefined
> grid points)?
>
> Thanks again for your assistance,
> Chris
>
>
>
//////////////////////////////////////////////////////////////////////
> //
>
> void NBRCNTInfo::compute_stats() {
>    double num, den;
>    int n;
>
>    //
>    // Compute FBS
>    //
>    n   = sl1l2_info.scount;
>    num = sl1l2_info.ffbar*n + sl1l2_info.oobar*n -
> 2.0*sl1l2_info.fobar*n;
>
>    if(n == 0) fbs.v = bad_data_double;
>    else       fbs.v = (double) num/n;
>
>    //
>    // Compute FSS
>    //
>    num = fbs.v;
>    den = sl1l2_info.ffbar + sl1l2_info.oobar;
>
>    if(is_eq(den, 0.0)) fss.v = bad_data_double;
>    else                fss.v = 1.0 - (num/den);
>
>    //
>    // Compute F_RATE and O_RATE
>    //
>    afss.v = compute_afss(f_rate.v, o_rate.v);
>    ufss.v = compute_ufss(o_rate.v);
>
>    return;
> }
>
> ################################
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian Meteorologist, Verification
> Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, February 16, 2018 2:33 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation
of AFSS
>
> Chris,
>
> Sure thing.  I've attached that source code file, but I know that
> sometimes attachments don't come through.  So here's the functions
for
> Asymptotic and Uniform Fractions Skill score:
>
>
////////////////////////////////////////////////////////////////////////
>
>
> double compute_afss(double f_rate, double o_rate) {
>
>    double num, den, afss;
>
>
>    //
>
>    // Compute Asymptotic Fractions Skill Score
>
>    //
>
>    num = 2.0*f_rate*o_rate;
>
>    den = f_rate*f_rate + o_rate*o_rate;
>
>
>    if(is_bad_data(f_rate) || is_bad_data(o_rate) || is_eq(den, 0.0))
{
>
>       afss = bad_data_double;
>
>    }
>
>    else {
>
>       afss = num/den;
>
>    }
>
>
>    return(afss);
>
> }
>
>
>
////////////////////////////////////////////////////////////////////////
>
>
> double compute_ufss(double o_rate) {
>
>    double ufss;
>
>
>    //
>
>    // Compute Uniform Fractions Skill Score
>
>    //
>
>    if(is_bad_data(o_rate)) ufss = bad_data_double;
>
>    else                    ufss = 0.5 + o_rate/2.0;
>
>
>    return(ufss);
>
> }
>
>
> On Fri, Feb 16, 2018 at 1:24 PM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104 >
> >
> > Hi John,
> >
> > We are having computer access problems on our end so I can't
access the
> > source code.   Is there any way you could please send the module
via
> e-mail
> > to me?  If not, no worries -- I can check next week.
> >
> > Thanks,
> > Chris
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, February 16, 2018 2:16 PM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation
of
> AFSS
> >
> > Chris,
> >
> > Yes, that's the same equation that MET uses to compute AFSS.  Take
a look
> > in met-6.1/src/libcode/vx_statistics/met_stats.cc at the function
named
> > "compute_afss()".  It is a function of the forecast and observed
rate of
> > the event over the verification region.
> >
> > Hope that helps.
> >
> > Thanks,
> > John
> >
> > On Fri, Feb 16, 2018 at 12:58 PM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104 >
> > >
> > > Hi Tressa,
> > >
> > > I think I know what you are saying but I am still a little
confused.
>  Is
> > > the idea to calculate a Fractions Skill Score (FSS) for each
> > > neighborhood box on the grid domain and then to average the
result?  I
> > > have never approached it that way:  Form what I understood in
Eqs. 5
> > > and 6 of Roberts and Lean (2008),  I figured it was a tally of
all the
> > probability
> > > differences across the domain based on different neighborhood
sizes.
>  As
> > > for your main question, I don't eliminate certain grid points in
the
> > > computation domain based on whether the neighborhood box has the
right
> > > number of grid points.   Would that affect the result for FSS
that
> much?
> > >
> > > For AFSS, I am using  (2 * fo *fm) / [(fo*fo) + (fm*fm)] from
Roberts
> and
> > > Lean (2008).   Is that the same formula that MET uses?
Wouldn't this
> > need
> > > to be re-calculated for each successive increase in the
neighborhood
> > length
> > > since I have more undefined grid boxes?   The addition of
missing grid
> > > points come about because I am using MET to do interpolation for
> > > different neighborhood sizes in grid_stat.
> > >
> > > Thanks,
> > > Chris
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Tressa Fowler via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, February 16, 2018 1:01 PM
> > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > christopher.melick at us.af.mil>
> > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > matthew.dawson.8 at us.af.mil>;
> > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #84104]
Computation of
> > AFSS
> > >
> > > Hi Chris,
> > >
> > > The missing grid locations are not exactly what I meant.
> > >
> > > Let's say for example that you have a domain of 5 x 5 points and
are
> > > interested in computing FSS over square neighborhoods of width 3
(e.g.
> 9
> > > grid points). At each of your 25 grid locations, you will find
the
> > > proportion of events in both forecast and observed fields at the
grid
> > > location and the surrounding 8 grid points that form a box
around it.
> You
> > > will calculate your score over 25 (overlapping) neighborhoods,
> preferably
> > > of size 9 points. However, sixteen of your grid locations are on
the
> edge
> > > of your domain and will be missing part of the 3x3 neighborhood.
A
> corner
> > > will have only 4 of the 9 neighborhood points. Are those 4 used
in the
> > FSS
> > > or are any locations with less than 9 neighborhood points
ignored?
> > > Regardless, that corner grid point gets counted less often.  It
can
> only
> > > get used for neighborhoods centered on itself and its 3
neighbors. It
> > might
> > > be used only once or it could be used 4 times, depending on how
you
> > handle
> > > less than 9 neighbors. The center point of the grid gets used in
> > > calculating the fraction of events in 9 neighborhoods, that
centered on
> > > itself and each of it's 8 immediately surrounding points.
> > >
> > > When you do AFSS, there is no problem with edges because you
have only
> > one
> > > neighborhood that is the entire domain of 25 points and you just
figure
> > out
> > > your event proportion for each of the forecast and observed
fields.
> > >
> > > Does that make sense?
> > >
> > > Tressa
> > >
> > >
> > >
> > >
> > > On Fri, Feb 16, 2018 at 10:37 AM, christopher.melick at us.af.mil
via RT
> <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104
>
> > > >
> > > > Hi Tressa,
> > > >
> > > > For each neighborhood size, I ensure that AFSS and FSS are
computed
> on
> > > > the same domain area by masking any grid points that are
> > > missing/undefined for
> > > > either the forecast or observed grids.   I double checked this
> > > computation
> > > > a couple of ways in Python to ensure the value I obtained
matched one
> > > > another.  Is this what you are talking about?  Is the
> > > > forecast/observed frequency just the fractional coverage of
> > > > forecast/observed events over the entire domain area
(excluding
> > > undefined grid points in the mask)?
> > > >
> > > > I will have to try your suggestion.   I have not done that
yet. That
> > is a
> > > > great way to ensure my approach and logic are working
properly.
> > > >
> > > > Thanks,
> > > > Chris
> > > >
> > > > //SIGNED//
> > > >
> > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Tressa Fowler via RT [mailto:met_help at ucar.edu]
> > > > Sent: Friday, February 16, 2018 11:18 AM
> > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > christopher.melick at us.af.mil>
> > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > matthew.dawson.8 at us.af.mil>;
> > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > > Subject: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation
of
> AFSS
> > > >
> > > > Hi Chris,
> > > >
> > > > Unless you have some type of edge effect, AFSS should be
higher than
> > FSS.
> > > > This assumes that you compute FSS over each smaller
neighborhood in
> the
> > > > identical domain that you are calculating AFSS over.
> > > >
> > > > Have you tried using your observation file as both the
forecast and
> > > > observed data to make sure you get 1 for your AFSS and average
FSS
> > > values?
> > > >
> > > > Sometimes in MET, neighborhoods with missing data at domain
> boundaries
> > > are
> > > > not included in calculations, which gives interior points
greater
> > > "weight"
> > > > than boundary points and could lead to slight differences.
Could this
> > be
> > > > happening? It is more likely to matter with large
neighborhoods and
> > small
> > > > domains.
> > > >
> > > > Tressa
> > > >
> > > > On Fri Feb 16 09:18:15 2018, christopher.melick at us.af.mil
wrote:
> > > > > Hi John,
> > > > >
> > > > > I have been computing Fractions Skill Score (FSS) on my own
which I
> > > > > have done in the past, just not in Python.  I'm relatively
> confident
> > > > > that my results are correct for the sample time period I
have been
> > > > > using for testing purposes.
> > > > > However, I have never computed the Asymptotic Fractions
Skill Score
> > > > > (AFSS) before.   I have double checked the logic and have
been
> using
> > > > > Eq. 8 in Roberts and Lean (2008) as my reference point.
Still, I
> get
> > > > > the impression that AFSS should always be higher than the
FSS I am
> > > > > computing for a specific neighborhood.   However, in this
case, I
> am
> > > > > getting lower values based on model-forecast and observed
> frequency.
> > > > > Is this even possible?
> > > > >
> > > > > Thanks,
> > > > > Chris
> > > > >
> > > > > //SIGNED//
> > > > >
> > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>




------------------------------------------------
Subject: Computation of AFSS
From: John Halley Gotway
Time: Fri Feb 16 15:31:09 2018

Chris,

I agree that Tressa would be a good resource for more info on the
neighborhood stats.  Yes, FSS should always be less than or equal to
AFSS.

I went ahead and remove the n/n from the FBS equation for the next
release.  But the regression test that we run every night should
ensure I
didn't accidentally break anything!

Thanks,
John

On Fri, Feb 16, 2018 at 3:23 PM, christopher.melick at us.af.mil via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104 >
>
> Hi John,
>
> Thank you for the detailed responses!
>
> 1)   I had never considered performing the computation for FSS that
way.
> I guess what is nice about that way is I suppose you would be able
to
> display results per grid box based on statistics you get for the
> neighborhood area.   The only problem I see is that the average
might mask
> out some of the good details in the domain since there could be
several
> areas with results of zero for the components (especially for a rare
type
> of forecast).   Was this the way that Roberts and Lean (2008) were
> inferring in computing FSS?  The other point is that the
neighborhood
> probabilities that are computed might not be the traditional way of
a
> fractional coverage of an event or explicitly represented.  I think
the SPC
> probability contours from an outlook is a good example which falls
into the
> latter case.   It is generally implicit that the probability values
mean
> within 25 miles/40 km of a point.    In this scenario, the squared
> probability differences between observations and forecast ar!
>  e just summed across the domain to compute the FBS.
>
> I don't really have any other questions for the other answers you
provided
> me.   I am still kind of perplexed as to why the AFSS is lower than
the
> FSS.   I will have a couple of colleagues here take a look at my
code just
> as a double check.  If there are still some issues and not a
solution, I
> might send along.   Would you or Tressa mind a proofread and
providing some
> critique?
>
> Thanks,
> Chris
>
> //SIGNED//
>
> Dr. Christopher J. Melick, DAF Civilian
> Meteorologist, Verification Exploitation Team
> 16th Weather Squadron (16 WS/WXEV)
> 557th Weather Wing, Offutt AFB, NE
> DSN 271-3693; Comm (402) 294-3693
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Friday, February 16, 2018 3:53 PM
> To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> christopher.melick at us.af.mil>
> Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE
<matthew.dawson.8 at us.af.mil>;
> CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN <robert.craig.2 at us.af.mil>
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation
of AFSS
>
> Chris,
>
> 1.)  Are the "bar"s in reference to an average?  If so, is this
where the
> computation for the input parameters is done over the neighborhood
boxes
> and then averaged?
>
> Yes, "bar" means an average.  The f's and o's here are the
fractional
> coverage values that have already been computed for a particular
threshold
> and neighborhood size.  The partial sums (fbar, obar, ffbar, oobar,
and
> fobar) are the mean of the fractional coverage values for all grid
points
> falling inside the current spatial masking area.
>
> 2.)  There are a couple of instances like this:  I take it that
> sl1l2_info.ffbar is the forecast probability squared?  I figured it
was
> done in another module.i
>
> Yes, ffbar is the mean(f*f) for all grid points in the current
spatial
> masking area.
>
> 3.)   Why is "n" multiplied in the numerator ("num") but then
divided by
> "n" in the denominator for calculating fbs?  I take it "n =
> sl1l2_info.scount" is the count of grid points over the domain.
>
> Good point!  There's no point in multiplying the numerator and
denominator
> by n.  Not sure why that's there.
>
> When aggregating results over multiple cases in STAT-Analysis we
compute
> weighted averages, where the weight is determined by n.  But that's
not
> what's going on here.
>
> 4.)  Are the "f_rate" and "o_rate" an average of some sort for every
> neighborhood area over the domain?  Are weights used in this as
well?  I
> figured the forecast/observed event was just the relative frequency
of
> defined events across the whole domain (excluding any undefined grid
> points)?
>
> For a given spatial masking area, f_rate and o_rate are just the
ratio of
> grid points in that area where the event is occurring.  So the
f_rate and
> o_rate are determined by the event threshold.  Looking at the output
from
> MET, you may notice that the f_rate and o_rate change slightly as
you
> increase the neighborhood size.  But look closely at the number of
matched
> pairs the TOTAL column.  As you increase the neighborhood size, you
get
> bigger edge effects, and reduce the number of matched pairs that are
> included.
>
> Thanks,
> John
>
>
> On Fri, Feb 16, 2018 at 2:37 PM, christopher.melick at us.af.mil via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104 >
> >
> > John,
> >
> > Thank you very much!   I am using the same logic for afss. I just
haven't
> > gotten around to calculating ufss but it appears to be
straightforward.
> >
> > I did have a couple of questions with this C subroutine below:
> > 1.)  Are the "bar"s in reference to an average?  If so, is this
where
> > the computation for the input parameters is done over the
neighborhood
> > boxes and then averaged?
> > 2.)  There are a couple of instances like this:  I take it that
> > sl1l2_info.ffbar is the forecast probability squared?  I figured
it
> > was done in another module.
> > 3.)   Why is "n" multiplied in the numerator ("num") but then
divided by
> > "n" in the denominator for calculating fbs?  I take it "n =
> > sl1l2_info.scount" is the count of grid points over the domain.
> > 4.)  Are the "f_rate" and "o_rate" an average of some sort for
every
> > neighborhood area over the domain?  Are weights used in this as
well?
> > I figured the forecast/observed event was just the relative
frequency
> > of defined events across the whole domain (excluding any undefined
> > grid points)?
> >
> > Thanks again for your assistance,
> > Chris
> >
> >
> >
//////////////////////////////////////////////////////////////////////
> > //
> >
> > void NBRCNTInfo::compute_stats() {
> >    double num, den;
> >    int n;
> >
> >    //
> >    // Compute FBS
> >    //
> >    n   = sl1l2_info.scount;
> >    num = sl1l2_info.ffbar*n + sl1l2_info.oobar*n -
> > 2.0*sl1l2_info.fobar*n;
> >
> >    if(n == 0) fbs.v = bad_data_double;
> >    else       fbs.v = (double) num/n;
> >
> >    //
> >    // Compute FSS
> >    //
> >    num = fbs.v;
> >    den = sl1l2_info.ffbar + sl1l2_info.oobar;
> >
> >    if(is_eq(den, 0.0)) fss.v = bad_data_double;
> >    else                fss.v = 1.0 - (num/den);
> >
> >    //
> >    // Compute F_RATE and O_RATE
> >    //
> >    afss.v = compute_afss(f_rate.v, o_rate.v);
> >    ufss.v = compute_ufss(o_rate.v);
> >
> >    return;
> > }
> >
> > ################################
> >
> > //SIGNED//
> >
> > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th Weather
> > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> >
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Friday, February 16, 2018 2:33 PM
> > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > christopher.melick at us.af.mil>
> > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> matthew.dawson.8 at us.af.mil>;
> > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #84104] Computation
of
> AFSS
> >
> > Chris,
> >
> > Sure thing.  I've attached that source code file, but I know that
> > sometimes attachments don't come through.  So here's the functions
for
> > Asymptotic and Uniform Fractions Skill score:
> >
> >
////////////////////////////////////////////////////////////////////////
> >
> >
> > double compute_afss(double f_rate, double o_rate) {
> >
> >    double num, den, afss;
> >
> >
> >    //
> >
> >    // Compute Asymptotic Fractions Skill Score
> >
> >    //
> >
> >    num = 2.0*f_rate*o_rate;
> >
> >    den = f_rate*f_rate + o_rate*o_rate;
> >
> >
> >    if(is_bad_data(f_rate) || is_bad_data(o_rate) || is_eq(den,
0.0)) {
> >
> >       afss = bad_data_double;
> >
> >    }
> >
> >    else {
> >
> >       afss = num/den;
> >
> >    }
> >
> >
> >    return(afss);
> >
> > }
> >
> >
> >
////////////////////////////////////////////////////////////////////////
> >
> >
> > double compute_ufss(double o_rate) {
> >
> >    double ufss;
> >
> >
> >    //
> >
> >    // Compute Uniform Fractions Skill Score
> >
> >    //
> >
> >    if(is_bad_data(o_rate)) ufss = bad_data_double;
> >
> >    else                    ufss = 0.5 + o_rate/2.0;
> >
> >
> >    return(ufss);
> >
> > }
> >
> >
> > On Fri, Feb 16, 2018 at 1:24 PM, christopher.melick at us.af.mil via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104 >
> > >
> > > Hi John,
> > >
> > > We are having computer access problems on our end so I can't
access the
> > > source code.   Is there any way you could please send the module
via
> > e-mail
> > > to me?  If not, no worries -- I can check next week.
> > >
> > > Thanks,
> > > Chris
> > >
> > > //SIGNED//
> > >
> > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Friday, February 16, 2018 2:16 PM
> > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > christopher.melick at us.af.mil>
> > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > matthew.dawson.8 at us.af.mil>;
> > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #84104]
Computation of
> > AFSS
> > >
> > > Chris,
> > >
> > > Yes, that's the same equation that MET uses to compute AFSS.
Take a
> look
> > > in met-6.1/src/libcode/vx_statistics/met_stats.cc at the
function
> named
> > > "compute_afss()".  It is a function of the forecast and observed
rate
> of
> > > the event over the verification region.
> > >
> > > Hope that helps.
> > >
> > > Thanks,
> > > John
> > >
> > > On Fri, Feb 16, 2018 at 12:58 PM, christopher.melick at us.af.mil
via RT
> <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104
>
> > > >
> > > > Hi Tressa,
> > > >
> > > > I think I know what you are saying but I am still a little
confused.
> >  Is
> > > > the idea to calculate a Fractions Skill Score (FSS) for each
> > > > neighborhood box on the grid domain and then to average the
result?
> I
> > > > have never approached it that way:  Form what I understood in
Eqs. 5
> > > > and 6 of Roberts and Lean (2008),  I figured it was a tally of
all
> the
> > > probability
> > > > differences across the domain based on different neighborhood
sizes.
> >  As
> > > > for your main question, I don't eliminate certain grid points
in the
> > > > computation domain based on whether the neighborhood box has
the
> right
> > > > number of grid points.   Would that affect the result for FSS
that
> > much?
> > > >
> > > > For AFSS, I am using  (2 * fo *fm) / [(fo*fo) + (fm*fm)] from
Roberts
> > and
> > > > Lean (2008).   Is that the same formula that MET uses?
Wouldn't
> this
> > > need
> > > > to be re-calculated for each successive increase in the
neighborhood
> > > length
> > > > since I have more undefined grid boxes?   The addition of
missing
> grid
> > > > points come about because I am using MET to do interpolation
for
> > > > different neighborhood sizes in grid_stat.
> > > >
> > > > Thanks,
> > > > Chris
> > > >
> > > > //SIGNED//
> > > >
> > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Tressa Fowler via RT [mailto:met_help at ucar.edu]
> > > > Sent: Friday, February 16, 2018 1:01 PM
> > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > christopher.melick at us.af.mil>
> > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > matthew.dawson.8 at us.af.mil>;
> > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil>
> > > > Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #84104]
Computation
> of
> > > AFSS
> > > >
> > > > Hi Chris,
> > > >
> > > > The missing grid locations are not exactly what I meant.
> > > >
> > > > Let's say for example that you have a domain of 5 x 5 points
and are
> > > > interested in computing FSS over square neighborhoods of width
3
> (e.g.
> > 9
> > > > grid points). At each of your 25 grid locations, you will find
the
> > > > proportion of events in both forecast and observed fields at
the grid
> > > > location and the surrounding 8 grid points that form a box
around it.
> > You
> > > > will calculate your score over 25 (overlapping) neighborhoods,
> > preferably
> > > > of size 9 points. However, sixteen of your grid locations are
on the
> > edge
> > > > of your domain and will be missing part of the 3x3
neighborhood. A
> > corner
> > > > will have only 4 of the 9 neighborhood points. Are those 4
used in
> the
> > > FSS
> > > > or are any locations with less than 9 neighborhood points
ignored?
> > > > Regardless, that corner grid point gets counted less often.
It can
> > only
> > > > get used for neighborhoods centered on itself and its 3
neighbors. It
> > > might
> > > > be used only once or it could be used 4 times, depending on
how you
> > > handle
> > > > less than 9 neighbors. The center point of the grid gets used
in
> > > > calculating the fraction of events in 9 neighborhoods, that
centered
> on
> > > > itself and each of it's 8 immediately surrounding points.
> > > >
> > > > When you do AFSS, there is no problem with edges because you
have
> only
> > > one
> > > > neighborhood that is the entire domain of 25 points and you
just
> figure
> > > out
> > > > your event proportion for each of the forecast and observed
fields.
> > > >
> > > > Does that make sense?
> > > >
> > > > Tressa
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Feb 16, 2018 at 10:37 AM, christopher.melick at us.af.mil
via
> RT
> > <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=84104 >
> > > > >
> > > > > Hi Tressa,
> > > > >
> > > > > For each neighborhood size, I ensure that AFSS and FSS are
computed
> > on
> > > > > the same domain area by masking any grid points that are
> > > > missing/undefined for
> > > > > either the forecast or observed grids.   I double checked
this
> > > > computation
> > > > > a couple of ways in Python to ensure the value I obtained
matched
> one
> > > > > another.  Is this what you are talking about?  Is the
> > > > > forecast/observed frequency just the fractional coverage of
> > > > > forecast/observed events over the entire domain area
(excluding
> > > > undefined grid points in the mask)?
> > > > >
> > > > > I will have to try your suggestion.   I have not done that
yet.
> That
> > > is a
> > > > > great way to ensure my approach and logic are working
properly.
> > > > >
> > > > > Thanks,
> > > > > Chris
> > > > >
> > > > > //SIGNED//
> > > > >
> > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
Verification
> > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
Weather
> > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Tressa Fowler via RT [mailto:met_help at ucar.edu]
> > > > > Sent: Friday, February 16, 2018 11:18 AM
> > > > > To: MELICK, CHRISTOPHER J GS-12 USAF ACC 16 WS/WXES <
> > > > > christopher.melick at us.af.mil>
> > > > > Cc: DAWSON, MATTHEW G Capt USAF ACC 16 WS/WXE <
> > > > matthew.dawson.8 at us.af.mil>;
> > > > > CRAIG, ROBERT J GS-12 USAF ACC 16 WS/WXN
<robert.craig.2 at us.af.mil
> >
> > > > > Subject: [Non-DoD Source] [rt.rap.ucar.edu #84104]
Computation of
> > AFSS
> > > > >
> > > > > Hi Chris,
> > > > >
> > > > > Unless you have some type of edge effect, AFSS should be
higher
> than
> > > FSS.
> > > > > This assumes that you compute FSS over each smaller
neighborhood in
> > the
> > > > > identical domain that you are calculating AFSS over.
> > > > >
> > > > > Have you tried using your observation file as both the
forecast and
> > > > > observed data to make sure you get 1 for your AFSS and
average FSS
> > > > values?
> > > > >
> > > > > Sometimes in MET, neighborhoods with missing data at domain
> > boundaries
> > > > are
> > > > > not included in calculations, which gives interior points
greater
> > > > "weight"
> > > > > than boundary points and could lead to slight differences.
Could
> this
> > > be
> > > > > happening? It is more likely to matter with large
neighborhoods and
> > > small
> > > > > domains.
> > > > >
> > > > > Tressa
> > > > >
> > > > > On Fri Feb 16 09:18:15 2018, christopher.melick at us.af.mil
wrote:
> > > > > > Hi John,
> > > > > >
> > > > > > I have been computing Fractions Skill Score (FSS) on my
own
> which I
> > > > > > have done in the past, just not in Python.  I'm relatively
> > confident
> > > > > > that my results are correct for the sample time period I
have
> been
> > > > > > using for testing purposes.
> > > > > > However, I have never computed the Asymptotic Fractions
Skill
> Score
> > > > > > (AFSS) before.   I have double checked the logic and have
been
> > using
> > > > > > Eq. 8 in Roberts and Lean (2008) as my reference point.
Still, I
> > get
> > > > > > the impression that AFSS should always be higher than the
FSS I
> am
> > > > > > computing for a specific neighborhood.   However, in this
case, I
> > am
> > > > > > getting lower values based on model-forecast and observed
> > frequency.
> > > > > > Is this even possible?
> > > > > >
> > > > > > Thanks,
> > > > > > Chris
> > > > > >
> > > > > > //SIGNED//
> > > > > >
> > > > > > Dr. Christopher J. Melick, DAF Civilian Meteorologist,
> Verification
> > > > > > Exploitation Team 16th Weather Squadron (16 WS/WXEV) 557th
> Weather
> > > > > > Wing, Offutt AFB, NE DSN 271-3693; Comm (402) 294-3693
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
>

------------------------------------------------